Case Study: Support Vector Regression: Risk Quadrangle Framework

Instructions for optimization with PSG Run-File, PSG MATLAB Toolbox, PSG Python, and PSG R.

Problem 1: Nu-SVR as Error Minimization (primal)
Minimize (1-alpha)*cvar_risk + quadratic
Value:
var_risk
——————————————————————————————
quadratic = quadratic function specified by a matrix_quadratic
cvar_risk = conditional value-at-risk specified by an abs(matrix of scenarios)
var_risk = value-at-risk specified by an abs(matrix of scenarios)
——————————————————————————————
Data and solution in different environments:

Problem 2: epsilon-SVR as Error Minimization (primal)
Minimize pm_pen + quadratic
Value:
pr_pen
——————————————————————–——————
pm_pen = partial moment function specified by an abs(matrix of scenarios)
pr_pen = probability of exceedance specified by an abs(matrix of scenarios)
quadratic = quadratic function specified by a matrix_quadratic
——————————————————————–——————
Data and solution in different environments:

Problem 3: Nu-SVR as Deviation Minimization (primal)
Minimize (1-alpha)/2*cvar_dev + (1+alpha)/2*cvar_dev + quadratic
Value:
var_risk
——————————————————————–——————
cvar_dev = conditional value-at-risk deviation specified by a reduced matrix of scenarios
quadratic = quadratic function specified by a reduced matrix_quadratic
——————————————————————–——————
Data and solution in different environments:

Problem 4a: Nu-SVR as Error Minimization (dual, constraints linearized automatically)
Maximize linear – 0.5*quadratic
Subject to
polynom_abs <= C*(1-alpha)
linear = 0
Box: <= C, >= -C
——————————————————————–——————
linear = linear fumction specified by a vector of dependent values (in objective) and by a unit vector (in constraint)
polynom_abs = l1 norm specified by a unit vector
quadratic = quadratic function specified by a linear kernel matrix
——————————————————————–——————
Data and solution in different environments:

Problem 4b: Nu-SVR as Error Minimization (dual, constraints linearized manually)
Maximize linear – 0.5*quadratic
Subject to
linear <= C*(1-alpha)
linear = 0
-1<=linearmulti<=1
limearmulti>=0
limearmulti>=0
——————————————————————–——————
linear = linear fumction specified by a vector of dependent values (in objective) and by a unit vector (in constraint)
polynom_abs = l1 norm specified by a unit vector
quadratic = quadratic function specified by a linear kernel matrix
linearmulti = linear multiple function specified by a sparse pmatrix in each case
——————————————————————–——————
Data and solution in different environments:

CASE STUDY SUMMARY

This case study conducts Support Vector Regression (SVR) as regularized error minimization, regularized deviation minimization, and constrained quadratic programming (CQP) problem on simulated data (1000 observations), cf. [1].
Specifically, four equivalent implementations of SVR are considered (cf. Proposition 4.2 and Corollary 4.5 in [1] for the proof of equivalence ):
1. The CVaR Norm plus quadratic regularization penalty is minimized. The CVaR Norm is calculated as the superposition of the CVaR Risk and the absolute value of regression residuals (i.e., the absolute value of the standard PSG linear losses).
2. The Partial Moment function plus quadratic regularization penalty is minimized.
3. Convex combination of two CVaR deviations plus quadratic regularization penalty is minimized. The intercept is calculated as the average of two corresponding symmetric quantiles.
4. QP problem (dual to the CVaR Norm plus quadratic regularization penalty minimization) is solved. The solution to the primal problem is constructed.

References
[1] Malandii, A. and S. Uryasev. Support Vector Regression: Risk Quadrangle Framework, Jan 2023, arXiv:2212.09178