# Examples of using L1General

These demos have been set-up to show how L1General can be used to optimize a variety of smooth objective functions subject to L1-regularization.

The code used to run these demos can be downloaded here (which includes a copy of the 2010 version of L1General).

To run all of the demos, type (in Matlab):

```>> cd L1GeneralExamples      % Change to the extracted directory
>> mexAll                    % Compile mex files (not necessary on all systems)
>> L1General_examples        % Run the examples
```

## LASSO

```% Generate some data
nInstances = 250;
nVars = 50;
X = randn(nInstances,nVars);
y = X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1);

% Least Squares Solution
wLS = X\y;

% Ridge Regression
lambda = 100*ones(nVars,1); % Penalize each element by the same amount
R = chol(X'*X + diag(lambda));
wRR = R\(R'\(X'*y));

% LASSO
lambda = 100*ones(nVars,1); % Penalize the absolute value of each element by the same amount
funObj = @(w)SquaredError(w,X,y); % Loss function that L1 regularization is applied to
w_init = wRR; % Initial value for iterative optimizer
fprintf('\nComputing LASSO Coefficients...\n');
wLASSO = L1General2_PSSgb(funObj,w_init,lambda);

fprintf('Number of non-zero variables in Least Squares solution: %d\n',nnz(wLS));
fprintf('Number of non-zero variables in Ridge Regression solution: %d\n',nnz(wRR));
fprintf('Number of non-zero variables in LASSO solution: %d\n',nnz(wLASSO));

figure;
clf;hold on;
subplot(2,2,1);
stem(wLS,'r');
xlim([1 nVars]);
yl = ylim;
title('Least Squares');
subplot(2,2,2);
stem(wRR,'b');
xlim([1 nVars]);
ylim(yl);
title('Ridge Regression');
subplot(2,2,3);
stem(wLASSO,'g');
xlim([1 nVars]);
title('LASSO');
ylim(yl);
pause;
```
```Computing LASSO Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 2.49592e-04 1.99301e+03 3.16398e+02     43
2      3 1.00000e+00 1.76918e+03 8.91349e+01     19
3      4 1.00000e+00 1.75367e+03 3.12464e+01     21
4      5 1.00000e+00 1.75160e+03 4.64930e+00     21
5      6 1.00000e+00 1.75145e+03 8.46870e-01     20
6      7 1.00000e+00 1.75145e+03 2.53280e-01     20
7      8 1.00000e+00 1.75145e+03 9.39831e-02     20
8      9 1.00000e+00 1.75145e+03 1.55213e-02     20
9     10 1.00000e+00 1.75145e+03 5.85301e-03     20
10     11 1.00000e+00 1.75145e+03 8.86230e-04     20
11     12 1.00000e+00 1.75145e+03 9.06708e-04     20
Progress in parameters or objective below progTol
Number of non-zero variables in Least Squares solution: 50
Number of non-zero variables in Ridge Regression solution: 50
Number of non-zero variables in LASSO solution: 20
```

## Elastic Net

```lambdaL2 = 100*ones(nVars,1);
lambdaL1 = 100*ones(nVars,1);
penalizedFunObj = @(w)penalizedL2(w,funObj,lambdaL2);
fprintf('\nComputing Elastic Net Coefficients...\n');
wElastic = L1General2_PSSgb(penalizedFunObj,w_init,lambdaL1);

fprintf('Number of non-zero variables in Elastic Net solution: %d\n',nnz(wElastic));

subplot(2,2,4);
stem(wElastic,'c');
xlim([1 nVars]);
ylim(yl);
title('Elastic Net');
pause;
```
```Computing Elastic Net Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 2.00000e-04 3.27151e+03 9.28964e+01     43
2      3 1.00000e+00 3.12917e+03 2.52775e+01     22
3      4 1.00000e+00 3.12721e+03 1.26982e+01     22
4      5 1.00000e+00 3.12675e+03 1.25912e+00     21
5      6 1.00000e+00 3.12674e+03 2.57555e-01     21
6      7 1.00000e+00 3.12674e+03 1.07619e-01     21
7      8 1.00000e+00 3.12674e+03 1.64976e-02     21
8      9 1.00000e+00 3.12674e+03 1.06693e-03     21
9     10 1.00000e+00 3.12674e+03 4.13000e-04     21
10     11 1.00000e+00 3.12674e+03 7.03746e-05     21
Progress in parameters or objective below progTol
Number of non-zero variables in Elastic Net solution: 21
```

## Logistic Regression

```X = [ones(nInstances,1) X]; % Add Bias element to features
y = sign(y); % Convert y to binary {-1,1} representation

funObj = @(w)LogisticLoss(w,X,y);
w_init = zeros(nVars+1,1);

% Maximum Likelihood
fprintf('\nComputing Maximum Likelihood Logistic Regression Coefficients\n');
mfOptions.Method = 'newton';
wLogML = minFunc(funObj,w_init,mfOptions);

% L2-Regularized Logistic Regression
fprintf('\nComputing L2-Regularized Logistic Regression Coefficients...\n');
lambda = 15*ones(nVars+1,1);
lambda(1) = 0; % Do not penalize bias variable
funObjL2 = @(w)penalizedL2(w,funObj,lambda);
wLogL2 = minFunc(funObjL2,w_init,mfOptions);

% L1-Regularized Logistic Regression
fprintf('\nComputing L1-Regularized Logistic Regression Coefficients...\n');
wLogL1 = L1General2_PSSgb(funObj,w_init,lambda);

% Elastic Net Logistic Regression
fprintf('\nComputing Elastic-Net Logistic Regression Coefficients...\n');
wLogL1L2 = L1General2_PSSgb(funObjL2,w_init,lambda);

figure;
clf;hold on;
subplot(2,2,1);
stem(wLogML,'r');
xlim([1 nVars+1]);
title('Maximum Likelihood Logistic Regression');
subplot(2,2,2);
stem(wLogL2,'b');
xlim([1 nVars+1]);
title('L2-Regularized Logistic Regression');
subplot(2,2,3);
stem(wLogL1,'g');
xlim([1 nVars+1]);
title('L1-Regularized Logistic Regression');
subplot(2,2,4);
stem(wLogL1L2,'c');
xlim([1 nVars+1]);
title('Elastic-Net Logistic Regression');

fprintf('Number of Features Selected by Maximum Likelihood Logistic Regression classifier: %d (out of %d)\n',nnz(wLogML(2:end)),nVars);
fprintf('Number of Features Selected by L2-regualrized Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL2(2:end)),nVars);
fprintf('Number of Features Selected by L1-regualrized Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL1(2:end)),nVars);
fprintf('Number of Features Selected by Elastic-Net Logistic Regression classifier: %d (out of %d)\n',nnz(wLogL1L2(2:end)),nVars);
fprintf('Classification error rate on training data for L1-regularied Logistic Regression: %.2f\n',sum(y ~= sign(X*wLogL1))/length(y));
pause;
```
```Computing Maximum Likelihood Logistic Regression Coefficients
Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          2     1.00000e+00     7.07045e+01     1.97507e+01
2          3     1.00000e+00     4.23053e+01     8.07759e+00
3          4     1.00000e+00     2.66959e+01     3.49396e+00
4          5     1.00000e+00     1.62143e+01     1.55590e+00
5          6     1.00000e+00     8.10661e+00     6.90238e-01
6          7     1.00000e+00     3.38291e+00     2.82238e-01
7          8     1.00000e+00     1.25949e+00     1.07228e-01
8          9     1.00000e+00     4.61854e-01     3.98042e-02
9         10     1.00000e+00     1.69851e-01     1.47850e-02
10         11     1.00000e+00     6.26252e-02     5.48087e-03
11         12     1.00000e+00     2.31072e-02     2.02741e-03
12         13     1.00000e+00     8.52514e-03     7.48738e-04
13         14     1.00000e+00     3.14421e-03     2.76176e-04
14         15     1.00000e+00     1.15922e-03     1.01770e-04
15         16     1.00000e+00     4.27240e-04     3.74723e-05
16         17     1.00000e+00     1.57415e-04     1.37887e-05
17         18     1.00000e+00     5.79822e-05     5.07135e-06
Optimality Condition below optTol

Computing L2-Regularized Logistic Regression Coefficients...
Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          2     1.00000e+00     1.05577e+02     8.83408e+00
2          3     1.00000e+00     1.03449e+02     5.21616e-01
3          4     1.00000e+00     1.03438e+02     3.09434e-03
4          5     1.00000e+00     1.03438e+02     1.26932e-07
Optimality Condition below optTol

Computing L1-Regularized Logistic Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 6.00892e-03 1.50194e+02 3.29826e+01     11
2      3 1.00000e+00 1.35832e+02 8.96380e+00      7
3      4 1.00000e+00 1.34305e+02 2.60035e+00      7
4      5 1.00000e+00 1.34139e+02 3.92810e-01      7
5      6 1.00000e+00 1.34136e+02 1.24296e-01      7
6      7 1.00000e+00 1.34135e+02 6.21160e-02      7
7      8 1.00000e+00 1.34135e+02 1.12322e-03      7
8      9 1.00000e+00 1.34135e+02 1.24925e-04      7
9     10 1.00000e+00 1.34135e+02 8.38783e-06      7
First-order optimality below optTol

Computing Elastic-Net Logistic Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 6.00892e-03 1.52889e+02 2.33081e+01     11
2      3 1.00000e+00 1.48589e+02 2.77799e+00      8
3      4 1.00000e+00 1.48514e+02 2.03804e-01      8
4      5 1.00000e+00 1.48514e+02 1.74764e-02      8
5      6 1.00000e+00 1.48514e+02 3.36673e-03      8
6      7 1.00000e+00 1.48514e+02 1.50247e-04      8
Directional derivative below progTol
Number of Features Selected by Maximum Likelihood Logistic Regression classifier: 50 (out of 50)
Number of Features Selected by L2-regualrized Logistic Regression classifier: 50 (out of 50)
Number of Features Selected by L1-regualrized Logistic Regression classifier: 6 (out of 50)
Number of Features Selected by Elastic-Net Logistic Regression classifier: 7 (out of 50)
Classification error rate on training data for L1-regularied Logistic Regression: 0.10
```

## L-BFGS Variant

```nVars = 5000;

X = [ones(nInstances,1) randn(nInstances,nVars-1)];
y = sign(X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1));

lambda = 10*ones(nVars,1);
lambda(1) = 0;
funObj = @(w)LogisticLoss(w,X,y);
fprintf('\nComputing Logistic Regression Coefficients for model with %d variables with L-BFGS\n',nVars);
wLogistic = L1General2_PSSgb(funObj,zeros(nVars,1),lambda);

figure;
clf;
stem(wLogistic,'g');
title(sprintf('L1-Regularized Logistic Regression (%d vars)',nVars));

fprintf('Number of Features Selected by Logistic Regression classifier: %d (out of %d)\n',nnz(wLogistic(2:end)),nVars);
fprintf('Classification error rate on training data: %.2f\n',sum(y ~= sign(X*wLogistic))/length(y));
pause;
```
```Computing Logistic Regression Coefficients for model with 5000 variables with L-BFGS
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 2.24113e-04 1.67068e+02 1.96701e+01   1147
2      3 1.00000e+00 1.48643e+02 8.84668e+00    768
3      4 1.00000e+00 1.45242e+02 6.51621e+00    583
4      5 1.00000e+00 1.42914e+02 4.75094e+00    386
5      6 1.00000e+00 1.39937e+02 3.44001e+00    312
6      7 1.00000e+00 1.36860e+02 2.99006e+00    191
7      8 1.00000e+00 1.36225e+02 4.07078e+00    122
Backtracking...
Backtracking...
Backtracking...
Backtracking...
8     13 6.59677e-04 1.36204e+02 3.92474e+00    125
9     14 1.00000e+00 1.35018e+02 2.49947e+00    128
10     15 1.00000e+00 1.34552e+02 1.19953e+00    120
11     16 1.00000e+00 1.34424e+02 1.19847e+00    110
12     17 1.00000e+00 1.34295e+02 6.10298e-01    106
Backtracking...
13     19 8.03893e-03 1.34287e+02 6.31270e-01    109
14     20 1.00000e+00 1.34267e+02 1.10060e+00    106
15     21 1.00000e+00 1.34223e+02 3.80897e-01    101
16     22 1.00000e+00 1.34209e+02 1.68550e-01    102
17     23 1.00000e+00 1.34203e+02 1.52216e-01    101
18     24 1.00000e+00 1.34201e+02 1.75575e-01    100
19     25 1.00000e+00 1.34199e+02 1.29820e-01     99
20     26 1.00000e+00 1.34199e+02 6.71145e-02     99
Backtracking...
21     28 3.08890e-01 1.34198e+02 3.54430e-02     99
22     29 1.00000e+00 1.34198e+02 2.49175e-02     99
23     30 1.00000e+00 1.34198e+02 1.44747e-02     99
24     31 1.00000e+00 1.34198e+02 2.46634e-02     99
25     32 1.00000e+00 1.34198e+02 3.25376e-02     98
26     33 1.00000e+00 1.34198e+02 4.11185e-03     99
27     34 1.00000e+00 1.34198e+02 2.29290e-03     99
28     35 1.00000e+00 1.34198e+02 2.41097e-03     99
29     36 1.00000e+00 1.34198e+02 1.80338e-03     99
30     37 1.00000e+00 1.34198e+02 4.79784e-04     99
31     38 1.00000e+00 1.34198e+02 2.82431e-04     99
32     39 1.00000e+00 1.34198e+02 1.58254e-04     99
33     40 1.00000e+00 1.34198e+02 1.12707e-04     99
34     41 1.00000e+00 1.34198e+02 1.27808e-04     99
35     42 1.00000e+00 1.34198e+02 3.38671e-05     99
Directional derivative below progTol
Number of Features Selected by Logistic Regression classifier: 98 (out of 5000)
Classification error rate on training data: 0.00
```

## Lasso Regularization Path

```% Generate some data
nInstances = 100;
nVars = 10;
X = randn(nInstances,nVars);
y = X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1);

% Find Maximum value for regularization parameter and pick step increment
[f,g] = SquaredError(zeros(nVars,1),X,y);
lambdaMax = max(abs(g));
lambdaInc = .01;

% Compute Regularization Path
fprintf('Computing Least Squares L1-Regularization path\n');
funObj = @(w)SquaredError(w,X,y);
w = zeros(nVars,1);
options = struct('verbose',0);
for mult = 1-lambdaInc:-lambdaInc:0
lambda = mult*lambdaMax*ones(nVars,1);
w(:,end+1) = L1General2_PSSgb(funObj,w(:,end),lambda,options);
end

figure;xData = 1:-lambdaInc:0;
yData = w;
plot(xData,yData);
title('Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Least Squares)');
xlabel('percent of lambdaMax');
ylabel('coefficient values');
pause;
```
```Computing Least Squares L1-Regularization path
```

## Huber Robust Regression Regularization Path

```changePoint = 1;

% Find Maximum value for regularization parameter and pick step increment
[f,g] = HuberLoss(zeros(nVars,1),X,y,changePoint);
lambdaMax = max(abs(g));
lambdaInc = .01;

% Compute Regularization Path
fprintf('Computing Huber Robust Regression L1-Regularization path\n');
funObj = @(w)HuberLoss(w,X,y,changePoint);
w = zeros(nVars,1);
options = struct('verbose',0);
for mult = 1-lambdaInc:-lambdaInc:0
lambda = mult*lambdaMax*ones(nVars,1);
w(:,end+1) = L1General2_PSSgb(funObj,w(:,end),lambda,options);
end

figure;xData = 1:-lambdaInc:0;
yData = w;
plot(xData,yData);
title('Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Huber Robust Regression)');
xlabel('percent of lambdaMax');
ylabel('coefficient values');
pause;
```
```Computing Huber Robust Regression L1-Regularization path
```

## Logistic Regression Regularization Path

```X = [ones(nInstances,1) X]; % Add Bias element to features
y = sign(y); % Convert y to binary {-1,1} representation

% Solve for Maximum Likelihood bias element
funObjBias = @(w)LogisticLoss(w,ones(nInstances,1),y);
bias = minFunc(funObjBias,0,struct('Display','none'));

% Find Maximum value for regularization parameter and pick step increment
w = [bias;zeros(nVars,1)];
[f,g] = LogisticLoss(w,X,y);
lambdaMax = max(abs(g));
lambdaInc = .01;

% Compute Regularization Path
fprintf('Computing Logistic Regression L1-Regularization path\n');
funObj = @(w)LogisticLoss(w,X,y);
for mult = 1-lambdaInc:-lambdaInc:lambdaInc
lambda = [0;mult*lambdaMax*ones(nVars,1)];
w(:,end+1) = L1General2_PSSgb(funObj,w(:,end),lambda,options);
end

figure;
plot(1:-lambdaInc:lambdaInc,w');
legend({'Bias'});
title('Regression Coefficients vs. L1-Regularization Strength (L1-Regularized Logistic Regression)');
xlabel('percent of lambdaMax');
ylabel('coefficient values');
pause;
```
```Computing Logistic Regression L1-Regularization path
```

## Probit Regression, Smooth SVM, Huberized SVM

```% Generate some data
nInstances = 500;
nVars = 100;
X = randn(nInstances,nVars);
y = sign(X*((rand(nVars,1) > .5).*randn(nVars,1)) + randn(nInstances,1)/5);

X = [ones(nInstances,1) X];
lambda = [0;10*ones(nVars,1)];

fprintf('\nComputing Logistic Regression Coefficients...\n');
funObj = @(w)LogisticLoss(w,X,y);
wLogit = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Probit Regression Coefficients...\n');
funObj = @(w)ProbitLoss(w,X,y);
wProbit = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Smooth Support Vector Machine Coefficients...\n');
funObj = @(w)SSVMLoss(w,X,y);
wSSVM = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Huberized Support Vector Machine Coefficients...\n');
t = .5;
funObj = @(w)HuberSVMLoss(w,X,y,t);
wHSVM = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

fprintf('\nComputing Extreme Value Regression Coefficients...\n');
funObj = @(w)ExtremeLoss(w,X,y);
wExtreme = L1General2_PSSgb(funObj,zeros(nVars+1,1),lambda);

figure;
clf;hold on;
subplot(3,2,1);
stem(wLogit,'r');
xlim([1 nVars]);
title('L1-Regularized Logistic Regression');
subplot(3,2,2);
stem(wProbit,'b');
xlim([1 nVars]);
title('L1-Regularized Probit Regression');
subplot(3,2,3);
stem(wSSVM,'g');
xlim([1 nVars]);
title('L1-Regularized Smooth SVM');
% subplot(3,2,4);
% stem(max(0,1-y.*(X*wSSVM)),'c');
% xlim([1 nInstances]);
% title('Hinge Loss for L1-Regularized SSVM');
subplot(3,2,4);
stem(wSSVM,'k');
xlim([1 nVars]);
title('L1-Regularized Huberized SVM');
subplot(3,2,5);
stem(max(0,1-y.*(X*wHSVM)),'m');
xlim([1 nInstances]);
title('Hinge Loss for L1-Regularized HSVM');
subplot(3,2,6)
stem(wExtreme,'c');
xlim([1 nVars]);
title('L1-Regularized Extreme-Value Regression');

fprintf('Number of non-zero variables for Logistic Regression: %d out of %d\n',nnz(wLogit(2:end)),nVars);
fprintf('Number of non-zero variables for Probit Regression:  %d out of %d\n',nnz(wProbit(2:end)),nVars);
fprintf('Number of non-zero variables for Smooth Support Vector Machine: %d out of %d',nnz(wSSVM(2:end)),nVars);
fprintf(' (%d support vectors)\n',sum(1-y.*(X*wSSVM)>=0));
fprintf('Number of non-zero variables for Huberized Support Vector Machine: %d out of %d',nnz(wHSVM(2:end)),nVars);
fprintf(' (%d support vectors)\n',sum(1-y.*(X*wHSVM)>=0));
fprintf('Number of non-zero variables for Extreme Value Regression: %d out of %d\n',nnz(wExtreme(2:end)),nVars);
pause;
```
```Computing Logistic Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 1.11219e-03 3.16670e+02 5.24024e+01     57
2      3 1.00000e+00 2.35672e+02 1.43651e+01     41
3      4 1.00000e+00 2.25972e+02 5.97437e+00     39
4      5 1.00000e+00 2.23176e+02 2.36338e+00     39
5      6 1.00000e+00 2.23072e+02 2.01425e+00     36
6      7 1.00000e+00 2.22938e+02 3.12517e-01     38
7      8 1.00000e+00 2.22937e+02 8.83875e-02     38
8      9 1.00000e+00 2.22937e+02 1.42327e-02     38
9     10 1.00000e+00 2.22937e+02 4.08680e-03     38
10     11 1.00000e+00 2.22937e+02 2.92593e-03     38
11     12 1.00000e+00 2.22937e+02 3.79409e-04     38
12     13 1.00000e+00 2.22937e+02 1.28940e-04     38
13     14 1.00000e+00 2.22937e+02 2.42499e-05     38
Progress in parameters or objective below progTol

Computing Probit Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 5.54304e-04 3.00238e+02 8.39334e+01     70
2      3 1.00000e+00 2.02146e+02 2.79057e+01     55
3      4 1.00000e+00 1.84718e+02 1.19909e+01     54
4      5 1.00000e+00 1.78376e+02 7.11523e+00     52
5      6 1.00000e+00 1.77394e+02 5.00142e+00     47
6      7 1.00000e+00 1.77010e+02 1.53044e+00     48
7      8 1.00000e+00 1.76994e+02 4.22121e-01     49
8      9 1.00000e+00 1.76991e+02 1.09574e-01     49
9     10 1.00000e+00 1.76990e+02 4.26942e-02     49
10     11 1.00000e+00 1.76990e+02 2.76763e-02     49
11     12 1.00000e+00 1.76990e+02 7.00188e-03     49
12     13 1.00000e+00 1.76990e+02 2.48059e-03     49
13     14 1.00000e+00 1.76990e+02 1.13110e-03     49
14     15 1.00000e+00 1.76990e+02 3.87316e-04     49
Backtracking...
15     17 4.11044e-01 1.76990e+02 9.68925e-05     49
Directional derivative below progTol

Computing Smooth Support Vector Machine Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 1.75512e-04 3.84887e+02 2.17640e+02     89
2      3 1.00000e+00 1.85515e+02 5.45835e+01     79
3      4 1.00000e+00 1.63225e+02 2.68508e+01     84
4      5 1.00000e+00 1.47825e+02 1.83658e+01     72
5      6 1.00000e+00 1.44710e+02 1.31008e+01     75
6      7 1.00000e+00 1.43129e+02 4.64824e+00     76
7      8 1.00000e+00 1.42845e+02 3.32625e+00     74
8      9 1.00000e+00 1.42638e+02 2.47059e+00     73
9     10 1.00000e+00 1.42542e+02 2.05454e+00     68
10     11 1.00000e+00 1.42527e+02 1.42891e+00     68
11     12 1.00000e+00 1.42512e+02 2.77524e-01     69
12     13 1.00000e+00 1.42511e+02 1.86336e-01     69
13     14 1.00000e+00 1.42510e+02 1.03574e-01     69
14     15 1.00000e+00 1.42510e+02 8.98808e-02     69
15     16 1.00000e+00 1.42510e+02 2.62123e-02     69
16     17 1.00000e+00 1.42510e+02 1.28982e-02     69
17     18 1.00000e+00 1.42510e+02 6.27322e-03     69
18     19 1.00000e+00 1.42510e+02 7.70964e-03     69
19     20 1.00000e+00 1.42510e+02 1.39856e-03     69
20     21 1.00000e+00 1.42510e+02 1.04135e-03     69
21     22 1.00000e+00 1.42510e+02 7.66865e-04     69
22     23 1.00000e+00 1.42510e+02 6.20470e-04     69
23     24 1.00000e+00 1.42510e+02 1.37625e-04     69
Directional derivative below progTol

Computing Huberized Support Vector Machine Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 4.09809e-04 3.09573e+02 1.37617e+02     76
Backtracking...
Backtracking...
Backtracking...
Backtracking...
Backtracking...
2      8 1.82115e-03 1.80320e+02 4.68808e+01     70
Backtracking...
3     10 1.90768e-01 1.70621e+02 3.96830e+01     69
4     11 1.00000e+00 1.44292e+02 1.89085e+01     65
5     12 1.00000e+00 1.42175e+02 2.90943e+01     56
6     13 1.00000e+00 1.36127e+02 9.61605e+00     58
7     14 1.00000e+00 1.34907e+02 4.21485e+00     61
8     15 1.00000e+00 1.34341e+02 3.61168e+00     59
9     16 1.00000e+00 1.34096e+02 3.43694e+00     61
10     17 1.00000e+00 1.33976e+02 2.31866e+00     61
11     18 1.00000e+00 1.33938e+02 9.81897e-01     60
12     19 1.00000e+00 1.33921e+02 5.69325e-01     61
13     20 1.00000e+00 1.33913e+02 4.59229e-01     61
14     21 1.00000e+00 1.33909e+02 3.31591e-01     61
15     22 1.00000e+00 1.33907e+02 2.17188e-01     61
16     23 1.00000e+00 1.33907e+02 1.01368e-01     60
17     24 1.00000e+00 1.33907e+02 4.39434e-02     60
18     25 1.00000e+00 1.33907e+02 3.06268e-02     60
19     26 1.00000e+00 1.33907e+02 3.10286e-02     60
20     27 1.00000e+00 1.33907e+02 1.04722e-02     60
21     28 1.00000e+00 1.33907e+02 8.10877e-03     60
22     29 1.00000e+00 1.33907e+02 3.55788e-03     60
23     30 1.00000e+00 1.33907e+02 1.75012e-03     60
24     31 1.00000e+00 1.33907e+02 1.35978e-03     60
Backtracking...
25     33 4.67586e-01 1.33907e+02 9.97574e-04     60
26     34 1.00000e+00 1.33907e+02 2.80376e-04     60
27     35 1.00000e+00 1.33907e+02 1.71551e-04     60
28     36 1.00000e+00 1.33907e+02 1.31008e-04     60
Progress in parameters or objective below progTol

Computing Extreme Value Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 5.40408e-04 3.13060e+02 7.98254e+01     66
2      3 1.00000e+00 2.23061e+02 2.81775e+01     57
3      4 1.00000e+00 1.99555e+02 1.32625e+01     52
4      5 1.00000e+00 1.90739e+02 6.44873e+00     50
5      6 1.00000e+00 1.89194e+02 4.24375e+00     47
6      7 1.00000e+00 1.88798e+02 1.24892e+00     46
7      8 1.00000e+00 1.88766e+02 4.46984e-01     47
8      9 1.00000e+00 1.88762e+02 1.16921e-01     47
9     10 1.00000e+00 1.88761e+02 6.50265e-02     47
10     11 1.00000e+00 1.88761e+02 3.61242e-02     47
11     12 1.00000e+00 1.88761e+02 4.43847e-03     47
12     13 1.00000e+00 1.88761e+02 1.99794e-03     47
13     14 1.00000e+00 1.88761e+02 8.04958e-04     47
14     15 1.00000e+00 1.88761e+02 8.22021e-04     47
15     16 1.00000e+00 1.88761e+02 1.02701e-04     47
Directional derivative below progTol
Number of non-zero variables for Logistic Regression: 37 out of 100
Number of non-zero variables for Probit Regression:  48 out of 100
Number of non-zero variables for Smooth Support Vector Machine: 68 out of 100 (246 support vectors)
Number of non-zero variables for Huberized Support Vector Machine: 59 out of 100 (256 support vectors)
Number of non-zero variables for Extreme Value Regression: 46 out of 100
```

## Non-Parametric Logistic Regression with Sparse Prototypes

```% Generate Non-Linear Data Set
nInstances = 100;
nVars = 2;
nExamplePoints = 4; % Set to 1 for linear classifier, higher for more non-linear
examplePoints = randn(nExamplePoints,nVars);
X = randn(nInstances,nVars);
y = zeros(nInstances,1);
for i = 1:nInstances
dists = sum((repmat(X(i,:),nExamplePoints,1) - examplePoints).^2,2);
[minVal minInd] = min(dists);
y(i,1) = sign(mod(minInd,2)-.5);
end

% Make Gram Matrix
XX = kernelRBF(X,X,1);

fprintf('Computing Non-Parametric Logistic Regression Coefficients...\n');
funObj = @(u)LogisticLoss(u,XX,y);
lambda = .5*ones(nInstances,1);
u = L1General2_PSSgb(funObj,zeros(nInstances,1),lambda);

% Plot Data
fprintf('Generating Plot...\n');
increment = 100;
figure;
clf; hold on;
plot(X(y==1,1),X(y==1,2),'.','color','g');
plot(X(y==-1,1),X(y==-1,2),'.','color','b');
domainx = xlim;
domain1 = domainx(1):(domainx(2)-domainx(1))/increment:domainx(2);
domainy = ylim;
domain2 = domainy(1):(domainy(2)-domainy(1))/increment:domainy(2);
d1 = repmat(domain1',[1 length(domain1)]);
d2 = repmat(domain2,[length(domain2) 1]);
vals = sign(kernelRBF([d1(:) d2(:)],X,1)*u);
zData = reshape(vals,size(d1));
contourf(d1,d2,zData+rand(size(zData))/1000,[-1 0],'k');
colormap([0 0 .5;0 .5 0]);
plot(X(y==1,1),X(y==1,2),'.','color','g');
plot(X(y==-1,1),X(y==-1,2),'.','color','b');
xlim(domainx);
ylim(domainy);

% Circle Prototypes
prototypes = X(u~=0,:);
fprintf('(%d prototypes)\n',size(prototypes,1));
h=plot(prototypes(:,1),prototypes(:,2),'ro');
set(h,'MarkerSize',10);
legend(h,'Non-Zero Data Points (Prototypes)');
pause;
```
```Computing Non-Parametric Logistic Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 5.20093e-03 6.67109e+01 3.98997e+00     95
2      3 1.00000e+00 4.58075e+01 2.30255e+00     87
3      4 1.00000e+00 3.94334e+01 1.18437e+00     95
4      5 1.00000e+00 3.76799e+01 5.71931e-01     94
5      6 1.00000e+00 3.69351e+01 4.62534e-01     81
6      7 1.00000e+00 3.65222e+01 3.60834e-01     74
7      8 1.00000e+00 3.60562e+01 2.90577e-01     60
8      9 1.00000e+00 3.56919e+01 3.29955e-01     58
9     10 1.00000e+00 3.51743e+01 5.80422e-01     39
Backtracking...
10     12 4.94147e-01 3.50166e+01 3.76194e-01     57
11     13 1.00000e+00 3.49203e+01 3.51194e-01     48
Backtracking...
12     15 4.24323e-01 3.48336e+01 5.50364e-01     43
13     16 1.00000e+00 3.47961e+01 3.82925e-01     38
Backtracking...
14     18 1.53452e-01 3.47810e+01 3.44684e-01     43
15     19 1.00000e+00 3.47560e+01 2.37006e-01     40
16     20 1.00000e+00 3.47112e+01 1.84156e-01     39
Backtracking...
17     22 3.77444e-01 3.46402e+01 2.80544e-01     36
Backtracking...
Backtracking...
18     25 1.52675e-03 3.46398e+01 1.26413e-01     34
19     26 1.00000e+00 3.46247e+01 1.34099e-01     31
20     27 1.00000e+00 3.45933e+01 8.93053e-02     32
Backtracking...
21     29 3.51940e-01 3.45605e+01 1.55374e-01     28
22     30 1.00000e+00 3.45233e+01 1.27413e-01     30
23     31 1.00000e+00 3.44337e+01 1.40339e-01     26
24     32 1.00000e+00 3.43791e+01 6.37444e-02     26
25     33 1.00000e+00 3.43549e+01 6.82287e-02     26
26     34 1.00000e+00 3.43109e+01 7.09055e-02     24
Backtracking...
27     36 4.64589e-01 3.42888e+01 7.98909e-02     24
28     37 1.00000e+00 3.42771e+01 7.27030e-02     22
Backtracking...
29     39 2.66997e-01 3.42700e+01 6.56971e-02     22
30     40 1.00000e+00 3.42575e+01 3.44141e-02     20
31     41 1.00000e+00 3.42456e+01 4.57312e-02     17
Backtracking...
32     43 1.74912e-01 3.42423e+01 1.72429e-02     18
Backtracking...
33     45 3.82482e-01 3.42399e+01 2.25954e-02     17
34     46 1.00000e+00 3.42365e+01 3.08340e-02     18
Backtracking...
35     48 1.54622e-01 3.42267e+01 4.95427e-02     17
Backtracking...
36     50 4.83985e-01 3.42238e+01 1.37321e-02     16
37     51 1.00000e+00 3.42193e+01 1.86491e-02     17
38     52 1.00000e+00 3.42045e+01 3.49788e-02     16
39     53 1.00000e+00 3.41973e+01 6.28513e-02     15
Backtracking...
40     55 1.63954e-01 3.41943e+01 4.28451e-02     17
41     56 1.00000e+00 3.41907e+01 2.04922e-02     17
42     57 1.00000e+00 3.41870e+01 2.52734e-02     19
43     58 1.00000e+00 3.41833e+01 2.02741e-02     15
Backtracking...
44     60 4.34681e-01 3.41824e+01 1.19566e-02     14
45     61 1.00000e+00 3.41817e+01 7.43078e-03     14
46     62 1.00000e+00 3.41803e+01 4.93236e-03     13
Backtracking...
47     64 5.58898e-01 3.41734e+01 9.05689e-03     12
Backtracking...
48     66 3.42166e-01 3.41725e+01 2.68754e-02     12
49     67 1.00000e+00 3.41677e+01 4.04584e-02     14
Backtracking...
50     69 9.78283e-02 3.41658e+01 1.14057e-02     15
Backtracking...
51     71 1.88171e-01 3.41655e+01 3.55383e-03     11
52     72 1.00000e+00 3.41654e+01 1.08598e-02     11
53     73 1.00000e+00 3.41653e+01 7.74173e-03     11
54     74 1.00000e+00 3.41651e+01 3.52344e-03     11
55     75 1.00000e+00 3.41645e+01 8.91291e-03     11
56     76 1.00000e+00 3.41631e+01 2.06320e-02     12
Backtracking...
57     78 2.31946e-01 3.41601e+01 1.94931e-02     12
Backtracking...
58     80 6.76889e-03 3.41599e+01 1.95370e-02     12
Backtracking...
59     82 4.16411e-03 3.41598e+01 1.95078e-02     12
Backtracking...
60     84 2.71201e-03 3.41597e+01 1.94916e-02     12
Backtracking...
61     86 1.99485e-03 3.41595e+01 1.94838e-02     12
Backtracking...
62     88 1.78948e-03 3.41594e+01 1.94696e-02     12
Backtracking...
63     90 1.47365e-03 3.41593e+01 1.94257e-02     12
Backtracking...
64     92 1.50794e-03 3.41591e+01 1.93989e-02     12
Backtracking...
65     94 1.37584e-03 3.41590e+01 1.93798e-02     12
Backtracking...
66     96 1.31671e-03 3.41589e+01 1.93498e-02     12
Backtracking...
67     98 1.24712e-03 3.41587e+01 1.93265e-02     12
Backtracking...
68    100 1.20677e-03 3.41586e+01 1.92994e-02     12
Backtracking...
69    102 1.15896e-03 3.41584e+01 1.92750e-02     12
Backtracking...
70    104 1.12421e-03 3.41583e+01 1.92489e-02     12
Backtracking...
71    106 1.08777e-03 3.41581e+01 1.92243e-02     12
Backtracking...
72    108 1.06066e-03 3.41580e+01 1.91987e-02     12
Backtracking...
73    110 1.02772e-03 3.41578e+01 1.91742e-02     12
Backtracking...
74    112 1.00146e-03 3.41577e+01 1.91494e-02     12
Backtracking...
75    114 1.00000e-03 3.41575e+01 1.91245e-02     12
Backtracking...
76    116 1.00000e-03 3.41574e+01 1.90990e-02     12
Backtracking...
77    118 1.00000e-03 3.41572e+01 1.90733e-02     12
Backtracking...
78    120 1.00000e-03 3.41570e+01 1.90470e-02     12
Backtracking...
79    122 1.00000e-03 3.41568e+01 1.90205e-02     12
Backtracking...
80    124 1.00000e-03 3.41566e+01 1.89936e-02     12
Backtracking...
81    126 1.00000e-03 3.41564e+01 1.89665e-02     12
Backtracking...
82    128 1.00000e-03 3.41561e+01 1.29535e-02     11
83    129 1.00000e+00 3.41538e+01 1.23784e-02     14
Backtracking...
Backtracking...
Backtracking...
84    133 5.50929e-05 3.41538e+01 1.25504e-02     13
85    134 1.00000e+00 3.41534e+01 2.51666e-02     10
Backtracking...
86    136 1.37384e-03 3.41533e+01 2.47699e-02     15
Backtracking...
87    138 3.51700e-03 3.41528e+01 1.76441e-02     13
Backtracking...
88    140 1.89601e-01 3.41517e+01 7.73706e-03     12
Backtracking...
89    142 3.37197e-03 3.41517e+01 9.62912e-03     12
Backtracking...
90    144 2.97847e-01 3.41515e+01 1.64100e-02     11
91    145 1.00000e+00 3.41511e+01 6.18655e-03     12
Backtracking...
Backtracking...
92    148 1.93642e-02 3.41511e+01 5.83796e-03     10
Backtracking...
93    150 8.28119e-02 3.41511e+01 7.42889e-03      9
94    151 1.00000e+00 3.41509e+01 1.07153e-02      9
95    152 1.00000e+00 3.41505e+01 7.41690e-03     11
Backtracking...
Backtracking...
96    155 2.93548e-02 3.41505e+01 7.50382e-03     11
Backtracking...
Backtracking...
97    158 8.68140e-03 3.41504e+01 9.12714e-03     10
Backtracking...
98    160 1.40617e-01 3.41504e+01 7.55935e-03     11
99    161 1.00000e+00 3.41499e+01 1.03130e-02     12
Backtracking...
Backtracking...
Backtracking...
100    165 8.82333e-03 3.41499e+01 1.01387e-02     10
Backtracking...
101    167 1.00000e-03 3.41489e+01 6.75874e-03     11
Backtracking...
Backtracking...
102    170 2.65946e-04 3.41489e+01 7.65777e-03     10
103    171 1.00000e+00 3.41486e+01 6.94239e-03     12
Backtracking...
104    173 1.00000e-03 3.41484e+01 4.77170e-03     12
105    174 1.00000e+00 3.41479e+01 4.86944e-03     12
106    175 1.00000e+00 3.41476e+01 3.78948e-03     12
107    176 1.00000e+00 3.41462e+01 6.57536e-03     12
Backtracking...
108    178 5.87849e-01 3.41448e+01 1.01789e-02     11
Backtracking...
109    180 2.22710e-02 3.41447e+01 1.02974e-02     11
Backtracking...
110    182 1.75794e-02 3.41444e+01 1.50959e-02     10
111    183 1.00000e+00 3.41438e+01 1.17165e-02     13
Backtracking...
Backtracking...
112    186 1.00000e-06 3.41438e+01 9.29967e-03     13
Backtracking...
113    188 1.00000e-03 3.41437e+01 8.84492e-03     13
Backtracking...
114    190 1.89882e-01 3.41437e+01 1.61095e-02     12
Backtracking...
Backtracking...
115    193 1.00000e-06 3.41434e+01 1.21053e-02     13
Backtracking...
Backtracking...
116    196 1.92639e-04 3.41434e+01 7.50063e-03     11
Backtracking...
117    198 4.55396e-02 3.41434e+01 5.90399e-03     12
Backtracking...
Backtracking...
118    201 2.18368e-03 3.41434e+01 5.12003e-03     11
Backtracking...
119    203 3.39055e-01 3.41433e+01 6.91099e-03     11
120    204 1.00000e+00 3.41432e+01 5.68694e-03     11
121    205 1.00000e+00 3.41429e+01 4.05702e-03     11
122    206 1.00000e+00 3.41425e+01 2.49618e-03     10
Backtracking...
Backtracking...
Backtracking...
123    210 7.54404e-06 3.41425e+01 3.39387e-03     11
Backtracking...
124    212 8.71145e-02 3.41424e+01 3.24684e-03     10
Backtracking...
125    214 4.12030e-03 3.41424e+01 3.01374e-03     10
126    215 1.00000e+00 3.41424e+01 4.33100e-03     10
Backtracking...
127    217 3.99388e-02 3.41424e+01 4.80645e-03     10
Backtracking...
Backtracking...
128    220 1.37085e-03 3.41424e+01 4.14702e-03     10
Backtracking...
129    222 7.43227e-02 3.41423e+01 2.64810e-03      9
Backtracking...
130    224 3.96939e-01 3.41422e+01 1.36154e-03      9
131    225 1.00000e+00 3.41422e+01 2.45462e-03     10
Backtracking...
132    227 6.40876e-03 3.41422e+01 1.29458e-03     10
133    228 1.00000e+00 3.41422e+01 6.06514e-04      9
134    229 1.00000e+00 3.41421e+01 1.68009e-03     10
135    230 1.00000e+00 3.41420e+01 2.64087e-03     10
Backtracking...
136    232 4.93517e-01 3.41419e+01 8.76492e-03      9
Backtracking...
137    234 2.96325e-01 3.41418e+01 4.52922e-03     12
Backtracking...
Backtracking...
138    237 7.01401e-03 3.41418e+01 2.71449e-03     11
Backtracking...
Backtracking...
139    240 1.76529e-04 3.41418e+01 3.18220e-03     10
140    241 1.00000e+00 3.41417e+01 1.71625e-03     10
Backtracking...
141    243 3.07791e-01 3.41417e+01 2.92124e-03      9
Backtracking...
142    245 1.41743e-01 3.41416e+01 2.41884e-03      9
Backtracking...
143    247 1.75011e-03 3.41416e+01 2.41427e-03      9
Backtracking...
144    249 1.09712e-03 3.41416e+01 2.40656e-03      9
Backtracking...
145    251 1.02424e-03 3.41416e+01 2.40346e-03      9
Backtracking...
146    253 1.00000e-03 3.41416e+01 2.39969e-03      9
Backtracking...
147    255 1.00000e-03 3.41416e+01 2.39668e-03      9
Backtracking...
148    257 1.00000e-03 3.41416e+01 2.39354e-03      9
Backtracking...
149    259 1.00000e-03 3.41416e+01 2.39055e-03      9
Backtracking...
150    261 1.00000e-03 3.41416e+01 2.38754e-03      9
Backtracking...
151    263 1.00000e-03 3.41416e+01 2.38455e-03      9
Backtracking...
152    265 1.00000e-03 3.41416e+01 2.38156e-03      9
Backtracking...
153    267 1.00000e-03 3.41416e+01 2.37858e-03      9
Backtracking...
154    269 1.00000e-03 3.41416e+01 2.37559e-03      9
Backtracking...
155    271 1.00000e-03 3.41416e+01 2.37260e-03      9
Backtracking...
156    273 1.00000e-03 3.41416e+01 2.36961e-03      9
Backtracking...
157    275 1.00000e-03 3.41416e+01 2.36663e-03      9
Backtracking...
158    277 1.00000e-03 3.41416e+01 2.36364e-03      9
Backtracking...
159    279 1.00000e-03 3.41416e+01 2.36065e-03      9
Backtracking...
160    281 1.00000e-03 3.41416e+01 2.35765e-03      9
Backtracking...
161    283 1.00000e-03 3.41416e+01 2.35466e-03      9
Backtracking...
162    285 1.00000e-03 3.41416e+01 2.35167e-03      9
Backtracking...
163    287 1.00000e-03 3.41416e+01 2.34867e-03      9
Backtracking...
164    289 1.00000e-03 3.41416e+01 2.34568e-03      9
Backtracking...
165    291 1.00000e-03 3.41416e+01 2.34268e-03      9
Backtracking...
166    293 1.00000e-03 3.41416e+01 2.33969e-03      9
Backtracking...
167    295 1.00000e-03 3.41416e+01 2.33669e-03      9
Backtracking...
168    297 1.00000e-03 3.41416e+01 2.33369e-03      9
Backtracking...
169    299 1.00000e-03 3.41416e+01 2.33069e-03      9
Backtracking...
Backtracking...
170    302 6.00000e-04 3.41416e+01 2.32889e-03      9
Backtracking...
Backtracking...
171    305 7.84761e-05 3.41416e+01 2.32866e-03      9
Backtracking...
Backtracking...
172    308 6.65533e-05 3.41416e+01 2.32846e-03      9
Backtracking...
Backtracking...
Backtracking...
173    312 8.75293e-06 3.41416e+01 2.32843e-03      9
Backtracking...
Backtracking...
Backtracking...
174    316 5.64212e-06 3.41416e+01 2.32841e-03      9
Backtracking...
Backtracking...
Backtracking...
175    320 4.52019e-06 3.41416e+01 2.32840e-03      9
Backtracking...
Backtracking...
Backtracking...
176    324 3.87492e-06 3.41416e+01 2.32839e-03      9
Backtracking...
Backtracking...
Backtracking...
Backtracking...
Backtracking...
177    330 9.55608e-08 3.41416e+01 2.32839e-03      9
Progress in parameters or objective below progTol
Generating Plot...
(9 prototypes)
```

## Multinomial Logistic Regression

```% Generate Data
nInstances = 200;
nVars = 10;
nClasses = 5;
X = [ones(nInstances,1) randn(nInstances,nVars-1)];
w = randn(nVars,nClasses-1).*(rand(nVars,nClasses-1)>.5);
[junk y] = max(X*[w zeros(nVars,1)],[],2);

% Initialize Weights and Objective Function
w_init = zeros(nVars,nClasses-1);
w_init = w_init(:);
funObj = @(w)SoftmaxLoss2(w,X,y,nClasses);

% Set up regularizer
lambda = 1*ones(nVars,nClasses-1);
lambda(1,:) = 0; % Don't regularize bias elements
lambda = lambda(:);

% Maximum Likelihood
fprintf('\nComputing Maximum Likelihood Multinomial Logistic Regression Coefficients\n');
mfOptions.Method = 'newton';
wMLR_ML = minFunc(funObj,w_init,mfOptions);
wMLR_ML = reshape(wMLR_ML,nVars,nClasses-1);

% L2-Regularized Logistic Regression
fprintf('\nComputing L2-Regularized Multinomial Logistic Regression Coefficients...\n');
funObjL2 = @(w)penalizedL2(w,funObj,lambda);
wMLR_L2 = minFunc(funObjL2,w_init,mfOptions);
wMLR_L2 = reshape(wMLR_L2,nVars,nClasses-1);

% L1-Regularized Logistic Regression
fprintf('\nComputing L1-Regularized Multinomial Logistic Regression Coefficients...\n');
wMLR_L1 = L1General2_PSSgb(funObj,w_init,lambda);
wMLR_L1 = reshape(wMLR_L1,nVars,nClasses-1);

% Elastic Net Logistic Regression
fprintf('\nComputing Elastic-Net Multinomial Logistic Regression Coefficients...\n');
wMLR_L1L2 = L1General2_PSSgb(funObjL2,w_init,lambda);
wMLR_L1L2 = reshape(wMLR_L1L2,nVars,nClasses-1);

% Report Number of non-zeros
fprintf('Number of Features Selected by Maximum Likelihood Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_ML(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by L2-regualrized Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L2(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by L1-regualrized Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L1(2:end,:)),(nVars-1)*(nClasses-1));
fprintf('Number of Features Selected by Elastic-Net Multinomial Logistic classifier: %d (out of %d)\n',nnz(wMLR_L1L2(2:end,:)),(nVars-1)*(nClasses-1));

% Show Stem Plots
figure;
clf;hold on;
subplot(2,2,1);
stem(wMLR_ML(:),'r');
title('Maximum Likelihood Multinomial Logistic');
subplot(2,2,2);
stem(wMLR_L2(:),'b');
title('L2-Regularized Multinomial Logistic');
subplot(2,2,3);
stem(wMLR_L1(:),'g');
title('L1-Regularized Multinomial Logistic');
subplot(2,2,4);
stem(wMLR_L1L2(:),'c');
title('Elastic-Net Multinomial Logistic');

% Compute training error
[junk yhat] = max(X*[wMLR_L1 zeros(nVars,1)],[],2);
fprintf('Classification error rate on training data for L1-regularied Multinomial Logistic: %.2f\n',sum(y ~= yhat)/length(y));
pause;
```
```Computing Maximum Likelihood Multinomial Logistic Regression Coefficients
Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          2     1.00000e+00     1.04179e+02     1.17219e+01
2          3     1.00000e+00     6.22764e+01     5.13364e+00
3          4     1.00000e+00     3.91936e+01     2.21929e+00
4          5     1.00000e+00     2.42730e+01     1.13250e+00
5          6     1.00000e+00     1.46139e+01     6.12513e-01
6          7     1.00000e+00     7.94355e+00     2.90802e-01
7          8     1.00000e+00     3.59489e+00     1.22797e-01
8          9     1.00000e+00     1.38213e+00     4.79815e-02
9         10     1.00000e+00     5.10257e-01     1.80193e-02
10         11     1.00000e+00     1.88486e-01     6.82512e-03
11         12     1.00000e+00     6.96973e-02     2.55603e-03
12         13     1.00000e+00     2.57607e-02     9.51627e-04
13         14     1.00000e+00     9.51241e-03     3.52916e-04
14         15     1.00000e+00     3.50930e-03     1.30531e-04
15         16     1.00000e+00     1.29366e-03     4.81926e-05
16         17     1.00000e+00     4.76609e-04     1.77716e-05
17         18     1.00000e+00     1.75517e-04     6.54829e-06
Optimality Condition below optTol

Computing L2-Regularized Multinomial Logistic Regression Coefficients...
Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          2     1.00000e+00     1.20545e+02     1.02673e+01
2          3     1.00000e+00     9.92603e+01     2.54050e+00
3          4     1.00000e+00     9.63988e+01     4.25870e-01
4          5     1.00000e+00     9.63028e+01     2.67172e-02
5          6     1.00000e+00     9.63026e+01     1.06566e-04
6          7     1.00000e+00     9.63026e+01     1.55572e-09
Optimality Condition below optTol

Computing L1-Regularized Multinomial Logistic Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 1.76103e-03 2.94845e+02 5.21182e+01     37
2      3 1.00000e+00 1.35657e+02 1.26400e+01     35
3      4 1.00000e+00 1.16746e+02 7.63398e+00     37
4      5 1.00000e+00 9.17691e+01 6.58194e+00     35
5      6 1.00000e+00 8.30630e+01 7.05694e+00     35
6      7 1.00000e+00 7.90554e+01 3.16541e+00     36
7      8 1.00000e+00 7.73113e+01 1.38246e+00     32
8      9 1.00000e+00 7.60217e+01 1.01551e+00     32
9     10 1.00000e+00 7.53268e+01 1.12175e+00     30
10     11 1.00000e+00 7.49766e+01 1.13828e+00     30
11     12 1.00000e+00 7.47120e+01 4.41812e-01     29
12     13 1.00000e+00 7.45730e+01 4.17875e-01     29
13     14 1.00000e+00 7.44940e+01 3.95614e-01     29
14     15 1.00000e+00 7.44419e+01 3.31846e-01     29
15     16 1.00000e+00 7.44120e+01 1.84727e-01     28
16     17 1.00000e+00 7.44051e+01 8.31556e-02     29
17     18 1.00000e+00 7.44018e+01 4.36642e-02     27
18     19 1.00000e+00 7.43996e+01 3.15859e-02     27
19     20 1.00000e+00 7.43993e+01 3.41207e-02     27
20     21 1.00000e+00 7.43991e+01 1.57060e-02     27
21     22 1.00000e+00 7.43990e+01 1.24550e-02     27
22     23 1.00000e+00 7.43989e+01 8.23950e-03     27
23     24 1.00000e+00 7.43989e+01 1.04371e-02     27
24     25 1.00000e+00 7.43989e+01 2.68026e-03     27
25     26 1.00000e+00 7.43989e+01 1.90409e-03     27
26     27 1.00000e+00 7.43989e+01 1.70540e-03     27
27     28 1.00000e+00 7.43988e+01 9.66254e-04     27
28     29 1.00000e+00 7.43988e+01 1.11360e-03     27
29     30 1.00000e+00 7.43988e+01 2.63725e-04     27
30     31 1.00000e+00 7.43988e+01 1.19382e-04     27
31     32 1.00000e+00 7.43988e+01 3.08955e-05     27
32     33 1.00000e+00 7.43988e+01 2.96448e-05     27
Progress in parameters or objective below progTol

Computing Elastic-Net Multinomial Logistic Regression Coefficients...
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 1.76103e-03 2.94888e+02 5.19179e+01     37
2      3 1.00000e+00 1.44710e+02 1.13796e+01     35
3      4 1.00000e+00 1.31208e+02 6.10043e+00     37
4      5 1.00000e+00 1.20146e+02 4.62499e+00     35
5      6 1.00000e+00 1.18592e+02 4.70310e+00     38
6      7 1.00000e+00 1.17639e+02 8.48525e-01     36
7      8 1.00000e+00 1.17521e+02 6.80493e-01     37
8      9 1.00000e+00 1.17378e+02 5.87050e-01     33
9     10 1.00000e+00 1.17330e+02 1.68602e-01     31
10     11 1.00000e+00 1.17324e+02 1.67430e-01     32
11     12 1.00000e+00 1.17321e+02 7.02801e-02     32
12     13 1.00000e+00 1.17320e+02 5.55746e-02     32
13     14 1.00000e+00 1.17320e+02 3.14440e-02     32
14     15 1.00000e+00 1.17320e+02 3.21882e-02     32
15     16 1.00000e+00 1.17319e+02 1.95754e-02     32
Backtracking...
16     18 3.59348e-01 1.17319e+02 1.42054e-02     32
17     19 1.00000e+00 1.17319e+02 9.01286e-03     32
18     20 1.00000e+00 1.17319e+02 4.78039e-03     32
19     21 1.00000e+00 1.17319e+02 3.84352e-03     32
Backtracking...
20     23 4.22797e-01 1.17319e+02 3.22387e-03     32
21     24 1.00000e+00 1.17319e+02 7.25821e-04     32
22     25 1.00000e+00 1.17319e+02 4.46354e-04     32
23     26 1.00000e+00 1.17319e+02 2.49745e-04     32
Backtracking...
24     28 4.83418e-01 1.17319e+02 1.18676e-04     32
25     29 1.00000e+00 1.17319e+02 3.63166e-05     32
Directional derivative below progTol
Number of Features Selected by Maximum Likelihood Multinomial Logistic classifier: 36 (out of 36)
Number of Features Selected by L2-regualrized Multinomial Logistic classifier: 36 (out of 36)
Number of Features Selected by L1-regualrized Multinomial Logistic classifier: 23 (out of 36)
Number of Features Selected by Elastic-Net Multinomial Logistic classifier: 28 (out of 36)
Classification error rate on training data for L1-regularied Multinomial Logistic: 0.03
```

## Compressed Sensing

```figure;
nVars = 256;
nNonZero = 5;
x = zeros(nVars,1);
x(ceil(rand(nNonZero,1)*nVars)) = 1;

subplot(2,2,1);
imagesc(reshape(x,[16 16]));
colormap gray
title(sprintf('Original Signal (%d elements, %d nonZero)',nVars,nNonZero));

nMeasurements = 49;
phi = rand(nMeasurements,nVars);

subplot(2,2,2);
imagesc(phi);
colormap gray
title('Measurement Matrix');

y = phi*x;

subplot(2,2,3);
imagesc(reshape(y,[7 7]));
colormap gray
title(sprintf('Measurements (%d elements)',nMeasurements));

funObj = @(f)SquaredError(f,phi,y);
f = zeros(nVars,1);
options = [];
for lambda = 10.^[1:-1:-2]
f = L1General2_PSSgb(funObj,f,lambda*ones(nVars,1),options);
residual = norm(phi*f-y)
reconstructionError = norm(f-x)

subplot(2,2,4);
imagesc(reshape(f,[16 16]));
colormap gray
title(sprintf('Reconstructed (lambda = %.2f, error = %.2f)',lambda,reconstructionError));
pause;
end
```
```  Iter fEvals      stepLen         fVal      optCond    nnz
1      2 3.35502e-05 2.49414e+02 1.21044e+02    256
2      3 1.00000e+00 7.50875e+01 1.22691e+01    256
3      4 1.00000e+00 7.45935e+01 1.20244e+01    256
Backtracking...
4      6 5.58838e-02 7.18038e+01 1.13193e+01    256
Backtracking...
5      8 3.90337e-02 6.98779e+01 1.07825e+01    254
Backtracking...
6     10 3.18286e-02 6.85003e+01 9.87135e+00    248
Backtracking...
7     12 3.66454e-02 6.72506e+01 9.01928e+00    243
Backtracking...
8     14 3.35227e-02 6.62559e+01 8.05011e+00    234
Backtracking...
9     16 5.31147e-02 6.52563e+01 7.07645e+00    224
Backtracking...
10     18 9.72282e-02 6.42318e+01 8.19072e+00    215
Backtracking...
11     20 1.44341e-01 6.33824e+01 9.56689e+00    200
Backtracking...
12     22 3.36791e-01 6.28361e+01 1.21604e+01    181
13     23 1.00000e+00 6.18708e+01 1.13793e+01    171
Backtracking...
14     25 2.96432e-02 6.13845e+01 1.14261e+01    165
Backtracking...
15     27 4.21529e-02 6.08119e+01 1.13936e+01    156
Backtracking...
16     29 4.63575e-02 6.02913e+01 1.13218e+01    146
Backtracking...
17     31 6.34011e-02 5.96866e+01 1.07890e+01    137
Backtracking...
18     33 7.40129e-02 5.88998e+01 1.01510e+01    128
Backtracking...
19     35 1.75350e-01 5.77169e+01 1.01594e+01    112
20     36 1.00000e+00 5.58082e+01 1.38145e+01     64
Backtracking...
21     38 4.43738e-01 5.40059e+01 1.16370e+01     44
Backtracking...
22     40 3.46341e-01 5.31614e+01 5.14224e+00    253
Backtracking...
23     42 1.82283e-01 5.29936e+01 4.57903e+00    200
Backtracking...
24     44 3.57707e-01 5.28811e+01 5.57518e+00    147
25     45 1.00000e+00 5.28628e+01 7.59500e+00     90
26     46 1.00000e+00 5.23126e+01 2.68277e+00     70
27     47 1.00000e+00 5.21133e+01 3.69019e+00     53
28     48 1.00000e+00 5.20151e+01 4.82135e+00     41
29     49 1.00000e+00 5.17088e+01 3.80668e+00     37
Backtracking...
30     51 2.63284e-01 5.13077e+01 4.90943e+00     32
31     52 1.00000e+00 5.08179e+01 5.47259e+00     25
32     53 1.00000e+00 5.02276e+01 4.54655e+00     21
Backtracking...
33     55 2.74455e-02 5.00975e+01 4.24341e+00    166
Backtracking...
34     57 4.68518e-02 4.99615e+01 3.75771e+00     83
Backtracking...
35     59 2.49336e-01 4.95742e+01 2.71343e+00     98
Backtracking...
36     61 5.12947e-02 4.95548e+01 3.20811e+00     47
Backtracking...
37     63 4.22201e-03 4.95288e+01 3.15771e+00     46
Backtracking...
38     65 2.25267e-01 4.93976e+01 3.95104e+00     38
39     66 1.00000e+00 4.93048e+01 6.97752e+00     20
40     67 1.00000e+00 4.90130e+01 4.07106e+00     19
Backtracking...
41     69 2.96373e-01 4.89290e+01 2.51626e+00    155
Backtracking...
42     71 1.77165e-02 4.89241e+01 2.65326e+00    144
Backtracking...
43     73 8.54997e-02 4.89215e+01 2.88308e+00    132
Backtracking...
Backtracking...
Backtracking...
Backtracking...
Backtracking...
44     79 5.88547e-04 4.89215e+01 2.88672e+00    131
Backtracking...
45     81 4.24407e-01 4.88942e+01 3.00351e+00     81
Backtracking...
46     83 2.89523e-01 4.88677e+01 3.54518e+00     68
Backtracking...
47     85 1.96644e-01 4.88420e+01 3.66921e+00     54
Backtracking...
48     87 3.08337e-01 4.88323e+01 3.99057e+00     46
49     88 1.00000e+00 4.86986e+01 3.53232e+00     26
Backtracking...
50     90 5.60421e-01 4.85536e+01 3.97612e+00     21
Backtracking...
51     92 1.69999e-01 4.85064e+01 2.92795e+00     20
52     93 1.00000e+00 4.84441e+01 2.74973e+00     20
Backtracking...
53     95 2.33193e-01 4.82388e+01 3.14110e+00     22
Backtracking...
54     97 5.33995e-01 4.81137e+01 3.96201e+00     16
Backtracking...
55     99 1.10694e-02 4.80970e+01 3.84399e+00     15
Backtracking...
56    101 3.54207e-02 4.80793e+01 4.02648e+00     15
Backtracking...
57    103 2.79191e-01 4.80710e+01 3.57274e+00     13
58    104 1.00000e+00 4.80036e+01 1.39523e+00     13
59    105 1.00000e+00 4.79970e+01 2.39056e+00     31
Backtracking...
60    107 2.61682e-01 4.79932e+01 2.41536e+00     26
Backtracking...
61    109 3.66070e-01 4.79906e+01 2.22905e+00     24
62    110 1.00000e+00 4.79802e+01 2.18098e+00     22
63    111 1.00000e+00 4.79649e+01 2.52345e+00     17
64    112 1.00000e+00 4.79286e+01 2.00820e+00     15
Backtracking...
65    114 1.00000e-03 4.79280e+01 2.00336e+00     15
Backtracking...
66    116 4.84845e-01 4.78797e+01 1.79993e+00     12
Backtracking...
Backtracking...
67    119 8.01789e-02 4.78755e+01 1.47826e+00     12
68    120 1.00000e+00 4.78534e+01 1.70749e+00     13
Backtracking...
69    122 2.16452e-01 4.78455e+01 1.38289e+00     12
70    123 1.00000e+00 4.78335e+01 2.19775e+00     18
Backtracking...
71    125 1.21312e-02 4.78286e+01 2.18760e+00     17
72    126 1.00000e+00 4.78151e+01 1.58356e+00     16
Backtracking...
73    128 4.16636e-01 4.78139e+01 2.62897e-01     17
74    129 1.00000e+00 4.78139e+01 4.58304e-01     15
75    130 1.00000e+00 4.78085e+01 2.25101e-01     15
76    131 1.00000e+00 4.78077e+01 2.89047e-01     17
77    132 1.00000e+00 4.78050e+01 1.02011e-01     16
78    133 1.00000e+00 4.78032e+01 1.17981e-01     16
Backtracking...
79    135 3.88570e-01 4.78023e+01 1.10667e-01     15
Backtracking...
80    137 3.96675e-01 4.78014e+01 2.01171e-01     15
Backtracking...
81    139 4.24047e-01 4.78003e+01 6.48385e-02     15
82    140 1.00000e+00 4.77999e+01 4.88097e-02     15
83    141 1.00000e+00 4.77998e+01 6.40268e-02     15
84    142 1.00000e+00 4.77991e+01 4.83560e-02     15
Backtracking...
85    144 3.71882e-01 4.77990e+01 3.98244e-02     15
86    145 1.00000e+00 4.77988e+01 2.93461e-02     15
87    146 1.00000e+00 4.77987e+01 2.24206e-02     15
88    147 1.00000e+00 4.77987e+01 1.82457e-02     15
89    148 1.00000e+00 4.77987e+01 1.31130e-02     15
90    149 1.00000e+00 4.77987e+01 1.45262e-02     15
91    150 1.00000e+00 4.77987e+01 1.41998e-02     15
92    151 1.00000e+00 4.77987e+01 1.17305e-02     15
93    152 1.00000e+00 4.77987e+01 7.98338e-03     15
94    153 1.00000e+00 4.77987e+01 5.00422e-03     15
95    154 1.00000e+00 4.77987e+01 3.40224e-03     15
96    155 1.00000e+00 4.77987e+01 1.98160e-03     15
97    156 1.00000e+00 4.77987e+01 1.05096e-03     15
98    157 1.00000e+00 4.77987e+01 5.64512e-04     15
99    158 1.00000e+00 4.77987e+01 3.50266e-04     15
100    159 1.00000e+00 4.77987e+01 1.75665e-04     15
101    160 1.00000e+00 4.77987e+01 7.10882e-05     15
102    161 1.00000e+00 4.77987e+01 1.34542e-05     15
Directional derivative below progTol

residual =

1.4837

reconstructionError =

0.5158

Iter fEvals      stepLen         fVal      optCond    nnz
Backtracking...
1      3 1.60236e-04 5.68849e+00 2.34637e+00    256
2      4 1.00000e+00 5.67463e+00 2.32625e+00    256
Backtracking...
3      6 2.25231e-02 5.63884e+00 2.27424e+00    256
Backtracking...
4      8 1.82919e-02 5.61157e+00 2.19021e+00    243
Backtracking...
5     10 2.50015e-02 5.59090e+00 2.02519e+00    230
Backtracking...
6     12 6.56450e-02 5.56967e+00 1.79559e+00    210
Backtracking...
7     14 1.50409e-01 5.55205e+00 1.43738e+00    185
8     15 1.00000e+00 5.53844e+00 2.08636e+00    155
9     16 1.00000e+00 5.51437e+00 1.35597e+00    213
10     17 1.00000e+00 5.49252e+00 1.41764e+00    176
Backtracking...
11     19 3.68872e-02 5.48114e+00 1.33078e+00    161
Backtracking...
12     21 1.06121e-01 5.46949e+00 1.11425e+00    147
Backtracking...
13     23 2.31599e-01 5.45976e+00 1.30507e+00    128
14     24 1.00000e+00 5.44864e+00 1.69869e+00    108
15     25 1.00000e+00 5.42534e+00 1.81223e+00    104
Backtracking...
16     27 2.64930e-01 5.40175e+00 1.95337e+00    103
17     28 1.00000e+00 5.36302e+00 1.93737e+00     98
18     29 1.00000e+00 5.30890e+00 1.12735e+00     84
Backtracking...
19     31 1.48389e-01 5.29321e+00 9.68236e-01     73
20     32 1.00000e+00 5.26064e+00 9.60619e-01     57
Backtracking...
Backtracking...
21     35 1.84666e-02 5.25926e+00 1.51982e+00     55
22     36 1.00000e+00 5.23156e+00 1.45435e+00     46
Backtracking...
Backtracking...
23     39 6.37792e-02 5.19768e+00 2.08783e+00     27
Backtracking...
24     41 2.63467e-01 5.19192e+00 1.53348e+00    230
25     42 1.00000e+00 5.18808e+00 1.36498e+00    171
26     43 1.00000e+00 5.18357e+00 1.25639e+00    126
27     44 1.00000e+00 5.18325e+00 1.53692e+00     92
28     45 1.00000e+00 5.16997e+00 1.00934e+00     87
29     46 1.00000e+00 5.16124e+00 1.25961e+00     72
30     47 1.00000e+00 5.13198e+00 1.09025e+00     37
Backtracking...
31     49 4.13547e-02 5.13057e+00 1.26071e+00     35
32     50 1.00000e+00 5.11300e+00 1.17742e+00     29
Backtracking...
33     52 2.80916e-01 5.07539e+00 1.84497e+00     21
Backtracking...
34     54 5.23250e-02 5.07332e+00 1.82016e+00     25
Backtracking...
35     56 1.46962e-02 5.04210e+00 9.63966e-01     19
36     57 1.00000e+00 5.02598e+00 4.11094e-01     17
37     58 1.00000e+00 5.02203e+00 6.04572e-01     14
Backtracking...
38     60 3.86740e-01 5.01287e+00 8.12598e-01     14
39     61 1.00000e+00 5.01036e+00 1.07048e+00     10
Backtracking...
40     63 2.48744e-01 4.99471e+00 3.52853e-01     10
41     64 1.00000e+00 4.99341e+00 8.89207e-01     95
Backtracking...
42     66 2.54049e-01 4.99044e+00 7.83691e-01     71
Backtracking...
43     68 1.98328e-01 4.98749e+00 4.74051e-01     65
Backtracking...
Backtracking...
44     71 8.60715e-02 4.98729e+00 5.06820e-01     56
45     72 1.00000e+00 4.98350e+00 3.42037e-01     22
Backtracking...
Backtracking...
46     75 4.72320e-02 4.98338e+00 3.28807e-01     18
Backtracking...
Backtracking...
47     78 7.07891e-02 4.98323e+00 2.25100e-01     17
Backtracking...
48     80 4.18919e-01 4.98293e+00 1.94093e-01     13
49     81 1.00000e+00 4.98238e+00 1.17359e-01     11
50     82 1.00000e+00 4.98198e+00 1.29208e-01     13
51     83 1.00000e+00 4.98051e+00 1.25856e-01     10
52     84 1.00000e+00 4.97900e+00 1.29891e-01      9
53     85 1.00000e+00 4.97872e+00 1.88020e-01      8
Backtracking...
54     87 4.30720e-03 4.97869e+00 1.87528e-01      8
55     88 1.00000e+00 4.97835e+00 4.69479e-02      8
56     89 1.00000e+00 4.97827e+00 3.12822e-02      8
57     90 1.00000e+00 4.97821e+00 2.65621e-02     13
58     91 1.00000e+00 4.97819e+00 6.62817e-02     14
Backtracking...
Backtracking...
59     94 5.17236e-04 4.97818e+00 6.93045e-02     14
Backtracking...
60     96 3.43483e-01 4.97814e+00 4.80769e-02     13
61     97 1.00000e+00 4.97811e+00 2.36317e-02     12
62     98 1.00000e+00 4.97811e+00 4.64090e-02     15
Backtracking...
Backtracking...
63    101 5.21036e-03 4.97811e+00 4.71775e-02     13
Backtracking...
Backtracking...
64    104 4.48009e-04 4.97811e+00 4.74027e-02     13
Backtracking...
65    106 8.98018e-02 4.97810e+00 3.76276e-02     13
Backtracking...
66    108 2.87381e-02 4.97808e+00 3.60121e-02     12
67    109 1.00000e+00 4.97804e+00 2.25433e-02     12
68    110 1.00000e+00 4.97799e+00 6.35032e-03     13
Backtracking...
69    112 1.72141e-01 4.97799e+00 6.71866e-03     12
70    113 1.00000e+00 4.97799e+00 5.32645e-03     15
Backtracking...
71    115 8.01806e-02 4.97799e+00 6.22308e-03     14
Backtracking...
72    117 3.96239e-01 4.97799e+00 9.08961e-03     14
73    118 1.00000e+00 4.97799e+00 5.18465e-03     14
74    119 1.00000e+00 4.97799e+00 4.70484e-03     15
75    120 1.00000e+00 4.97799e+00 1.20413e-02     15
Backtracking...
76    122 4.91569e-01 4.97799e+00 5.21229e-03     15
77    123 1.00000e+00 4.97799e+00 1.72499e-03     15
78    124 1.00000e+00 4.97799e+00 1.38505e-03     15
79    125 1.00000e+00 4.97799e+00 5.78246e-04     15
80    126 1.00000e+00 4.97799e+00 2.71256e-04     15
81    127 1.00000e+00 4.97799e+00 2.37731e-04     15
82    128 1.00000e+00 4.97799e+00 1.53940e-04     15
83    129 1.00000e+00 4.97799e+00 4.19501e-05     15
Directional derivative below progTol

residual =

0.1484

reconstructionError =

0.0516

Iter fEvals      stepLen         fVal      optCond    nnz
Backtracking...
1      3 1.60236e-04 5.06885e-01 2.34663e-01    256
2      4 1.00000e+00 5.06747e-01 2.32651e-01    256
Backtracking...
3      6 2.25218e-02 5.06389e-01 2.27449e-01    256
Backtracking...
4      8 1.82910e-02 5.06116e-01 2.19045e-01    243
Backtracking...
5     10 2.50012e-02 5.05909e-01 2.02543e-01    230
Backtracking...
6     12 6.56438e-02 5.05697e-01 1.79583e-01    210
Backtracking...
7     14 1.50413e-01 5.05521e-01 1.43745e-01    185
8     15 1.00000e+00 5.05385e-01 2.08634e-01    155
9     16 1.00000e+00 5.05144e-01 1.35624e-01    213
10     17 1.00000e+00 5.04926e-01 1.41787e-01    176
Backtracking...
11     19 3.68734e-02 5.04812e-01 1.33108e-01    161
Backtracking...
12     21 1.06018e-01 5.04695e-01 1.11461e-01    147
Backtracking...
13     23 2.31630e-01 5.04598e-01 1.30482e-01    128
14     24 1.00000e+00 5.04487e-01 1.69865e-01    108
15     25 1.00000e+00 5.04254e-01 1.81175e-01    104
Backtracking...
16     27 2.64380e-01 5.04018e-01 1.95298e-01    103
17     28 1.00000e+00 5.03631e-01 1.93785e-01     98
18     29 1.00000e+00 5.03090e-01 1.12675e-01     84
Backtracking...
19     31 1.49860e-01 5.02932e-01 9.69812e-02     73
20     32 1.00000e+00 5.02606e-01 9.59290e-02     57
Backtracking...
Backtracking...
21     35 1.81477e-02 5.02589e-01 1.48497e-01     55
22     36 1.00000e+00 5.02312e-01 1.45984e-01     46
Backtracking...
Backtracking...
23     39 8.50074e-02 5.02006e-01 2.19799e-01     27
Backtracking...
24     41 2.42453e-01 5.01938e-01 1.58443e-01    248
25     42 1.00000e+00 5.01897e-01 1.27765e-01    172
26     43 1.00000e+00 5.01855e-01 1.18496e-01    125
Backtracking...
27     45 3.09388e-01 5.01819e-01 1.06252e-01     97
28     46 1.00000e+00 5.01758e-01 1.13786e-01     72
Backtracking...
29     48 3.14004e-01 5.01709e-01 1.49059e-01     43
30     49 1.00000e+00 5.01571e-01 1.05665e-01     40
31     50 1.00000e+00 5.01059e-01 2.32989e-01     24
Backtracking...
32     52 1.55235e-01 5.00981e-01 1.55188e-01    155
Backtracking...
33     54 1.30240e-01 5.00876e-01 1.12352e-01    128
Backtracking...
34     56 6.52915e-03 5.00872e-01 1.12149e-01    114
Backtracking...
35     58 4.02910e-01 5.00750e-01 1.57582e-01     45
Backtracking...
36     60 2.11957e-03 5.00747e-01 1.55973e-01     44
Backtracking...
37     62 1.95066e-01 5.00675e-01 1.38185e-01     26
Backtracking...
38     64 1.91051e-03 5.00664e-01 1.40163e-01     26
Backtracking...
39     66 1.42964e-01 5.00432e-01 1.63985e-01     23
Backtracking...
Backtracking...
40     69 1.39894e-01 5.00275e-01 6.71169e-02     18
Backtracking...
Backtracking...
41     72 1.55481e-04 5.00259e-01 5.90533e-02     78
Backtracking...
Backtracking...
42     75 1.77622e-05 5.00243e-01 4.83415e-02     49
Backtracking...
43     77 3.47796e-02 5.00232e-01 3.59950e-02     37
Backtracking...
44     79 4.07696e-01 5.00154e-01 4.49068e-02     30
Backtracking...
45     81 1.00000e-03 5.00146e-01 4.15653e-02     30
46     82 1.00000e+00 5.00122e-01 8.86935e-02     21
Backtracking...
Backtracking...
47     85 1.06394e-06 5.00119e-01 8.87844e-02     21
Backtracking...
48     87 2.58230e-01 5.00042e-01 7.43836e-02     19
Backtracking...
Backtracking...
49     90 3.44717e-04 5.00039e-01 7.22945e-02     19
Backtracking...
50     92 4.86071e-01 4.99980e-01 5.27179e-02     18
Backtracking...
51     94 1.75207e-01 4.99958e-01 4.10484e-02     17
Backtracking...
Backtracking...
52     97 2.51529e-04 4.99956e-01 3.83362e-02     17
Backtracking...
53     99 2.99588e-01 4.99923e-01 4.59696e-02     16
54    100 1.00000e+00 4.99882e-01 3.31557e-02     13
Backtracking...
Backtracking...
55    103 8.47893e-05 4.99879e-01 3.47154e-02     13
Backtracking...
56    105 3.64022e-01 4.99860e-01 2.26680e-02     13
57    106 1.00000e+00 4.99822e-01 4.35095e-02     43
Backtracking...
58    108 8.33157e-03 4.99821e-01 4.41126e-02     39
Backtracking...
Backtracking...
59    111 1.78770e-03 4.99821e-01 4.40774e-02     38
Backtracking...
Backtracking...
60    114 1.63657e-02 4.99820e-01 4.33328e-02     33
Backtracking...
61    116 1.00000e-03 4.99819e-01 4.19340e-02     30
Backtracking...
62    118 2.45278e-03 4.99818e-01 4.15687e-02     29
Backtracking...
63    120 2.68086e-01 4.99817e-01 4.30702e-02     23
Backtracking...
64    122 4.58076e-03 4.99816e-01 4.25081e-02     22
65    123 1.00000e+00 4.99808e-01 1.02988e-02     14
Backtracking...
66    125 1.00000e-03 4.99808e-01 1.08631e-02     23
Backtracking...
Backtracking...
67    128 2.78058e-05 4.99807e-01 1.34438e-02     21
Backtracking...
68    130 5.91947e-02 4.99806e-01 1.70758e-02     16
Backtracking...
Backtracking...
Backtracking...
69    134 1.12997e-05 4.99806e-01 1.75139e-02     14
Backtracking...
Backtracking...
Backtracking...
Backtracking...
70    139 6.92373e-08 4.99806e-01 9.47738e-03     11
Backtracking...
Backtracking...
Backtracking...
71    143 6.71580e-08 4.99805e-01 1.01570e-02     17
Backtracking...
Backtracking...
72    146 1.03015e-05 4.99805e-01 1.09567e-02     14
Backtracking...
Backtracking...
73    149 1.27424e-06 4.99805e-01 1.09161e-02     16
Backtracking...
Backtracking...
74    152 4.20603e-04 4.99805e-01 1.29725e-02     14
Backtracking...
75    154 2.71789e-01 4.99801e-01 1.65521e-02     12
Backtracking...
76    156 3.59669e-01 4.99790e-01 1.49572e-02     11
Backtracking...
77    158 1.00000e-03 4.99790e-01 1.46746e-02     11
Backtracking...
78    160 2.04111e-01 4.99788e-01 1.18333e-02     11
Backtracking...
79    162 4.47321e-01 4.99784e-01 5.64353e-03     11
80    163 1.00000e+00 4.99782e-01 7.52085e-03     12
Backtracking...
81    165 1.17572e-02 4.99782e-01 6.50864e-03     20
Backtracking...
82    167 1.81064e-03 4.99782e-01 5.86296e-03     15
83    168 1.00000e+00 4.99781e-01 3.70905e-03     17
Backtracking...
84    170 1.00000e-03 4.99781e-01 3.10069e-03     16
Backtracking...
85    172 6.34585e-02 4.99781e-01 1.41734e-03     14
86    173 1.00000e+00 4.99780e-01 2.46138e-03     14
Backtracking...
Backtracking...
87    176 2.12295e-03 4.99780e-01 1.41918e-03     14
88    177 1.00000e+00 4.99780e-01 1.25534e-03     14
89    178 1.00000e+00 4.99780e-01 3.82409e-03     15
Backtracking...
90    180 3.79342e-02 4.99780e-01 3.33626e-03     15
Backtracking...
91    182 5.00220e-03 4.99780e-01 3.38568e-03     15
Backtracking...
92    184 3.86838e-01 4.99780e-01 2.71508e-03     15
93    185 1.00000e+00 4.99780e-01 1.49377e-03     17
Backtracking...
Backtracking...
94    188 4.42474e-02 4.99780e-01 1.45697e-03     16
Backtracking...
95    190 5.95538e-03 4.99780e-01 1.48054e-03     16
96    191 1.00000e+00 4.99780e-01 1.78872e-03     15
97    192 1.00000e+00 4.99780e-01 6.27976e-04     15
98    193 1.00000e+00 4.99780e-01 8.24886e-04     15
99    194 1.00000e+00 4.99780e-01 6.87867e-04     15
100    195 1.00000e+00 4.99780e-01 3.09047e-04     15
101    196 1.00000e+00 4.99780e-01 1.95504e-04     15
102    197 1.00000e+00 4.99780e-01 2.07888e-04     15
103    198 1.00000e+00 4.99780e-01 1.90420e-04     15
104    199 1.00000e+00 4.99780e-01 1.05667e-04     15
105    200 1.00000e+00 4.99780e-01 6.84437e-05     15
Directional derivative below progTol

residual =

0.0148

reconstructionError =

0.0052

Iter fEvals      stepLen         fVal      optCond    nnz
Backtracking...
1      3 1.60237e-04 5.00688e-02 2.34886e-02    256
2      4 1.00000e+00 5.00675e-02 2.32872e-02    256
Backtracking...
3      6 2.25313e-02 5.00639e-02 2.27664e-02    256
Backtracking...
4      8 1.82917e-02 5.00612e-02 2.19247e-02    243
Backtracking...
5     10 2.50170e-02 5.00591e-02 2.02739e-02    230
Backtracking...
6     12 6.56545e-02 5.00570e-02 1.79802e-02    210
Backtracking...
7     14 1.50108e-01 5.00552e-02 1.43878e-02    185
8     15 1.00000e+00 5.00538e-02 2.08023e-02    155
9     16 1.00000e+00 5.00514e-02 1.35429e-02    213
10     17 1.00000e+00 5.00492e-02 1.42249e-02    176
Backtracking...
11     19 3.60938e-02 5.00481e-02 1.33511e-02    161
Backtracking...
12     21 1.05507e-01 5.00470e-02 1.12078e-02    146
Backtracking...
13     23 2.23302e-01 5.00460e-02 1.28397e-02    128
14     24 1.00000e+00 5.00449e-02 1.68841e-02    109
15     25 1.00000e+00 5.00424e-02 1.71820e-02    105
Backtracking...
16     27 1.76697e-01 5.00400e-02 1.79058e-02     94
17     28 1.00000e+00 5.00380e-02 2.48689e-02     77
Backtracking...
18     30 4.05591e-01 5.00354e-02 1.15970e-02    204
19     31 1.00000e+00 5.00318e-02 1.51273e-02    112
Backtracking...
20     33 2.12182e-01 5.00312e-02 2.04940e-02     70
21     34 1.00000e+00 5.00287e-02 1.81159e-02     92
Backtracking...
22     36 3.66422e-01 5.00237e-02 1.19228e-02     42
23     37 1.00000e+00 5.00200e-02 1.10827e-02     31
24     38 1.00000e+00 5.00160e-02 1.29873e-02     22
Backtracking...
Backtracking...
25     41 2.57354e-01 5.00155e-02 2.46521e-02     20
Backtracking...
26     43 3.44045e-01 5.00133e-02 1.51349e-02    254
27     44 1.00000e+00 5.00123e-02 1.12316e-02    197
28     45 1.00000e+00 5.00113e-02 1.30541e-02    134
29     46 1.00000e+00 5.00106e-02 1.60556e-02     80
Backtracking...
30     48 3.14852e-01 5.00095e-02 1.23390e-02     67
31     49 1.00000e+00 5.00089e-02 1.24535e-02     66
32     50 1.00000e+00 5.00088e-02 1.48105e-02     46
33     51 1.00000e+00 5.00065e-02 7.97435e-03     42
Backtracking...
34     53 3.30928e-01 5.00061e-02 8.89595e-03     35
Backtracking...
35     55 1.86199e-01 5.00059e-02 8.23941e-03     32
Backtracking...
Backtracking...
36     58 1.00000e-06 5.00057e-02 6.43879e-03     32
37     59 1.00000e+00 5.00040e-02 6.64221e-03     29
Backtracking...
38     61 4.64992e-03 5.00039e-02 6.50540e-03     29
Backtracking...
39     63 2.89876e-01 5.00031e-02 8.43046e-03     26
Backtracking...
40     65 1.52717e-02 5.00025e-02 7.55624e-03     25
41     66 1.00000e+00 5.00008e-02 6.59390e-03     22
42     67 1.00000e+00 5.00003e-02 4.99323e-03     21
Backtracking...
43     69 4.18505e-01 5.00000e-02 3.26197e-03    150
Backtracking...
44     71 1.81638e-01 5.00000e-02 4.00789e-03    126
Backtracking...
45     73 4.36390e-03 5.00000e-02 3.96882e-03    124
Backtracking...
Backtracking...
46     76 3.26264e-02 5.00000e-02 4.04217e-03    115
Backtracking...
47     78 2.14299e-03 5.00000e-02 4.03723e-03    107
Backtracking...
48     80 1.00000e-03 5.00000e-02 3.97649e-03    106
Backtracking...
49     82 1.00000e-03 4.99999e-02 3.87416e-03    104
Backtracking...
50     84 3.92451e-03 4.99999e-02 3.48224e-03    102
Backtracking...
51     86 1.13284e-01 4.99999e-02 3.83733e-03     84
Backtracking...
52     88 2.82300e-02 4.99999e-02 3.65085e-03     81
53     89 1.00000e+00 4.99997e-02 2.53542e-03     66
Backtracking...
54     91 2.46015e-01 4.99996e-02 3.51781e-03     65
Backtracking...
Backtracking...
55     94 3.50341e-04 4.99996e-02 3.59789e-03     58
Backtracking...
56     96 1.00000e-03 4.99995e-02 2.92133e-03     51
Backtracking...
57     98 9.23513e-03 4.99995e-02 2.95674e-03     49
Backtracking...
58    100 1.00000e-03 4.99995e-02 3.04431e-03     44
59    101 1.00000e+00 4.99991e-02 4.12260e-03     38
Backtracking...
Backtracking...
60    104 5.23265e-06 4.99991e-02 4.06727e-03     37
Backtracking...
61    106 1.00000e-03 4.99990e-02 3.61422e-03     37
62    107 1.00000e+00 4.99989e-02 4.04238e-03     21
Backtracking...
63    109 2.88973e-03 4.99989e-02 3.43913e-03     21
Backtracking...
64    111 2.29268e-01 4.99987e-02 2.12131e-03     21
65    112 1.00000e+00 4.99985e-02 2.67954e-03     28
Backtracking...
Backtracking...
Backtracking...
66    116 4.15876e-06 4.99985e-02 2.63678e-03     27
Progress in parameters or objective below progTol

residual =

0.0015

reconstructionError =

6.2500e-04

```

## Chain-structured conditional random field

```load wordData.mat
lambda = 1;

% Initialize
[w,v_start,v_end,v] = crfChain_initWeights(nFeatures,nStates,'zeros');
featureStart = cumsum([1 nFeatures(1:end)]); % data structure which relates high-level 'features' to elements of w
sentences = crfChain_initSentences(y);
nSentences = size(sentences,1);
maxSentenceLength = 1+max(sentences(:,2)-sentences(:,1));

fprintf('Training chain-structured CRF\n');
wv = [w(:);v_start;v_end;v(:)];
funObj = @(wv)crfChain_loss(wv,X,y,nStates,nFeatures,featureStart,sentences);
wv = L1General2_PSSgb(funObj,wv,lambda*ones(size(wv)),options);

% Split up weights
[w,v_start,v_end,v] = crfChain_splitWeights(wv,featureStart,nStates);

% Measure error
trainErr = 0;
trainZ = 0;
yhat = zeros(size(y));
for s = 1:nSentences
y_s = y(sentences(s,1):sentences(s,2));
[nodePot,edgePot]=crfChain_makePotentials(X,w,v_start,v_end,v,nFeatures,featureStart,sentences,s);
[nodeBel,edgeBel,logZ] = crfChain_infer(nodePot,edgePot);
[junk yhat(sentences(s,1):sentences(s,2))] = max(nodeBel,[],2);
end
trainErrRate = sum(y~=yhat)/length(y)

figure;
imagesc([y yhat]);
colormap gray
title('True sequence (left), sequence of marginally most likely states (right)');
figure;
subplot(1,2,1);
imagesc(w);
title('CRF Feature Potentials');
subplot(1,2,2);
imagesc([v_start' 0;v v_end]);
title('CRF Transition Potentials');
pause;
```
```Training chain-structured CRF
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 4.86086e-04 1.14143e+03 1.17753e+02     78
2      3 1.00000e+00 6.63197e+02 3.76440e+01     70
3      4 1.00000e+00 5.82276e+02 4.57577e+01     68
4      5 1.00000e+00 5.25860e+02 1.59569e+01     77
5      6 1.00000e+00 4.91382e+02 6.99251e+00     71
6      7 1.00000e+00 4.80566e+02 4.91975e+00     75
7      8 1.00000e+00 4.75227e+02 3.16558e+00     75
8      9 1.00000e+00 4.70818e+02 4.92525e+00     73
9     10 1.00000e+00 4.68909e+02 5.41885e+00     68
10     11 1.00000e+00 4.66096e+02 2.13411e+00     74
11     12 1.00000e+00 4.64156e+02 2.64028e+00     69
12     13 1.00000e+00 4.62693e+02 3.33599e+00     65
13     14 1.00000e+00 4.61455e+02 1.75989e+00     66
14     15 1.00000e+00 4.60940e+02 1.66060e+00     66
15     16 1.00000e+00 4.60451e+02 1.22923e+00     65
16     17 1.00000e+00 4.59987e+02 1.48194e+00     66
17     18 1.00000e+00 4.59660e+02 1.21257e+00     64
18     19 1.00000e+00 4.59362e+02 2.13169e+00     61
19     20 1.00000e+00 4.59175e+02 1.26938e+00     59
20     21 1.00000e+00 4.59062e+02 6.69563e-01     60
21     22 1.00000e+00 4.58984e+02 1.11080e+00     60
22     23 1.00000e+00 4.58956e+02 5.26510e-01     59
23     24 1.00000e+00 4.58917e+02 4.26663e-01     59
24     25 1.00000e+00 4.58896e+02 2.76077e-01     58
25     26 1.00000e+00 4.58883e+02 2.61146e-01     58
26     27 1.00000e+00 4.58861e+02 2.19739e-01     56
27     28 1.00000e+00 4.58849e+02 3.06288e-01     56
28     29 1.00000e+00 4.58835e+02 2.35700e-01     58
29     30 1.00000e+00 4.58829e+02 1.19030e-01     58
30     31 1.00000e+00 4.58823e+02 8.89767e-02     58
31     32 1.00000e+00 4.58819e+02 1.49742e-01     58
32     33 1.00000e+00 4.58817e+02 5.09001e-02     57
33     34 1.00000e+00 4.58816e+02 4.17467e-02     58
34     35 1.00000e+00 4.58815e+02 2.90042e-02     59
35     36 1.00000e+00 4.58814e+02 4.90847e-02     59
36     37 1.00000e+00 4.58814e+02 2.03209e-02     59
37     38 1.00000e+00 4.58814e+02 1.70537e-02     59
38     39 1.00000e+00 4.58814e+02 1.05995e-02     59
39     40 1.00000e+00 4.58814e+02 2.75948e-02     58
40     41 1.00000e+00 4.58814e+02 6.81921e-03     58
41     42 1.00000e+00 4.58814e+02 5.25640e-03     58
42     43 1.00000e+00 4.58814e+02 7.96413e-03     58
43     44 1.00000e+00 4.58813e+02 8.19545e-03     58
44     45 1.00000e+00 4.58813e+02 9.21045e-03     57
45     46 1.00000e+00 4.58813e+02 2.46348e-03     57
46     47 1.00000e+00 4.58813e+02 2.04623e-03     57
47     48 1.00000e+00 4.58813e+02 1.51764e-03     57
48     49 1.00000e+00 4.58813e+02 1.90082e-03     57
49     50 1.00000e+00 4.58813e+02 6.58426e-04     57
50     51 1.00000e+00 4.58813e+02 4.29058e-04     57
51     52 1.00000e+00 4.58813e+02 3.59842e-04     57
52     53 1.00000e+00 4.58813e+02 2.59996e-04     57
53     54 1.00000e+00 4.58813e+02 1.56046e-04     57
54     55 1.00000e+00 4.58813e+02 1.83365e-04     57
55     56 1.00000e+00 4.58813e+02 5.31323e-05     57
56     57 1.00000e+00 4.58813e+02 3.68219e-05     57
57     58 1.00000e+00 4.58813e+02 2.98633e-05     57
Directional derivative below progTol

trainErrRate =

0.1030

```

## Graphical Lasso

```lambda = .05;

% Generate a sparse positive-definite precision matrix
nNodes = 10;
P = (P+P')/2;
tau = 1;
X = P + tau*eye(nNodes);
while ~ispd(X)
tau = tau*2;
X = P + tau*eye(nNodes);
end
mu = randn(nNodes,1);

% Sample from the GGM
C = inv(X);
R = chol(C)';
X = zeros(nInstances,nNodes);
for i = 1:nInstances
X(i,:) = (mu + R*randn(nNodes,1))';
end

% Center and Standardize
X = standardizeCols(X);

% Train Full GGM
sigma_emp = cov(X);
nonZero = find(ones(nNodes));
funObj = @(x)sparsePrecisionObj(x,nNodes,nonZero,sigma_emp);
Kfull = eye(nNodes);
fprintf('Fitting full Gaussian graphical model\n');
Kfull(nonZero) = minFunc(funObj,Kfull(nonZero),options);

% Train GGM w/ L1-regularization
funObj = @(x)sparsePrecisionObj(x,nNodes,nonZero,sigma_emp);
Ksparse = eye(nNodes);
fprintf('Fitting sparse Gaussian graphical model\n');
Ksparse(nonZero) = L1General2_PSSgb(funObj,Ksparse(nonZero),lambda*ones(nNodes*nNodes,1),options);

figure;
subplot(1,2,1);
imagesc(Kfull);
title('Estimated Full Precision Matrix');
subplot(1,2,2);
imagesc(Ksparse);
title('Estimated Sparse Precision Matrix');
pause;
```
```Fitting full Gaussian graphical model
Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          4     4.56509e-01     8.71032e+00     3.59715e-01
2          5     1.00000e+00     8.36687e+00     2.68547e-01
3          6     1.00000e+00     8.03081e+00     1.37907e-01
4          7     1.00000e+00     7.89537e+00     1.20463e-01
5          8     1.00000e+00     7.82045e+00     6.59930e-02
6          9     1.00000e+00     7.79321e+00     4.23325e-02
7         10     1.00000e+00     7.78270e+00     4.04436e-02
8         11     1.00000e+00     7.77689e+00     2.48096e-02
9         12     1.00000e+00     7.77590e+00     1.28531e-02
10         13     1.00000e+00     7.77565e+00     7.40367e-03
11         14     1.00000e+00     7.77539e+00     4.47854e-03
12         15     1.00000e+00     7.77522e+00     5.34326e-03
13         16     1.00000e+00     7.77496e+00     4.07508e-03
14         17     1.00000e+00     7.77467e+00     2.88622e-03
15         19     2.24018e-01     7.77463e+00     2.41829e-03
16         20     1.00000e+00     7.77458e+00     6.65333e-04
17         21     1.00000e+00     7.77458e+00     8.04254e-05
18         22     1.00000e+00     7.77458e+00     1.78096e-05
19         23     1.00000e+00     7.77458e+00     1.18396e-05
Directional Derivative below progTol
Fitting sparse Gaussian graphical model
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 9.68356e-02 1.02198e+01 5.51510e-01     68
Backtracking...
Halving Step Size
2      4 5.00000e-01 9.47195e+00 2.65052e-01     68
3      5 1.00000e+00 9.30345e+00 1.55103e-01     70
4      6 1.00000e+00 9.14340e+00 9.95526e-02     66
5      7 1.00000e+00 9.11583e+00 8.57312e-02     58
6      8 1.00000e+00 9.09648e+00 3.34571e-02     62
7      9 1.00000e+00 9.09328e+00 2.54082e-02     60
8     10 1.00000e+00 9.09255e+00 1.75467e-02     62
9     11 1.00000e+00 9.09176e+00 7.25425e-03     58
10     12 1.00000e+00 9.09167e+00 4.48355e-03     60
11     13 1.00000e+00 9.09164e+00 1.20526e-03     60
12     14 1.00000e+00 9.09163e+00 1.31647e-03     58
13     15 1.00000e+00 9.09162e+00 1.40570e-03     58
14     16 1.00000e+00 9.09162e+00 6.74013e-04     58
15     17 1.00000e+00 9.09162e+00 4.52143e-04     58
16     18 1.00000e+00 9.09162e+00 1.52627e-04     58
17     19 1.00000e+00 9.09162e+00 1.13633e-04     58
18     20 1.00000e+00 9.09162e+00 1.06931e-04     58
19     21 1.00000e+00 9.09162e+00 1.21023e-04     58
20     22 1.00000e+00 9.09162e+00 2.43673e-05     58
21     23 1.00000e+00 9.09162e+00 1.73472e-05     58
22     24 1.00000e+00 9.09162e+00 9.34430e-06     58
First-order optimality below optTol
```

## Markov Random Field Structure Learning

```lambda = 5;

% Generate Data
nInstances = 500;
nNodes = 10;
edgeDensity = .5;
nStates = 2;
ising = 1;
tied = 0;
useMex = 1;
y = UGM_generate(nInstances,0,nNodes,edgeDensity,nStates,ising,tied);

% Set up MRF and initialize parameters
Xnode = ones(nInstances,1,nNodes);
Xedge = ones(nInstances,1,edgeStruct.nEdges);
[nodeMap,edgeMap] = UGM_makeMRFmaps(nNodes,edgeStruct,ising,tied);
nNodeParams = max(nodeMap(:));
nParams = max(edgeMap(:));
nEdgeParams = nParams-nNodeParams;
w = zeros(nParams,1);

% Make Edge Regularizer
regularizer = [zeros(nNodeParams,1);lambda*ones(nEdgeParams,1)];

% Inference Method
inferFunc = @UGM_Infer_Exact;

% Train
suffStat = UGM_MRF_computeSuffStat(y,nodeMap,edgeMap,edgeStruct);
funObj = @(w)UGM_MRF_NLL(w,nInstances,suffStat,nodeMap,edgeMap,edgeStruct,inferFunc);
w = L1General2_PSSgb(funObj,w,regularizer,options);

% Find Active Edges
for e = 1:edgeStruct.nEdges
params = edgeMap(:,:,e);
params = params(params(:)~=0);
if any(w(params(:)) ~= 0)
n1 = edgeStruct.edgeEnds(e,1);
n2 = edgeStruct.edgeEnds(e,2);
end
end

% Display Graph (requires graphviz)
figure;
title('Estimated Structure');
pause
```
```Exact Sampling
Iter fEvals      stepLen         fVal      optCond    nnz
1      2 9.66837e-05 3.28728e+03 2.28945e+02     64
Backtracking...
2      4 3.33072e-01 1.73173e+03 1.66001e+02     64
3      5 1.00000e+00 1.52885e+03 8.51719e+01     63
4      6 1.00000e+00 1.47799e+03 9.43565e+01     62
5      7 1.00000e+00 1.39067e+03 6.06855e+01     60
6      8 1.00000e+00 1.32590e+03 3.55498e+01     62
7      9 1.00000e+00 1.30238e+03 2.27209e+01     59
8     10 1.00000e+00 1.29383e+03 1.91823e+01     63
9     11 1.00000e+00 1.28507e+03 1.30392e+01     58
10     12 1.00000e+00 1.27722e+03 1.15441e+01     63
11     13 1.00000e+00 1.26751e+03 2.03941e+01     55
12     14 1.00000e+00 1.26067e+03 9.82268e+00     60
13     15 1.00000e+00 1.25943e+03 2.51742e+01     51
14     16 1.00000e+00 1.25590e+03 1.67151e+01     60
15     17 1.00000e+00 1.25460e+03 9.33454e+00     58
16     18 1.00000e+00 1.25265e+03 6.48217e+00     55
17     19 1.00000e+00 1.25062e+03 1.51427e+01     49
18     20 1.00000e+00 1.24950e+03 5.93236e+00     57
19     21 1.00000e+00 1.24880e+03 6.35659e+00     54
20     22 1.00000e+00 1.24809e+03 7.18483e+00     49
21     23 1.00000e+00 1.24596e+03 8.38261e+00     50
22     24 1.00000e+00 1.24554e+03 7.47506e+00     48
23     25 1.00000e+00 1.24423e+03 1.09210e+01     48
Backtracking...
24     27 3.53224e-02 1.24412e+03 9.80552e+00     52
25     28 1.00000e+00 1.24314e+03 4.80184e+00     47
26     29 1.00000e+00 1.24306e+03 5.16322e+00     44
27     30 1.00000e+00 1.24268e+03 3.45606e+00     44
28     31 1.00000e+00 1.24220e+03 5.18534e+00     41
29     32 1.00000e+00 1.24197e+03 5.47500e+00     41
30     33 1.00000e+00 1.24148e+03 5.22893e+00     39
31     34 1.00000e+00 1.24115e+03 4.99996e+00     40
32     35 1.00000e+00 1.24104e+03 4.16699e+00     39
33     36 1.00000e+00 1.24097e+03 3.30432e+00     39
34     37 1.00000e+00 1.24089e+03 1.78895e+00     39
Backtracking...
Backtracking...
35     40 4.80636e-02 1.24089e+03 9.29785e-01     38
36     41 1.00000e+00 1.24088e+03 9.27248e-01     38
37     42 1.00000e+00 1.24084e+03 7.14728e-01     38
38     43 1.00000e+00 1.24080e+03 9.86125e-01     37
39     44 1.00000e+00 1.24076e+03 1.40380e+00     37
40     45 1.00000e+00 1.24072e+03 1.42731e+00     37
41     46 1.00000e+00 1.24069e+03 1.03998e+00     36
Backtracking...
42     48 5.10320e-03 1.24069e+03 7.79459e-01     36
43     49 1.00000e+00 1.24068e+03 6.11590e-01     36
44     50 1.00000e+00 1.24067e+03 1.02219e+00     37
45     51 1.00000e+00 1.24066e+03 8.22553e-01     36
46     52 1.00000e+00 1.24065e+03 5.66586e-01     36
47     53 1.00000e+00 1.24064e+03 3.44650e-01     36
Backtracking...
48     55 4.68473e-01 1.24064e+03 5.70245e-01     35
Backtracking...
49     57 3.02884e-01 1.24063e+03 3.83324e-01     36
Backtracking...
50     59 3.80878e-01 1.24063e+03 5.00663e-01     38
Backtracking...
51     61 3.28994e-01 1.24063e+03 5.28505e-01     37
52     62 1.00000e+00 1.24063e+03 1.72054e-01     37
53     63 1.00000e+00 1.24063e+03 2.11417e-01     36
54     64 1.00000e+00 1.24063e+03 4.02944e-01     35
55     65 1.00000e+00 1.24063e+03 1.97025e-01     35
56     66 1.00000e+00 1.24063e+03 1.31077e-01     35
57     67 1.00000e+00 1.24062e+03 2.31428e-01     35
58     68 1.00000e+00 1.24062e+03 1.78554e-01     35
59     69 1.00000e+00 1.24062e+03 1.01245e-01     35
60     70 1.00000e+00 1.24062e+03 9.32327e-02     35
61     71 1.00000e+00 1.24062e+03 1.00376e-01     35
62     72 1.00000e+00 1.24062e+03 8.95266e-02     35
63     73 1.00000e+00 1.24062e+03 7.26738e-02     35
64     74 1.00000e+00 1.24062e+03 4.67728e-02     35
65     75 1.00000e+00 1.24062e+03 2.58079e-02     35
66     76 1.00000e+00 1.24062e+03 2.28698e-02     35
67     77 1.00000e+00 1.24062e+03 3.77576e-02     35
68     78 1.00000e+00 1.24062e+03 3.47257e-02     35
69     79 1.00000e+00 1.24062e+03 2.42335e-02     35
70     80 1.00000e+00 1.24062e+03 1.67902e-02     35
71     81 1.00000e+00 1.24062e+03 1.08630e-02     35
72     82 1.00000e+00 1.24062e+03 8.51776e-03     35
73     83 1.00000e+00 1.24062e+03 6.40910e-03     35
74     84 1.00000e+00 1.24062e+03 4.00380e-03     35
75     85 1.00000e+00 1.24062e+03 1.62359e-03     35
76     86 1.00000e+00 1.24062e+03 6.93276e-04     35
77     87 1.00000e+00 1.24062e+03 6.15525e-04     35
78     88 1.00000e+00 1.24062e+03 5.46709e-04     35
79     89 1.00000e+00 1.24062e+03 3.98170e-04     35
80     90 1.00000e+00 1.24062e+03 2.63634e-04     35
81     91 1.00000e+00 1.24062e+03 1.69251e-04     35
82     92 1.00000e+00 1.24062e+03 1.40953e-04     35
83     93 1.00000e+00 1.24062e+03 1.10144e-04     35
84     94 1.00000e+00 1.24062e+03 4.16877e-05     35
Directional derivative below progTol
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)
```

## Neural Network with Sparse Connections

```fig = 100;
lambda = 1;
options.verbose = 0;

% Generate non-linear regression data set
nInstances = 200;
nVars = 1;
[X,y] = makeData('regressionNonlinear',nInstances,nVars);

X = [ones(nInstances,1) X];
nVars = nVars+1;

% Train neural network w/ multiple hiden layers
nHidden = [9 9];
nParams = nVars*nHidden(1);
for h = 2:length(nHidden);
nParams = nParams+nHidden(h-1)*nHidden(h);
end
nParams = nParams+nHidden(end);

funObj = @(weights)MLPregressionLoss_efficient(weights,X,y,nHidden);
fprintf('Training neural network for regression...\n');
lambdaL2 = 1e-3;
wMLP = randn(nParams,1);
funObjL2 = @(w)penalizedL2(w,funObj,lambdaL2);
while 1
w_old = wMLP;
wMLP = L1General2_PSSgb(funObjL2,wMLP,lambda*ones(nParams,1),options);
if norm(w_old-wMLP,inf) < 1e-4
break;
end
end

% Plot results
figure;
hold on
Xtest = [-5:.05:5]';
Xtest = [ones(size(Xtest,1),1) Xtest];
yhat = MLPregressionPredict_efficient(wMLP,Xtest,nHidden);
plot(X(:,2),y,'.');
h=plot(Xtest(:,2),yhat,'g-');
set(h,'LineWidth',3);
legend({'Data','Neural Net'});

% Form weights
inputWeights = reshape(wMLP(1:nVars*nHidden(1)),nVars,nHidden(1));
offset = nVars*nHidden(1);
for h = 2:length(nHidden)
hiddenWeights{h-1} = reshape(wMLP(offset+1:offset+nHidden(h-1)*nHidden(h)),nHidden(h-1),nHidden(h));
offset = offset+nHidden(h-1)*nHidden(h);
end
outputWeights = wMLP(offset+1:offset+nHidden(end));

for i = 1:nVars
for j = 1:nHidden(1)
if abs(inputWeights(i,j)) > 1e-4
end
end
end
for h = 1:length(nHidden)-1
for i = 1:nHidden(h)
for j = 1:nHidden(h+1)
if abs(hiddenWeights{h}(i,j)) > 1e-4
end
end
end
end
for i = 1:nHidden(end)
if abs(outputWeights(i)) > 1e-4
end
end

for i = 1:nVars
labels{i,1} = sprintf('x_%d',i);
end
for h = 1:length(nHidden)
i = i + 1;
labels{i,1} = sprintf('b_%d',h);
for j = 2:nHidden(h)
i = i + 1;
labels{i,1} = sprintf('h_%d_%d',h,j);
end
end
labels{end,1} = 'y';

% Plot Network
figure;
title('Neural Network');
pause
```
```Training neural network for regression...
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)
```

## Deep Network with Sparse Connections

```fig = 1000;
lambda = 2;
options.maxIter = 100; % Restart more often for non-convex
options.verbose = 0;

% Generate non-linear regression data set
nInstances = 200;
nVars = 1;
[X,y] = makeData('regressionNonlinear2',nInstances,nVars);

X = [ones(nInstances,1) X];
nVars = nVars+1;

% Train neural network w/ multiple hiden layers
nHidden = [9 9 9 9 9 9 9 9 9];
nParams = nVars*nHidden(1);
for h = 2:length(nHidden);
nParams = nParams+nHidden(h-1)*nHidden(h);
end
nParams = nParams+nHidden(end);

funObj = @(weights)MLPregressionLoss_efficient(weights,X,y,nHidden);
fprintf('Training neural network for regression...\n');
lambdaL2 = 1e-3;
wMLP = randn(nParams,1);
funObjL2 = @(w)penalizedL2(w,funObj,lambdaL2);
for i = 1:250
fprintf('Training:');
w_old = wMLP;
wMLP = L1General2_PSSgb(funObjL2,wMLP,lambda*ones(nParams,1),options);
fprintf(' (nnz = %d, max change = %f)\n',nnz(wMLP),norm(w_old-wMLP,inf));
if norm(w_old-wMLP,inf) < 1e-5
break;
end
end

% Plot results
figure;
hold on
Xtest = [-5:.05:5]';
Xtest = [ones(size(Xtest,1),1) Xtest];
yhat = MLPregressionPredict_efficient(wMLP,Xtest,nHidden);
plot(X(:,2),y,'.');
h=plot(Xtest(:,2),yhat,'g-');
set(h,'LineWidth',3);
legend({'Data','Deep Neural Net'});

% Form weights
inputWeights = reshape(wMLP(1:nVars*nHidden(1)),nVars,nHidden(1));
offset = nVars*nHidden(1);
for h = 2:length(nHidden)
hiddenWeights{h-1} = reshape(wMLP(offset+1:offset+nHidden(h-1)*nHidden(h)),nHidden(h-1),nHidden(h));
offset = offset+nHidden(h-1)*nHidden(h);
end
outputWeights = wMLP(offset+1:offset+nHidden(end));

for i = 1:nVars
for j = 1:nHidden(1)
if abs(inputWeights(i,j)) > 1e-4
end
end
end
for h = 1:length(nHidden)-1
for i = 1:nHidden(h)
for j = 1:nHidden(h+1)
if abs(hiddenWeights{h}(i,j)) > 1e-4
end
end
end
end
for i = 1:nHidden(end)
if abs(outputWeights(i)) > 1e-4
end
end

for i = 1:nVars
labels{i,1} = sprintf('x_%d',i);
end
for h = 1:length(nHidden)
i = i + 1;
labels{i,1} = sprintf('b_%d',h);
for j = 2:nHidden(h)
i = i + 1;
labels{i,1} = sprintf('h_%d_%d',h,j);
end
end
labels{end,1} = 'y';

% Plot Network
figure;
title('Neural Network');
```
```Training neural network for regression...
Training: (nnz = 667, max change = 0.967185)
Training: (nnz = 635, max change = 0.779682)
Training: (nnz = 641, max change = 2.636920)
Training: (nnz = 633, max change = 0.596286)
Training: (nnz = 595, max change = 0.402751)
Training: (nnz = 588, max change = 0.474491)
Training: (nnz = 566, max change = 0.237261)
Training: (nnz = 542, max change = 0.594388)
Training: (nnz = 475, max change = 0.411673)
Training: (nnz = 457, max change = 0.290917)
Training: (nnz = 404, max change = 0.177169)
Training: (nnz = 360, max change = 0.286962)
Training: (nnz = 351, max change = 0.245158)
Training: (nnz = 301, max change = 0.236724)
Training: (nnz = 253, max change = 0.197978)
Training: (nnz = 229, max change = 0.234248)
Training: (nnz = 214, max change = 0.110784)
Training: (nnz = 209, max change = 0.134172)
Training: (nnz = 200, max change = 0.112478)
Training: (nnz = 183, max change = 0.048467)
Training: (nnz = 183, max change = 0.075826)
Training: (nnz = 178, max change = 0.065721)
Training: (nnz = 175, max change = 0.090798)
Training: (nnz = 166, max change = 0.052506)
Training: (nnz = 165, max change = 0.098766)
Training: (nnz = 162, max change = 0.047602)
Training: (nnz = 152, max change = 0.220688)
Training: (nnz = 145, max change = 0.054457)
Training: (nnz = 150, max change = 0.055629)
Training: (nnz = 147, max change = 0.027674)
Training: (nnz = 142, max change = 0.021358)
Training: (nnz = 144, max change = 0.023483)
Training: (nnz = 135, max change = 0.016194)
Training: (nnz = 138, max change = 0.047708)
Training: (nnz = 144, max change = 0.363394)
Training: (nnz = 134, max change = 0.039826)
Training: (nnz = 135, max change = 0.031717)
Training: (nnz = 141, max change = 0.038360)
Training: (nnz = 131, max change = 0.017409)
Training: (nnz = 129, max change = 0.068170)
Training: (nnz = 117, max change = 0.039581)
Training: (nnz = 123, max change = 0.021510)
Training: (nnz = 109, max change = 0.033459)
Training: (nnz = 109, max change = 0.022375)
Training: (nnz = 110, max change = 0.018931)
Training: (nnz = 104, max change = 0.028466)
Training: (nnz = 116, max change = 0.054193)
Training: (nnz = 107, max change = 0.029349)
Training: (nnz = 106, max change = 0.016000)
Training: (nnz = 106, max change = 0.046723)
Training: (nnz = 104, max change = 0.038716)
Training: (nnz = 98, max change = 0.023524)
Training: (nnz = 98, max change = 0.023039)
Training: (nnz = 114, max change = 0.055210)
Training: (nnz = 99, max change = 0.036755)
Training: (nnz = 97, max change = 0.090063)
Training: (nnz = 88, max change = 0.045667)
Training: (nnz = 106, max change = 0.035059)
Training: (nnz = 102, max change = 0.076933)
Training: (nnz = 100, max change = 0.028952)
Training: (nnz = 92, max change = 0.051420)
Training: (nnz = 86, max change = 0.070376)
Training: (nnz = 83, max change = 0.036400)
Training: (nnz = 86, max change = 0.059172)
Training: (nnz = 77, max change = 0.032287)
Training: (nnz = 84, max change = 0.030322)
Training: (nnz = 80, max change = 0.026959)
Training: (nnz = 71, max change = 0.028908)
Training: (nnz = 75, max change = 0.020683)
Training: (nnz = 78, max change = 0.019452)
Training: (nnz = 78, max change = 0.016166)
Training: (nnz = 74, max change = 0.028601)
Training: (nnz = 68, max change = 0.021595)
Training: (nnz = 74, max change = 0.062828)
Training: (nnz = 73, max change = 0.030521)
Training: (nnz = 74, max change = 0.017069)
Training: (nnz = 69, max change = 0.011962)
Training: (nnz = 71, max change = 0.054461)
Training: (nnz = 72, max change = 0.018591)
Training: (nnz = 69, max change = 0.015757)
Training: (nnz = 70, max change = 0.016410)
Training: (nnz = 68, max change = 0.017009)
Training: (nnz = 62, max change = 0.009676)
Training: (nnz = 67, max change = 0.026478)
Training: (nnz = 60, max change = 0.012650)
Training: (nnz = 60, max change = 0.012972)
Training: (nnz = 62, max change = 0.025775)
Training: (nnz = 63, max change = 0.019782)
Training: (nnz = 57, max change = 0.003912)
Training: (nnz = 57, max change = 0.066164)
Training: (nnz = 56, max change = 0.017881)
Training: (nnz = 56, max change = 0.015960)
Training: (nnz = 49, max change = 0.003901)
Training: (nnz = 60, max change = 0.031464)
Training: (nnz = 52, max change = 0.007451)
Training: (nnz = 47, max change = 0.013513)
Training: (nnz = 48, max change = 0.018263)
Training: (nnz = 48, max change = 0.005887)
Training: (nnz = 49, max change = 0.011130)
Training: (nnz = 64, max change = 0.340443)
Training: (nnz = 57, max change = 0.026491)
Training: (nnz = 59, max change = 0.023941)
Training: (nnz = 56, max change = 0.012665)
Training: (nnz = 53, max change = 0.012904)
Training: (nnz = 48, max change = 0.002348)
Training: (nnz = 49, max change = 0.007044)
Training: (nnz = 46, max change = 0.002118)
Training: (nnz = 54, max change = 0.004446)
Training: (nnz = 43, max change = 0.001986)
Training: (nnz = 52, max change = 0.004125)
Training: (nnz = 42, max change = 0.002107)
Training: (nnz = 58, max change = 0.051137)
Training: (nnz = 49, max change = 0.004627)
Training: (nnz = 45, max change = 0.002313)
Training: (nnz = 41, max change = 0.001677)
Training: (nnz = 55, max change = 0.049206)
Training: (nnz = 52, max change = 0.005081)
Training: (nnz = 45, max change = 0.002384)
Training: (nnz = 54, max change = 0.015250)
Training: (nnz = 48, max change = 0.001602)
Training: (nnz = 48, max change = 0.005457)
Training: (nnz = 44, max change = 0.002047)
Training: (nnz = 57, max change = 0.070230)
Training: (nnz = 48, max change = 0.003937)
Training: (nnz = 54, max change = 0.017014)
Training: (nnz = 49, max change = 0.003003)
Training: (nnz = 47, max change = 0.002668)
Training: (nnz = 45, max change = 0.005518)
Training: (nnz = 46, max change = 0.003938)
Training: (nnz = 56, max change = 0.071594)
Training: (nnz = 54, max change = 0.004618)
Training: (nnz = 51, max change = 0.004196)
Training: (nnz = 47, max change = 0.002295)
Training: (nnz = 50, max change = 0.001950)
Training: (nnz = 49, max change = 0.029989)
Training: (nnz = 45, max change = 0.002197)
Training: (nnz = 42, max change = 0.004006)
Training: (nnz = 46, max change = 0.005113)
Training: (nnz = 44, max change = 0.002699)
Training: (nnz = 51, max change = 0.013256)
Training: (nnz = 47, max change = 0.002213)
Training: (nnz = 41, max change = 0.005064)
Training: (nnz = 40, max change = 0.001433)
Training: (nnz = 49, max change = 0.049151)
Training: (nnz = 45, max change = 0.002600)
Training: (nnz = 40, max change = 0.003848)
Training: (nnz = 44, max change = 0.005215)
Training: (nnz = 44, max change = 0.003579)
Training: (nnz = 42, max change = 0.010221)
Training: (nnz = 40, max change = 0.003269)
Training: (nnz = 43, max change = 0.002905)
Training: (nnz = 44, max change = 0.004377)
Training: (nnz = 39, max change = 0.003928)
Training: (nnz = 40, max change = 0.004021)
Training: (nnz = 47, max change = 0.004692)
Training: (nnz = 42, max change = 0.003924)
Training: (nnz = 39, max change = 0.003004)
Training: (nnz = 43, max change = 0.020384)
Training: (nnz = 38, max change = 0.002724)
Training: (nnz = 42, max change = 0.005549)
Training: (nnz = 39, max change = 0.003570)
Training: (nnz = 39, max change = 0.004197)
Training: (nnz = 36, max change = 0.002509)
Training: (nnz = 55, max change = 0.083566)
Training: (nnz = 54, max change = 0.004167)
Training: (nnz = 46, max change = 0.002941)
Training: (nnz = 43, max change = 0.011560)
Training: (nnz = 42, max change = 0.004004)
Training: (nnz = 39, max change = 0.005500)
Training: (nnz = 37, max change = 0.000033)
Training: (nnz = 41, max change = 0.004653)
Training: (nnz = 39, max change = 0.004093)
Training: (nnz = 36, max change = 0.003060)
Training: (nnz = 38, max change = 0.016099)
Training: (nnz = 37, max change = 0.003698)
Training: (nnz = 41, max change = 0.004591)
Training: (nnz = 38, max change = 0.003958)
Training: (nnz = 38, max change = 0.004611)
Training: (nnz = 52, max change = 0.012214)
Training: (nnz = 39, max change = 0.002305)
Training: (nnz = 40, max change = 0.003415)
Training: (nnz = 46, max change = 0.019619)
Training: (nnz = 39, max change = 0.002502)
Training: (nnz = 49, max change = 0.119394)
Training: (nnz = 52, max change = 0.002777)
Training: (nnz = 46, max change = 0.002635)
Training: (nnz = 40, max change = 0.001766)
Training: (nnz = 49, max change = 0.003978)
Training: (nnz = 39, max change = 0.003955)
Training: (nnz = 41, max change = 0.022689)
Training: (nnz = 44, max change = 0.003586)
Training: (nnz = 40, max change = 0.007178)
Training: (nnz = 43, max change = 0.011511)
Training: (nnz = 40, max change = 0.003482)
Training: (nnz = 38, max change = 0.006133)
Training: (nnz = 43, max change = 0.007837)
Training: (nnz = 36, max change = 0.004371)
Training: (nnz = 35, max change = 0.007612)
Training: (nnz = 62, max change = 0.778716)
Training: (nnz = 58, max change = 0.025350)
Training: (nnz = 47, max change = 0.007105)
Training: (nnz = 51, max change = 0.005689)
Training: (nnz = 57, max change = 0.003822)
Training: (nnz = 59, max change = 0.014219)
Training: (nnz = 49, max change = 0.001399)
Training: (nnz = 61, max change = 0.521767)
Training: (nnz = 58, max change = 0.027560)
Training: (nnz = 55, max change = 0.011300)
Training: (nnz = 55, max change = 0.013688)
Training: (nnz = 55, max change = 0.010813)
Training: (nnz = 53, max change = 0.013645)
Training: (nnz = 53, max change = 0.006686)
Training: (nnz = 58, max change = 0.018081)
Training: (nnz = 50, max change = 0.015506)
Training: (nnz = 51, max change = 0.002396)
Training: (nnz = 54, max change = 0.032205)
Training: (nnz = 49, max change = 0.006285)
Training: (nnz = 56, max change = 0.017473)
Training: (nnz = 47, max change = 0.003555)
Training: (nnz = 52, max change = 0.003757)
Training: (nnz = 59, max change = 0.029159)
Training: (nnz = 52, max change = 0.009495)
Training: (nnz = 51, max change = 0.008131)
Training: (nnz = 55, max change = 0.007305)
Training: (nnz = 54, max change = 0.007790)
Training: (nnz = 46, max change = 0.003035)
Training: (nnz = 45, max change = 0.007596)
Training: (nnz = 61, max change = 0.143070)
Training: (nnz = 60, max change = 0.025799)
Training: (nnz = 53, max change = 0.014376)
Training: (nnz = 63, max change = 0.017579)
Training: (nnz = 52, max change = 0.005534)
Training: (nnz = 52, max change = 0.019374)
Training: (nnz = 50, max change = 0.005859)
Training: (nnz = 56, max change = 0.013841)
Training: (nnz = 49, max change = 0.007306)
Training: (nnz = 51, max change = 0.017447)
Training: (nnz = 42, max change = 0.001871)
Training: (nnz = 44, max change = 0.005889)
Training: (nnz = 43, max change = 0.004112)
Training: (nnz = 49, max change = 0.020014)
Training: (nnz = 45, max change = 0.002811)
Training: (nnz = 39, max change = 0.003848)
Training: (nnz = 51, max change = 0.018842)
Training: (nnz = 37, max change = 0.001834)
Training: (nnz = 34, max change = 0.004377)
Training: (nnz = 42, max change = 0.013879)
Training: (nnz = 35, max change = 0.001855)
Training: (nnz = 40, max change = 0.016622)
Training: (nnz = 36, max change = 0.001042)
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)
```

Mark Schmidt > Software > L1General > Examples of using L1General