# Examples of using minConf_TMP

These demos have been set up to show how minConf_TMP from the minConf package can be used to optimize a variety of smooth objective functions subject to bound constraints.

The code used to run these demos can be downloaded here (which includes a copy of the 2009 version of minConf as well as the 2012 version of minFunc package).

To run all of the demos, type (in Matlab):

```>> cd TMPexamples            % Change to the extracted directory
>> mexAll                    % Compile mex files (not necessary on all systems)
>> TMP_examples              % Run the examples
```

## Non-negative Least Squares

Solve min_w (1/2)norm(X*w-y)^2, s.t. w >= 0.

```nInstances = 1000;
nVars = 100;
X = randn(nInstances,nVars);
w = randn(nVars,1);
y = X*w + randn(nInstances,1);

funObj = @(w)SquaredError(w,X,y);
LB = zeros(nVars,1);
UB = inf(nVars,1);

fprintf('Solving non-negative least-squares problem...\n');
w = minConf_TMP(funObj,zeros(nVars,1),LB,UB);
stem(w);title('Non-negative least-squares coefficients');
pause
```
```Solving non-negative least-squares problem...
Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     1.24808e-05     8.89869e+04     7.79994e+04
2          3     1.00000e+00     4.69101e+04     3.71020e+03
3          4     1.00000e+00     4.67942e+04     8.92785e+02
4          5     1.00000e+00     4.67885e+04     2.15416e+02
5          6     1.00000e+00     4.67882e+04     3.74646e+01
6          7     1.00000e+00     4.67882e+04     1.24189e+01
7          8     1.00000e+00     4.67882e+04     2.13076e+00
8          9     1.00000e+00     4.67882e+04     3.42795e-01
9         10     1.00000e+00     4.67882e+04     1.60818e-01
Function value changing by less than optTol
```

## L1-Regularization Least Squares

Solve min_w (1/2)norm(X*w-y)^2 + lambda*sum(abs(w)) by formulating as min_w (1/2)norm([X -X]*w - y)^2 + lambda*sum(w), s.t. w >= 0

```lambda = 1000*ones(nVars,1);

LB = zeros(2*nVars,1);
UB = inf(2*nVars,1);

fprintf('Solving L1-regularized least-squares problem...\n');
w = minConf_TMP(regObj,zeros(2*nVars,1),LB,UB);
w = w(1:nVars)-w(nVars+1:end);
stem(w);title('L1-regularized least-squares coefficients');
pause
```
```Solving L1-regularized least-squares problem...
Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     1.33277e-05     8.92479e+04     7.27051e+04
2          3     1.00000e+00     5.65089e+04     1.01573e+04
3          4     1.00000e+00     5.60961e+04     5.85940e+03
4          5     1.00000e+00     5.58765e+04     1.27821e+03
5          6     1.00000e+00     5.58677e+04     8.21728e+02
6          7     1.00000e+00     5.58628e+04     1.75008e+02
7          8     1.00000e+00     5.58626e+04     8.13803e+01
8          9     1.00000e+00     5.58626e+04     1.96652e+01
9         10     1.00000e+00     5.58626e+04     9.32875e+00
10         11     1.00000e+00     5.58626e+04     3.11834e+00
11         12     1.00000e+00     5.58626e+04     1.39486e+00
12         13     1.00000e+00     5.58626e+04     4.80418e-01
13         14     1.00000e+00     5.58626e+04     1.97328e-01
Directional Derivative below optTol
```

## Logistic Regression with Bounded Coefficients

Solve min_w sum_i log(1 + exp(-y(i)*w*X(i,:)), s.t. -1 <= w <= 1

```y = sign(y);

funObj = @(w)LogisticLoss(w,X,y);
LB = -ones(nVars,1);
UB = ones(nVars,1);

fprintf('Solving bounded logistic regression problem...\n');
w = minConf_TMP(funObj,zeros(nVars,1),LB,UB);
stem(w);title('Bounded logistic regression coefficients');
pause
```
```Solving bounded logistic regression problem...
Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     2.94363e-04     6.43855e+02     3.11272e+03
2          3     1.00000e+00     3.07472e+02     1.04097e+03
3          4     1.00000e+00     2.30254e+02     6.41644e+02
4          5     1.00000e+00     1.69623e+02     5.34782e+02
5          6     1.00000e+00     1.37826e+02     1.81410e+02
6          7     1.00000e+00     1.33822e+02     8.91419e+01
7          8     1.00000e+00     1.32733e+02     3.77198e+01
8          9     1.00000e+00     1.32525e+02     2.07840e+01
9         10     1.00000e+00     1.32484e+02     8.77170e+00
10         11     1.00000e+00     1.32475e+02     4.18951e+00
11         12     1.00000e+00     1.32472e+02     2.13249e+00
12         13     1.00000e+00     1.32471e+02     1.75750e+00
13         14     1.00000e+00     1.32471e+02     4.65247e-01
14         15     1.00000e+00     1.32471e+02     3.07314e-01
15         16     1.00000e+00     1.32471e+02     1.51880e-01
16         17     1.00000e+00     1.32471e+02     1.09456e-01
17         18     1.00000e+00     1.32471e+02     4.28841e-02
Function value changing by less than optTol
```

## Dual Support Vector Machines (no bias or regularized bias)

Solve min_alpha (1/2)alpha'*A*alpha - sum(alpha), s.t. 0 <= alpha <= C, where A_ij = y_iy_jK(x_i,x_j), and we use the RBF kernel with sigma = 1.

```% Generate data
nInstances = 250;
[X,y] = makeData('classificationNonlinear',nInstances,2,2);

sigma = 1;
K = kernelRBF(X,X,sigma);
A = diag(y)*K*diag(y);

C = 1;
funObj = @(alpha)dualSVMLoss_noBias(alpha,A,y);
LB = zeros(nInstances,1);
UB = C*ones(nInstances,1);

fprintf('Solving dual SVM problem (no bias)...\n');
alpha = minConf_TMP(funObj,zeros(nInstances,1),LB,UB);
stem(alpha);title('Dual SVM coefficients');

yhat = sign(sum((diag(alpha.*y)*K)))';
trainErr = sum(yhat~=y)/numel(y)
pause
```
```Solving dual SVM problem (no bias)...
Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     4.00000e-03    -9.26917e-01     2.13459e+02
2          3     1.00000e+00    -5.75999e+00     1.22354e+02
3          4     1.00000e+00    -9.03139e+00     1.90347e+02
4          5     1.00000e+00    -1.97189e+01     2.80367e+02
5          6     1.00000e+00    -2.72854e+01     4.65150e+02
6          7     1.00000e+00    -3.13428e+01     2.20405e+02
7          8     1.00000e+00    -4.02030e+01     2.28478e+02
8          9     1.00000e+00    -4.41960e+01     1.27701e+02
9         10     1.00000e+00    -4.78569e+01     8.68537e+01
10         11     1.00000e+00    -4.97190e+01     6.82303e+01
11         12     1.00000e+00    -5.34507e+01     4.15025e+01
Cubic Backtracking
12         14     1.19267e-01    -5.40919e+01     2.06099e+01
13         15     1.00000e+00    -5.46082e+01     1.64300e+01
14         16     1.00000e+00    -5.54528e+01     5.04322e+00
15         17     1.00000e+00    -5.55415e+01     4.37118e+00
16         18     1.00000e+00    -5.57400e+01     3.02294e+00
17         19     1.00000e+00    -5.57908e+01     3.68629e+00
18         20     1.00000e+00    -5.58890e+01     1.91893e+00
19         21     1.00000e+00    -5.59069e+01     1.60164e+00
20         22     1.00000e+00    -5.59286e+01     1.29089e+00
21         23     1.00000e+00    -5.59386e+01     9.61066e-01
22         24     1.00000e+00    -5.59445e+01     1.19811e+00
23         25     1.00000e+00    -5.59631e+01     1.00647e+00
24         26     1.00000e+00    -5.59823e+01     8.48519e-01
25         27     1.00000e+00    -5.59933e+01     7.11928e-01
26         28     1.00000e+00    -5.60026e+01     6.59629e-01
27         29     1.00000e+00    -5.60134e+01     9.78925e-01
Cubic Backtracking
28         31     4.48271e-02    -5.60148e+01     8.91318e-01
29         32     1.00000e+00    -5.60191e+01     5.14404e-01
30         33     1.00000e+00    -5.60256e+01     3.83046e-01
31         34     1.00000e+00    -5.60383e+01     3.81827e-01
Cubic Backtracking
32         36     2.39554e-01    -5.60446e+01     4.20466e-01
33         37     1.00000e+00    -5.60522e+01     5.11830e-01
Cubic Backtracking
34         39     6.78098e-02    -5.60539e+01     4.49363e-01
35         40     1.00000e+00    -5.60619e+01     4.95361e-01
36         41     1.00000e+00    -5.60745e+01     2.96566e-01
Cubic Backtracking
37         43     3.00525e-02    -5.60766e+01     2.83966e-01
Cubic Backtracking
38         45     3.40924e-02    -5.60787e+01     2.22022e-01
Cubic Backtracking
Cubic Backtracking
39         48     7.82637e-03    -5.60821e+01     2.37324e-01
Cubic Backtracking
Cubic Backtracking
40         51     3.79133e-02    -5.60858e+01     2.32438e-01
41         52     1.00000e+00    -5.60889e+01     3.69327e-01
Cubic Backtracking
Cubic Backtracking
42         55     2.72854e-02    -5.60916e+01     2.11027e-01
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
43         59     8.19016e-03    -5.60921e+01     2.12031e-01
Cubic Backtracking
44         61     1.33245e-01    -5.60973e+01     1.47493e-01
Cubic Backtracking
Cubic Backtracking
45         64     8.24514e-03    -5.60982e+01     1.57583e-01
46         65     1.00000e+00    -5.61010e+01     1.27420e-01
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
47         90     3.36137e-07    -5.61020e+01     1.47095e-01
48         91     1.00000e+00    -5.61037e+01     1.13463e-01
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
49        114     1.21311e-06    -5.61045e+01     6.81895e-02
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
50        131     6.01075e-06    -5.61046e+01     5.04040e-02
51        132     1.00000e+00    -5.61054e+01     5.23710e-02
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
52        159     9.35330e-09    -5.61054e+01     4.80074e-02
53        160     1.00000e+00    -5.61058e+01     5.61451e-02
54        161     1.00000e+00    -5.61065e+01     6.50190e-02
55        162     1.00000e+00    -5.61072e+01     5.95579e-02
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
56        172     3.29516e-04    -5.61073e+01     4.05834e-02
57        173     1.00000e+00    -5.61078e+01     2.31386e-02
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
Cubic Backtracking
58        183     4.67633e-05    -5.61079e+01     2.40555e-02
59        184     1.00000e+00    -5.61082e+01     1.99010e-03
60        185     1.00000e+00    -5.61082e+01     1.26033e-03
Directional Derivative below optTol

trainErr =

0.0720

```

## Ordinal Logistic Regression

Solve min_{w,gamma} -sum(log(F(gamma(y+1) - X*w) - F(gamma(y) - X*w))), where F(x) = 1/(1+exp(-x), -inf < 0 < gamma(1) < gamma(2) < ... < inf

```% Generate Data
nInstances = 1000;
nVars = 10;
nClasses = 5;
X = randn(nInstances,nVars);
w = randn(nVars,1);
gamma = sort(randn(nClasses-1,1));
z = X*w;
y = zeros(nInstances,1);
y(z < gamma(1)) = 1;
for class = 2:nClasses-1
y(z >= gamma(class-1) & z < gamma(class)) = class;
end
y(z >= gamma(nClasses-1)) = nClasses;

% Standardize columns and add bias
X = standardizeCols(X);
X = [ones(nInstances,1) X];
nVars = nVars+1;

% First try Multinomial Logistic
fprintf('Training multinomial logistic classifier\n');
model = classificationSoftmax(X,y,struct('nClasses',nClasses));
yhat = model.predictFunc(model,X);
trainErr_MLR = sum(abs(yhat-y));

% Ordinal Logistic
funObj = @(w)OrdinalLogisticLoss2(w,X,y,nClasses);
LB = [-inf(nVars,1);zeros(nClasses-2,1)];
UB = inf(nVars+nClasses-2,1);
w_init = zeros(nVars,1);
gamma_init = sort(rand(nClasses-2,1));
fprintf('Training ordinal logistic classifier\n');
wGamma = minConf_TMP(funObj,[w_init;gamma_init],LB,UB);
w = wGamma(1:nVars);
gamma = [-inf;0;cumsum(wGamma(nVars+1:end));inf];

% Predict labels on training data
z = X*w;
yhat = zeros(nInstances,1);
for c = 1:nClasses
yhat(z > gamma(c)) = c;
end
trainErr_OLR = sum(abs(yhat-y));

fprintf('Training error of multinomial logistic regression: %f\n',trainErr_MLR/nInstances);
fprintf('Training error of ordinal logistic regression: %f\n',trainErr_OLR/nInstances);
pause
```
```Training multinomial logistic classifier
Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          2     5.73173e-04     1.47346e+03     2.38265e+02
2          3     1.00000e+00     8.92317e+02     7.51778e+01
3          4     1.00000e+00     7.61403e+02     4.14169e+01
4          5     1.00000e+00     6.31136e+02     5.74177e+01
5          6     1.00000e+00     5.53854e+02     7.22049e+01
6          7     1.00000e+00     5.05579e+02     2.31216e+01
7          8     1.00000e+00     4.59853e+02     1.34245e+01
8          9     1.00000e+00     4.00381e+02     1.74819e+01
9         11     3.83398e-01     3.64375e+02     3.30776e+01
10         12     1.00000e+00     2.75826e+02     2.54871e+01
11         13     1.00000e+00     2.12170e+02     2.13205e+01
12         14     1.00000e+00     1.78035e+02     2.22624e+01
13         15     1.00000e+00     1.57119e+02     1.16054e+01
14         16     1.00000e+00     1.32527e+02     9.24013e+00
15         17     1.00000e+00     1.11340e+02     1.06892e+01
16         18     1.00000e+00     1.05829e+02     2.40035e+01
17         19     1.00000e+00     7.77646e+01     1.14787e+01
18         20     1.00000e+00     6.47514e+01     9.04428e+00
19         24     1.25000e-01     6.28998e+01     8.71543e+00
20         32     7.81250e-03     6.26581e+01     8.68506e+00
21         42     1.95312e-03     6.25575e+01     8.67661e+00
22         53     9.76562e-04     6.25213e+01     8.67219e+00
23         73     1.90735e-06     6.25212e+01     8.67218e+00
24         95     4.76837e-07     6.25212e+01     8.67217e+00
25        118     2.38419e-07     6.25212e+01     8.67217e+00
26        142     1.19209e-07     6.25212e+01     8.67217e+00
27        167     5.96046e-08     6.25212e+01     8.67217e+00
28        194     1.49012e-08     6.25212e+01     8.67217e+00
29        222     7.45058e-09     6.25212e+01     8.67217e+00
30        254     4.65661e-10     6.25212e+01     8.67217e+00
31        288     1.16415e-10     6.25212e+01     8.67217e+00
32        326     0.00000e+00     6.25212e+01     8.67217e+00
Step Size below progTol
Training ordinal logistic classifier
Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     5.80099e-04     1.20021e+03     1.33661e+03
2          3     1.00000e+00     7.97824e+02     9.60976e+02
3          4     1.00000e+00     6.05549e+02     3.56318e+02
4          5     1.00000e+00     5.16553e+02     2.65410e+02
5          6     1.00000e+00     3.73899e+02     1.78636e+02
6          7     1.00000e+00     3.23800e+02     4.56331e+02
7          8     1.00000e+00     2.38817e+02     1.98526e+02
8          9     1.00000e+00     1.87020e+02     5.15878e+01
9         10     1.00000e+00     1.53108e+02     6.96180e+01
10         11     1.00000e+00     1.17863e+02     1.04043e+02
11         12     1.00000e+00     8.43583e+01     1.26589e+02
12         13     1.00000e+00     6.44784e+01     6.13154e+01
13         14     1.00000e+00     5.03957e+01     3.06560e+01
14         15     1.00000e+00     4.08723e+01     2.43359e+01
15         16     1.00000e+00     3.75675e+01     6.16553e+01
16         17     1.00000e+00     2.97615e+01     2.56469e+01
17         18     1.00000e+00     2.63149e+01     1.39195e+01
18         19     1.00000e+00     2.33707e+01     1.37799e+01
19         20     1.00000e+00     1.99202e+01     1.33241e+01
Cubic Backtracking
20         22     2.90055e-01     1.84987e+01     1.92716e+01
21         23     1.00000e+00     1.60015e+01     1.22602e+01
22         24     1.00000e+00     1.41525e+01     9.60725e+00
23         25     1.00000e+00     1.29064e+01     1.17556e+01
24         26     1.00000e+00     1.17360e+01     1.10115e+01
25         27     1.00000e+00     1.13209e+01     1.01109e+01
26         28     1.00000e+00     1.08303e+01     9.44927e+00
27         29     1.00000e+00     1.03740e+01     9.41551e+00
28         30     1.00000e+00     9.83721e+00     8.33298e+00
29         31     1.00000e+00     9.35614e+00     9.25077e+00
30         32     1.00000e+00     8.99625e+00     9.78427e+00
31         33     1.00000e+00     8.52401e+00     1.01430e+01
32         34     1.00000e+00     8.06894e+00     9.62485e+00
33         35     1.00000e+00     6.87749e+00     7.33461e+00
34         36     1.00000e+00     5.90297e+00     5.44936e+00
35         37     1.00000e+00     4.65697e+00     3.79086e+00
36         38     1.00000e+00     3.61993e+00     4.69347e+00
37         39     1.00000e+00     2.56286e+00     6.36885e+00
38         40     1.00000e+00     1.44746e+00     5.52026e+00
39         41     1.00000e+00     5.91079e-01     2.47167e+00
40         42     1.00000e+00     2.92462e-01     1.18089e+00
41         43     1.00000e+00     1.44326e-01     5.46954e-01
42         44     1.00000e+00     7.29289e-02     2.62604e-01
43         45     1.00000e+00     3.68666e-02     1.27048e-01
44         46     1.00000e+00     1.86729e-02     6.18512e-02
45         47     1.00000e+00     9.43570e-03     3.05331e-02
46         48     1.00000e+00     4.75981e-03     1.51698e-02
47         49     1.00000e+00     2.39742e-03     7.72119e-03
48         50     1.00000e+00     1.20658e-03     3.88779e-03
49         51     1.00000e+00     6.07011e-04     1.94112e-03
50         52     1.00000e+00     3.05357e-04     9.62478e-04
51         53     1.00000e+00     1.53628e-04     4.74365e-04
52         54     1.00000e+00     7.73091e-05     2.32521e-04
53         55     1.00000e+00     3.89143e-05     1.13401e-04
54         56     1.00000e+00     1.95930e-05     5.50488e-05
55         57     1.00000e+00     9.86679e-06     2.66103e-05
56         58     1.00000e+00     4.96927e-06     1.28172e-05
57         59     1.00000e+00     2.50263e-06     6.27295e-06
58         60     1.00000e+00     1.26018e-06     3.07341e-06
Directional Derivative below optTol
Training error of multinomial logistic regression: 0.021000
Training error of ordinal logistic regression: 0.000000
```

## Kernel Ordinal Logistic Regression

```% Generate data
nInstances = 500;
nVars = 2;
nClasses = 5;
X = randn(nInstances,nVars);
nExamplePoints = 3;
examplePoints = randn(nExamplePoints,nVars);
thresholds = [0;cumsum(2*rand(nClasses-1,1))];
y = zeros(nInstances,1);
for i = 1:nInstances
dists = sum((repmat(X(i,:),nExamplePoints,1) - examplePoints).^2,2);
y(i,1) = max(find(min(dists) > thresholds));
end
X = [ones(nInstances,1) standardizeCols(X)];

nTrain = nInstances/2;
Xtrain = X(1:nTrain,:);
ytrain = y(1:nTrain);
Xtest = X(nTrain+1:end,:);
ytest = y(nTrain+1:end);

% First try kernel multinomial logistic
sigma = 1;
lambda = 1e-5;
model = classificationKernelSoftmax(Xtrain,ytrain,struct('nClasses',nClasses,'kernelFunc',@kernelRBF,'kernelArgs',sigma,'lambda',lambda));
yhat = model.predictFunc(model,Xtest);
size(yhat)
size(ytest)
testErr_KMLR = sum(yhat~=ytest)/length(ytest);
testDist_KMLR = sum(abs(yhat-ytest))/length(ytest);

% Now try kernel ordinal logistic
Ktrain = kernelRBF(Xtrain,Xtrain,sigma);

% Set up problem
w = zeros(nTrain,1);
gamma = ones(nClasses-2,1);
LB = [-inf(nTrain,1);zeros(nClasses-2,1)];
UB = inf(nTrain+nClasses-2,1);
funObj_sub = @(w)OrdinalLogisticLoss2(w,Ktrain,ytrain,nClasses);
funObj = @(w)penalizedKernelL2_subset(w,Ktrain,1:nTrain,funObj_sub,lambda);

% Solve optiamization
wGamma = minConf_TMP(funObj,[w;gamma],LB,UB);
w = wGamma(1:nTrain);
gamma = [-inf;0;cumsum(wGamma(nTrain+1:end));inf];

% Predict on test data
Ktest = kernelRBF(Xtest,Xtrain,sigma);
z = Ktest*w;
yhat = zeros(size(ytest));
for c = 1:nClasses
yhat(z > gamma(c)) = c;
end
testErr_KOLR = sum(yhat~=ytest)/length(ytest);
testDist_KOLR = sum(abs(yhat-ytest))/length(ytest);

fprintf('Test error of kernel multinomial logistic regression: %f (distance = %f)\n',testErr_KMLR,testDist_KMLR);
fprintf('Test error of kernel ordinal logistic regression: %f (distance = %f)\n',testErr_KOLR,testDist_KOLR);
```
``` Iteration   FunEvals     Step Length    Function Val        Opt Cond
1          3     2.13324e-03     3.54854e+02     8.33044e+00
2          4     1.00000e+00     3.18694e+02     5.00661e+00
3          5     1.00000e+00     2.91184e+02     3.43676e+00
4          7     4.41823e-01     2.79312e+02     5.95776e+00
5          8     1.00000e+00     2.61627e+02     2.99960e+00
6          9     1.00000e+00     2.55434e+02     2.48155e+00
7         10     1.00000e+00     2.49740e+02     2.40483e+00
8         11     1.00000e+00     2.38219e+02     2.71464e+00
9         12     1.00000e+00     2.22439e+02     2.29682e+00
10         13     1.00000e+00     2.02191e+02     1.89955e+00
11         14     1.00000e+00     1.88408e+02     2.23119e+00
12         15     1.00000e+00     1.80382e+02     1.23146e+00
13         16     1.00000e+00     1.77407e+02     9.90214e-01
14         17     1.00000e+00     1.72991e+02     7.49834e-01
15         18     1.00000e+00     1.69072e+02     9.58311e-01
16         19     1.00000e+00     1.63715e+02     1.02299e+00
17         20     1.00000e+00     1.57243e+02     1.16288e+00
18         21     1.00000e+00     1.48634e+02     1.34369e+00
19         22     1.00000e+00     1.43571e+02     2.63967e+00
20         23     1.00000e+00     1.37850e+02     8.31940e-01
21         24     1.00000e+00     1.36567e+02     6.93175e-01
22         25     1.00000e+00     1.32644e+02     7.76764e-01
23         26     1.00000e+00     1.31193e+02     6.92903e-01
24         27     1.00000e+00     1.29112e+02     5.46202e-01
25         28     1.00000e+00     1.27049e+02     4.39825e-01
26         29     1.00000e+00     1.25549e+02     6.00014e-01
27         30     1.00000e+00     1.23622e+02     1.00955e+00
28         31     1.00000e+00     1.20992e+02     1.23773e+00
29         32     1.00000e+00     1.16444e+02     1.06610e+00
30         33     1.00000e+00     1.12199e+02     4.46398e-01
31         34     1.00000e+00     1.09975e+02     4.64400e-01
32         35     1.00000e+00     1.08262e+02     5.00294e-01
33         36     1.00000e+00     1.06224e+02     5.84955e-01
34         37     1.00000e+00     1.03314e+02     6.14773e-01
35         38     1.00000e+00     9.99716e+01     6.43601e-01
36         39     1.00000e+00     9.65581e+01     6.11731e-01
37         40     1.00000e+00     9.26869e+01     4.34685e-01
38         41     1.00000e+00     9.00205e+01     4.89075e-01
39         42     1.00000e+00     8.81637e+01     4.26777e-01
40         43     1.00000e+00     8.55292e+01     4.81282e-01
41         44     1.00000e+00     8.33355e+01     5.30098e-01
42         45     1.00000e+00     8.05128e+01     5.88957e-01
43         46     1.00000e+00     7.80316e+01     5.90707e-01
44         47     1.00000e+00     7.60752e+01     5.23750e-01
45         48     1.00000e+00     7.49316e+01     3.28400e-01
46         49     1.00000e+00     7.38952e+01     2.25505e-01
47         50     1.00000e+00     7.30590e+01     2.10095e-01
48         51     1.00000e+00     7.24767e+01     2.31355e-01
49         52     1.00000e+00     7.19834e+01     2.59055e-01
50         53     1.00000e+00     7.11627e+01     2.83822e-01
51         54     1.00000e+00     6.98997e+01     3.32865e-01
52         55     1.00000e+00     6.80022e+01     4.46002e-01
53         56     1.00000e+00     6.64773e+01     3.24199e-01
54         57     1.00000e+00     6.55106e+01     2.90727e-01
55         58     1.00000e+00     6.48317e+01     2.73925e-01
56         59     1.00000e+00     6.43246e+01     2.28600e-01
57         60     1.00000e+00     6.35772e+01     1.96386e-01
58         61     1.00000e+00     6.29918e+01     2.27184e-01
59         62     1.00000e+00     6.23547e+01     3.18440e-01
60         63     1.00000e+00     6.17582e+01     2.76975e-01
61         64     1.00000e+00     6.08353e+01     2.86630e-01
62         65     1.00000e+00     5.97196e+01     2.04173e-01
63         66     1.00000e+00     5.89917e+01     1.48082e-01
64         67     1.00000e+00     5.85154e+01     1.47877e-01
65         68     1.00000e+00     5.82248e+01     1.39592e-01
66         69     1.00000e+00     5.79756e+01     1.36212e-01
67         70     1.00000e+00     5.76751e+01     1.27488e-01
68         71     1.00000e+00     5.71578e+01     1.63947e-01
69         72     1.00000e+00     5.65886e+01     2.11440e-01
70         73     1.00000e+00     5.59795e+01     1.77772e-01
71         74     1.00000e+00     5.53545e+01     1.94517e-01
72         75     1.00000e+00     5.47410e+01     2.08660e-01
73         76     1.00000e+00     5.41624e+01     2.09681e-01
74         77     1.00000e+00     5.34943e+01     2.11299e-01
75         78     1.00000e+00     5.28778e+01     1.98703e-01
76         79     1.00000e+00     5.24042e+01     1.87274e-01
77         80     1.00000e+00     5.19629e+01     1.68527e-01
78         81     1.00000e+00     5.14987e+01     1.45424e-01
79         82     1.00000e+00     5.10795e+01     1.99074e-01
80         83     1.00000e+00     5.07154e+01     1.65776e-01
81         84     1.00000e+00     5.05044e+01     1.51417e-01
82         85     1.00000e+00     5.01888e+01     1.30166e-01
83         86     1.00000e+00     4.98565e+01     1.21368e-01
84         87     1.00000e+00     4.95596e+01     1.26365e-01
85         88     1.00000e+00     4.93135e+01     1.08510e-01
86         89     1.00000e+00     4.90967e+01     1.32252e-01
87         90     1.00000e+00     4.89327e+01     6.75589e-02
88         91     1.00000e+00     4.88255e+01     4.64096e-02
89         92     1.00000e+00     4.87354e+01     6.86643e-02
90         93     1.00000e+00     4.86320e+01     7.94012e-02
91         94     1.00000e+00     4.84475e+01     1.12847e-01
92         95     1.00000e+00     4.81182e+01     1.27913e-01
93         96     1.00000e+00     4.76946e+01     1.01822e-01
94         97     1.00000e+00     4.72271e+01     1.54352e-01
95         98     1.00000e+00     4.67618e+01     1.40412e-01
96         99     1.00000e+00     4.65337e+01     1.41989e-01
97        100     1.00000e+00     4.61213e+01     1.05646e-01
98        101     1.00000e+00     4.56980e+01     1.19658e-01
99        102     1.00000e+00     4.54045e+01     1.58827e-01
100        103     1.00000e+00     4.49888e+01     1.47646e-01
101        104     1.00000e+00     4.47842e+01     1.00358e-01
102        105     1.00000e+00     4.46401e+01     7.24773e-02
103        106     1.00000e+00     4.44948e+01     6.02935e-02
104        107     1.00000e+00     4.43679e+01     1.16092e-01
105        108     1.00000e+00     4.42176e+01     1.54380e-01
106        109     1.00000e+00     4.40809e+01     1.64771e-01
107        110     1.00000e+00     4.39516e+01     2.27888e-01
108        111     1.00000e+00     4.38029e+01     7.98124e-02
109        112     1.00000e+00     4.36656e+01     1.05189e-01
110        113     1.00000e+00     4.35948e+01     1.17858e-01
111        114     1.00000e+00     4.35087e+01     1.34100e-01
112        116     4.75793e-01     4.34494e+01     1.57602e-01
113        117     1.00000e+00     4.33641e+01     9.48323e-02
114        118     1.00000e+00     4.32502e+01     9.36612e-02
115        119     1.00000e+00     4.31783e+01     8.55612e-02
116        120     1.00000e+00     4.30811e+01     1.01411e-01
117        121     1.00000e+00     4.29434e+01     8.03430e-02
118        122     1.00000e+00     4.28102e+01     9.30981e-02
119        123     1.00000e+00     4.26689e+01     1.01123e-01
120        124     1.00000e+00     4.24702e+01     5.04561e-02
121        125     1.00000e+00     4.23408e+01     6.80296e-02
122        126     1.00000e+00     4.23262e+01     1.63732e-01
123        127     1.00000e+00     4.22512e+01     7.79125e-02
124        128     1.00000e+00     4.22225e+01     5.18407e-02
125        129     1.00000e+00     4.21794e+01     4.31252e-02
126        130     1.00000e+00     4.21503e+01     3.25736e-02
127        131     1.00000e+00     4.21167e+01     3.18588e-02
128        132     1.00000e+00     4.20949e+01     9.30823e-02
129        133     1.00000e+00     4.20300e+01     7.71008e-02
130        134     1.00000e+00     4.19062e+01     7.22154e-02
131        135     1.00000e+00     4.16971e+01     8.39423e-02
132        136     1.00000e+00     4.13794e+01     1.17511e-01
133        137     1.00000e+00     4.09690e+01     1.22768e-01
134        139     1.69431e-01     4.08770e+01     2.29873e-01
135        140     1.00000e+00     4.04544e+01     1.27484e-01
136        141     1.00000e+00     4.01541e+01     6.10053e-02
137        142     1.00000e+00     3.99784e+01     6.04868e-02
138        143     1.00000e+00     3.99036e+01     8.45610e-02
139        144     1.00000e+00     3.97892e+01     5.97570e-02
140        145     1.00000e+00     3.96635e+01     9.30054e-02
141        146     1.00000e+00     3.94677e+01     1.15860e-01
142        147     1.00000e+00     3.91579e+01     1.29154e-01
143        148     1.00000e+00     3.84125e+01     1.40913e-01
144        149     1.00000e+00     3.81053e+01     1.47753e-01
145        150     1.00000e+00     3.76143e+01     1.29875e-01
146        152     3.62470e-01     3.73281e+01     9.44510e-02
147        153     1.00000e+00     3.70085e+01     6.88468e-02
148        154     1.00000e+00     3.68434e+01     5.20709e-02
149        155     1.00000e+00     3.67717e+01     1.21176e-01
150        156     1.00000e+00     3.66803e+01     7.10954e-02
151        157     1.00000e+00     3.66114e+01     4.47466e-02
152        158     1.00000e+00     3.65334e+01     5.53025e-02
153        159     1.00000e+00     3.64045e+01     8.20290e-02
154        160     1.00000e+00     3.61355e+01     1.12661e-01
155        161     1.00000e+00     3.55249e+01     1.17523e-01
156        162     1.00000e+00     3.46061e+01     1.62455e-01
157        164     5.68194e-01     3.43029e+01     1.66512e-01
158        165     1.00000e+00     3.40478e+01     7.70333e-02
159        166     1.00000e+00     3.39672e+01     3.06119e-02
160        167     1.00000e+00     3.38942e+01     4.20252e-02
161        168     1.00000e+00     3.38093e+01     6.39820e-02
162        169     1.00000e+00     3.37452e+01     5.29902e-02
163        170     1.00000e+00     3.36923e+01     4.76744e-02
164        171     1.00000e+00     3.36580e+01     3.44912e-02
165        172     1.00000e+00     3.36378e+01     3.30893e-02
166        173     1.00000e+00     3.36079e+01     3.42213e-02
167        174     1.00000e+00     3.35744e+01     4.45973e-02
168        175     1.00000e+00     3.35207e+01     3.22907e-02
169        176     1.00000e+00     3.34506e+01     4.31404e-02
170        177     1.00000e+00     3.33617e+01     3.37945e-02
171        178     1.00000e+00     3.32406e+01     4.94269e-02
172        179     1.00000e+00     3.30708e+01     7.80331e-02
173        180     1.00000e+00     3.28954e+01     7.83057e-02
174        181     1.00000e+00     3.27255e+01     8.26793e-02
175        182     1.00000e+00     3.25748e+01     6.46835e-02
176        183     1.00000e+00     3.24279e+01     4.34906e-02
177        184     1.00000e+00     3.23623e+01     3.86816e-02
178        185     1.00000e+00     3.22893e+01     3.65844e-02
179        186     1.00000e+00     3.22372e+01     3.63775e-02
180        187     1.00000e+00     3.21410e+01     3.67822e-02
181        188     1.00000e+00     3.20466e+01     4.47560e-02
182        189     1.00000e+00     3.19515e+01     5.15427e-02
183        190     1.00000e+00     3.17619e+01     4.84240e-02
184        191     1.00000e+00     3.15260e+01     5.75948e-02
185        192     1.00000e+00     3.13484e+01     4.88022e-02
186        193     1.00000e+00     3.11844e+01     9.80825e-02
187        194     1.00000e+00     3.10681e+01     3.76497e-02
188        195     1.00000e+00     3.10222e+01     2.04167e-02
189        196     1.00000e+00     3.09716e+01     2.02444e-02
190        197     1.00000e+00     3.09225e+01     1.98161e-02
191        198     1.00000e+00     3.09015e+01     7.02329e-02
192        199     1.00000e+00     3.08594e+01     2.38482e-02
193        200     1.00000e+00     3.08375e+01     1.18977e-02
194        201     1.00000e+00     3.08214e+01     1.22985e-02
195        202     1.00000e+00     3.07969e+01     3.04791e-02
196        203     1.00000e+00     3.07690e+01     1.97455e-02
197        204     1.00000e+00     3.07357e+01     2.75860e-02
198        205     1.00000e+00     3.06800e+01     3.29343e-02
199        206     1.00000e+00     3.06118e+01     3.72502e-02
200        207     1.00000e+00     3.05165e+01     4.53003e-02
201        208     1.00000e+00     3.04566e+01     7.14695e-02
202        209     1.00000e+00     3.03769e+01     2.23796e-02
203        210     1.00000e+00     3.03322e+01     2.22776e-02
204        211     1.00000e+00     3.03202e+01     1.52094e-02
205        212     1.00000e+00     3.03151e+01     1.18358e-02
206        213     1.00000e+00     3.03137e+01     1.69261e-02
207        214     1.00000e+00     3.03056e+01     9.14823e-03
208        215     1.00000e+00     3.02841e+01     2.29854e-02
209        216     1.00000e+00     3.02621e+01     5.54825e-02
210        217     1.00000e+00     3.02101e+01     5.03800e-02
211        219     4.95752e-02     3.02033e+01     1.27666e-01
212        220     1.00000e+00     3.01568e+01     7.23437e-02
213        221     1.00000e+00     3.01094e+01     2.95132e-02
214        222     1.00000e+00     3.00693e+01     5.00521e-02
215        223     1.00000e+00     3.00057e+01     7.41982e-02
216        224     1.00000e+00     2.98973e+01     1.02199e-01
217        225     1.00000e+00     2.98465e+01     2.03804e-01
218        226     1.00000e+00     2.96591e+01     3.39804e-02
219        227     1.00000e+00     2.95613e+01     6.30799e-02
220        228     1.00000e+00     2.94821e+01     4.00120e-02
221        229     1.00000e+00     2.94658e+01     2.28889e-02
222        230     1.00000e+00     2.94520e+01     1.38365e-02
223        232     2.86999e-01     2.94481e+01     2.02530e-02
224        233     1.00000e+00     2.94449e+01     9.53947e-03
225        234     1.00000e+00     2.94430e+01     1.03153e-02
226        235     1.00000e+00     2.94415e+01     1.14361e-02
227        236     1.00000e+00     2.94386e+01     9.99272e-03
228        237     1.00000e+00     2.94368e+01     1.55556e-02
229        238     1.00000e+00     2.94345e+01     8.48339e-03
230        239     1.00000e+00     2.94320e+01     1.10855e-02
231        240     1.00000e+00     2.94300e+01     1.40278e-02
232        241     1.00000e+00     2.94246e+01     2.02731e-02
233        242     1.00000e+00     2.94132e+01     2.28292e-02
234        243     1.00000e+00     2.93902e+01     2.62066e-02
235        244     1.00000e+00     2.93405e+01     1.87045e-02
236        245     1.00000e+00     2.92423e+01     3.21045e-02
237        246     1.00000e+00     2.90070e+01     7.85549e-02
238        248     5.59840e-01     2.88402e+01     8.90409e-02
239        250     1.27368e-01     2.87612e+01     1.97536e-01
240        251     1.00000e+00     2.84379e+01     9.33830e-02
241        252     1.00000e+00     2.81936e+01     3.07257e-02
242        253     1.00000e+00     2.81112e+01     2.33671e-02
243        255     2.85376e-01     2.80812e+01     2.96049e-02
244        256     1.00000e+00     2.80615e+01     2.54678e-02
245        257     1.00000e+00     2.80527e+01     2.72494e-02
246        258     1.00000e+00     2.80453e+01     2.71344e-02
247        259     1.00000e+00     2.80384e+01     2.30478e-02
248        260     1.00000e+00     2.80292e+01     2.54061e-02
249        261     1.00000e+00     2.80153e+01     2.13541e-02
250        262     1.00000e+00     2.79842e+01     3.61825e-02
251        263     1.00000e+00     2.79340e+01     3.99559e-02
252        264     1.00000e+00     2.78582e+01     6.76647e-02
253        265     1.00000e+00     2.77547e+01     5.93795e-02
254        266     1.00000e+00     2.76246e+01     6.92120e-02
255        268     5.31821e-01     2.75487e+01     1.34255e-01
256        269     1.00000e+00     2.74509e+01     4.94029e-02
257        270     1.00000e+00     2.73987e+01     5.64874e-02
258        271     1.00000e+00     2.73168e+01     1.44504e-02
259        272     1.00000e+00     2.72734e+01     2.11049e-02
260        273     1.00000e+00     2.72532e+01     1.72707e-02
261        274     1.00000e+00     2.72471e+01     1.55459e-02
262        275     1.00000e+00     2.72343e+01     1.19456e-02
263        276     1.00000e+00     2.72299e+01     8.79717e-03
264        277     1.00000e+00     2.72243e+01     8.55031e-03
265        278     1.00000e+00     2.72173e+01     8.83136e-03
266        279     1.00000e+00     2.72063e+01     1.00044e-02
267        280     1.00000e+00     2.71862e+01     1.20220e-02
268        281     1.00000e+00     2.71625e+01     8.90839e-03
269        282     1.00000e+00     2.71219e+01     9.70785e-03
270        283     1.00000e+00     2.70673e+01     1.87904e-02
271        284     1.00000e+00     2.70103e+01     1.78864e-02
272        285     1.00000e+00     2.69360e+01     2.72790e-02
273        286     1.00000e+00     2.68905e+01     5.43844e-02
274        287     1.00000e+00     2.68466e+01     2.99264e-02
275        288     1.00000e+00     2.68115e+01     2.59628e-02
276        289     1.00000e+00     2.67862e+01     3.23528e-02
277        290     1.00000e+00     2.67679e+01     2.08323e-02
278        291     1.00000e+00     2.67543e+01     9.08080e-03
279        292     1.00000e+00     2.67483e+01     9.91359e-03
280        293     1.00000e+00     2.67438e+01     8.96554e-03
281        294     1.00000e+00     2.67403e+01     6.00181e-03
282        295     1.00000e+00     2.67366e+01     6.91457e-03
283        296     1.00000e+00     2.67329e+01     7.70078e-03
284        297     1.00000e+00     2.67266e+01     9.47308e-03
285        298     1.00000e+00     2.67184e+01     9.24212e-03
286        300     4.15455e-01     2.67100e+01     1.09366e-02
287        301     1.00000e+00     2.66846e+01     1.73159e-02
288        302     1.00000e+00     2.66366e+01     7.29436e-03
289        303     1.00000e+00     2.65750e+01     2.13661e-02
290        304     1.00000e+00     2.64902e+01     2.70449e-02
291        305     1.00000e+00     2.64255e+01     3.22887e-02
292        306     1.00000e+00     2.63627e+01     3.35494e-02
293        307     1.00000e+00     2.63064e+01     1.19987e-02
294        308     1.00000e+00     2.62610e+01     2.10776e-02
295        309     1.00000e+00     2.62295e+01     4.81601e-02
296        310     1.00000e+00     2.62076e+01     1.58605e-02
297        311     1.00000e+00     2.61927e+01     8.59292e-03
298        312     1.00000e+00     2.61854e+01     1.57499e-02
299        313     1.00000e+00     2.61812e+01     1.18851e-02
300        314     1.00000e+00     2.61777e+01     7.06455e-03
301        315     1.00000e+00     2.61756e+01     7.58195e-03
302        316     1.00000e+00     2.61738e+01     7.94838e-03
303        317     1.00000e+00     2.61711e+01     8.19341e-03
304        318     1.00000e+00     2.61659e+01     8.64237e-03
305        319     1.00000e+00     2.61556e+01     9.26629e-03
306        321     5.22271e-01     2.61472e+01     4.49733e-02
307        322     1.00000e+00     2.61241e+01     2.89460e-02
308        323     1.00000e+00     2.60425e+01     5.46943e-02
309        324     1.00000e+00     2.59595e+01     8.44463e-02
310        325     1.00000e+00     2.57956e+01     1.07032e-01
311        326     1.00000e+00     2.57362e+01     5.94891e-02
312        327     1.00000e+00     2.56807e+01     9.00210e-02
313        329     3.39744e-01     2.56640e+01     2.77529e-02
314        330     1.00000e+00     2.56445e+01     2.28717e-02
315        331     1.00000e+00     2.56336e+01     2.65292e-02
316        332     1.00000e+00     2.56223e+01     9.18636e-03
317        333     1.00000e+00     2.56180e+01     6.99928e-03
318        334     1.00000e+00     2.56149e+01     7.73560e-03
319        335     1.00000e+00     2.56141e+01     9.47073e-03
320        336     1.00000e+00     2.56120e+01     6.77255e-03
321        337     1.00000e+00     2.56103e+01     6.26592e-03
322        338     1.00000e+00     2.56074e+01     5.87769e-03
323        339     1.00000e+00     2.56041e+01     6.43137e-03
324        340     1.00000e+00     2.55991e+01     8.38271e-03
325        341     1.00000e+00     2.55877e+01     1.27723e-02
326        342     1.00000e+00     2.55721e+01     3.20210e-02
327        343     1.00000e+00     2.55503e+01     1.02927e-02
328        344     1.00000e+00     2.55071e+01     1.60746e-02
329        345     1.00000e+00     2.54514e+01     3.24633e-02
330        346     1.00000e+00     2.53793e+01     3.88586e-02
331        348     2.13924e-01     2.53624e+01     4.96568e-02
332        349     1.00000e+00     2.52949e+01     2.36512e-02
333        350     1.00000e+00     2.52463e+01     1.91633e-02
334        351     1.00000e+00     2.52241e+01     1.45881e-02
335        352     1.00000e+00     2.52171e+01     1.37720e-02
336        353     1.00000e+00     2.52086e+01     1.20246e-02
337        354     1.00000e+00     2.52038e+01     4.76758e-03
338        355     1.00000e+00     2.52007e+01     3.77340e-03
339        356     1.00000e+00     2.51983e+01     3.87452e-03
340        357     1.00000e+00     2.51980e+01     2.26410e-02
341        358     1.00000e+00     2.51959e+01     1.07287e-02
342        359     1.00000e+00     2.51940e+01     3.67110e-03
343        360     1.00000e+00     2.51909e+01     1.06478e-02
344        361     1.00000e+00     2.51859e+01     1.65802e-02
345        362     1.00000e+00     2.51763e+01     1.80168e-02
346        363     1.00000e+00     2.51552e+01     1.70757e-02
347        364     1.00000e+00     2.51139e+01     2.37655e-02
348        365     1.00000e+00     2.50747e+01     3.73052e-02
349        366     1.00000e+00     2.50021e+01     1.92621e-02
350        367     1.00000e+00     2.49101e+01     3.41625e-02
351        368     1.00000e+00     2.48541e+01     2.62886e-02
352        370     4.01985e-01     2.48240e+01     1.64217e-02
353        371     1.00000e+00     2.47781e+01     1.68553e-02
354        372     1.00000e+00     2.47243e+01     6.66864e-02
355        373     1.00000e+00     2.46615e+01     3.83054e-02
356        374     1.00000e+00     2.46023e+01     2.65901e-02
357        375     1.00000e+00     2.45768e+01     2.31162e-02
358        376     1.00000e+00     2.45608e+01     1.37715e-02
359        377     1.00000e+00     2.45594e+01     2.60802e-02
360        378     1.00000e+00     2.45495e+01     1.48365e-02
361        379     1.00000e+00     2.45447e+01     5.91351e-03
362        380     1.00000e+00     2.45362e+01     1.43631e-02
363        381     1.00000e+00     2.45306e+01     1.98067e-02
364        382     1.00000e+00     2.45242e+01     1.11928e-02
365        383     1.00000e+00     2.45204e+01     1.02427e-02
366        384     1.00000e+00     2.45179e+01     9.44899e-03
367        385     1.00000e+00     2.45168e+01     7.02636e-03
368        386     1.00000e+00     2.45153e+01     5.33594e-03
369        387     1.00000e+00     2.45133e+01     6.13122e-03
370        388     1.00000e+00     2.45105e+01     8.96674e-03
371        389     1.00000e+00     2.45053e+01     8.64318e-03
372        390     1.00000e+00     2.44947e+01     2.11485e-02
373        391     1.00000e+00     2.44797e+01     1.37217e-02
374        392     1.00000e+00     2.44732e+01     2.31176e-02
375        393     1.00000e+00     2.44657e+01     8.57644e-03
376        395     3.10863e-01     2.44625e+01     2.04165e-02
377        396     1.00000e+00     2.44590e+01     6.93848e-03
378        397     1.00000e+00     2.44571e+01     5.46501e-03
379        398     1.00000e+00     2.44555e+01     7.06469e-03
380        399     1.00000e+00     2.44526e+01     5.68369e-03
381        400     1.00000e+00     2.44483e+01     7.55429e-03
382        401     1.00000e+00     2.44388e+01     1.76874e-02
383        402     1.00000e+00     2.44235e+01     3.22140e-02
384        403     1.00000e+00     2.43946e+01     4.66094e-02
385        404     1.00000e+00     2.43517e+01     4.83179e-02
386        405     1.00000e+00     2.42904e+01     4.19260e-02
387        407     5.15645e-01     2.42537e+01     5.32171e-02
388        408     1.00000e+00     2.41914e+01     4.24810e-02
389        409     1.00000e+00     2.41189e+01     2.64852e-02
390        410     1.00000e+00     2.40669e+01     3.00664e-02
391        411     1.00000e+00     2.40352e+01     3.17958e-02
392        412     1.00000e+00     2.39943e+01     4.99683e-02
393        413     1.00000e+00     2.39774e+01     2.24926e-02
394        414     1.00000e+00     2.39702e+01     1.58115e-02
395        415     1.00000e+00     2.39651e+01     7.89845e-03
396        416     1.00000e+00     2.39641e+01     8.85209e-03
397        417     1.00000e+00     2.39622e+01     8.28971e-03
398        418     1.00000e+00     2.39618e+01     8.31722e-03
399        419     1.00000e+00     2.39601e+01     8.75020e-03
400        420     1.00000e+00     2.39593e+01     8.75963e-03
401        422     3.00872e-01     2.39577e+01     2.26824e-02
402        423     1.00000e+00     2.39543e+01     1.83850e-02
403        424     1.00000e+00     2.39466e+01     8.56938e-03
404        425     1.00000e+00     2.39319e+01     1.19659e-02
405        426     1.00000e+00     2.39068e+01     2.86816e-02
406        427     1.00000e+00     2.38684e+01     4.83226e-02
407        428     1.00000e+00     2.38273e+01     3.66681e-02
408        430     5.30141e-01     2.38042e+01     6.20594e-02
409        431     1.00000e+00     2.37729e+01     1.75063e-02
410        432     1.00000e+00     2.37507e+01     1.44546e-02
411        433     1.00000e+00     2.37431e+01     1.35318e-02
412        434     1.00000e+00     2.37416e+01     1.29760e-02
413        435     1.00000e+00     2.37376e+01     3.60228e-03
414        436     1.00000e+00     2.37370e+01     1.97760e-03
415        437     1.00000e+00     2.37367e+01     6.28328e-03
416        438     1.00000e+00     2.37367e+01     8.77765e-03
417        439     1.00000e+00     2.37364e+01     1.71496e-03
418        440     1.00000e+00     2.37364e+01     1.74979e-03
419        441     1.00000e+00     2.37363e+01     1.62166e-03
420        442     1.00000e+00     2.37359e+01     1.92984e-03
421        444     2.15508e-01     2.37356e+01     2.86346e-03
422        445     1.00000e+00     2.37346e+01     2.72545e-03
423        446     1.00000e+00     2.37309e+01     7.44360e-03
424        447     1.00000e+00     2.37235e+01     7.92363e-03
425        448     1.00000e+00     2.37063e+01     1.55918e-02
426        449     1.00000e+00     2.36659e+01     2.24012e-02
427        450     1.00000e+00     2.36130e+01     2.73267e-02
428        452     2.03018e-01     2.36031e+01     2.24518e-02
429        453     1.00000e+00     2.35560e+01     7.11714e-03
430        454     1.00000e+00     2.35352e+01     1.05376e-02
431        455     1.00000e+00     2.35259e+01     4.46291e-03
432        456     1.00000e+00     2.35218e+01     4.93204e-03
433        458     5.11905e-01     2.35207e+01     4.02000e-03
434        459     1.00000e+00     2.35195e+01     4.43380e-03
435        460     1.00000e+00     2.35185e+01     4.50371e-03
436        461     1.00000e+00     2.35175e+01     3.91763e-03
437        462     1.00000e+00     2.35161e+01     3.39180e-03
438        463     1.00000e+00     2.35138e+01     5.82242e-03
439        464     1.00000e+00     2.35084e+01     1.54905e-02
440        465     1.00000e+00     2.34972e+01     2.52770e-02
441        466     1.00000e+00     2.34747e+01     3.24918e-02
442        467     1.00000e+00     2.34366e+01     2.45183e-02
443        468     1.00000e+00     2.33659e+01     3.98665e-02
444        470     1.82663e-01     2.33488e+01     2.17608e-02
445        471     1.00000e+00     2.32616e+01     1.24277e-02
446        472     1.00000e+00     2.31905e+01     1.18941e-02
447        473     1.00000e+00     2.31728e+01     1.88516e-02
448        474     1.00000e+00     2.31560e+01     1.87211e-02
449        475     1.00000e+00     2.31516e+01     5.55212e-03
450        476     1.00000e+00     2.31499e+01     4.16077e-03
451        477     1.00000e+00     2.31487e+01     3.16337e-03
452        478     1.00000e+00     2.31471e+01     2.99031e-03
453        479     1.00000e+00     2.31456e+01     3.00954e-03
454        480     1.00000e+00     2.31443e+01     3.68186e-03
455        481     1.00000e+00     2.31437e+01     7.25491e-03
456        482     1.00000e+00     2.31423e+01     3.43972e-03
457        483     1.00000e+00     2.31412e+01     3.04449e-03
458        484     1.00000e+00     2.31389e+01     5.87739e-03
459        485     1.00000e+00     2.31356e+01     1.05838e-02
460        486     1.00000e+00     2.31320e+01     2.04623e-02
461        487     1.00000e+00     2.31257e+01     1.19428e-02
462        488     1.00000e+00     2.31126e+01     7.76018e-03
463        489     1.00000e+00     2.30948e+01     1.64888e-02
464        490     1.00000e+00     2.30651e+01     2.37466e-02
465        491     1.00000e+00     2.30285e+01     1.71763e-02
466        493     3.78106e-01     2.30130e+01     4.33541e-02
467        494     1.00000e+00     2.29777e+01     1.13696e-02
468        495     1.00000e+00     2.29629e+01     9.33313e-03
469        496     1.00000e+00     2.29600e+01     5.30803e-03
470        498     4.19377e-01     2.29590e+01     3.79786e-03
471        499     1.00000e+00     2.29584e+01     2.48131e-03
472        500     1.00000e+00     2.29575e+01     2.71134e-03
473        502     4.90488e-01     2.29570e+01     4.82526e-03
474        503     1.00000e+00     2.29562e+01     2.50364e-03
475        504     1.00000e+00     2.29555e+01     1.93177e-03
476        505     1.00000e+00     2.29550e+01     2.25900e-03
477        507     1.96156e-01     2.29548e+01     4.27373e-03
478        508     1.00000e+00     2.29544e+01     3.02387e-03
479        509     1.00000e+00     2.29538e+01     3.88725e-03
480        510     1.00000e+00     2.29532e+01     2.72546e-03
481        511     1.00000e+00     2.29519e+01     3.23956e-03
482        512     1.00000e+00     2.29505e+01     2.81186e-03
483        513     1.00000e+00     2.29488e+01     4.71549e-03
484        514     1.00000e+00     2.29455e+01     6.11357e-03
485        515     1.00000e+00     2.29400e+01     8.49084e-03
486        516     1.00000e+00     2.29352e+01     2.27402e-02
487        517     1.00000e+00     2.29165e+01     1.79676e-02
488        518     1.00000e+00     2.28891e+01     1.20347e-02
489        519     1.00000e+00     2.28387e+01     1.37714e-02
490        521     2.42526e-02     2.28363e+01     1.59818e-02
491        522     1.00000e+00     2.28087e+01     1.03203e-02
492        523     1.00000e+00     2.27943e+01     1.01622e-02
493        525     3.51512e-01     2.27865e+01     1.18775e-02
494        526     1.00000e+00     2.27850e+01     1.33197e-02
495        527     1.00000e+00     2.27706e+01     6.48670e-03
496        528     1.00000e+00     2.27584e+01     9.50515e-03
497        529     1.00000e+00     2.27488e+01     3.77761e-03
498        530     1.00000e+00     2.27413e+01     2.82148e-03
499        531     1.00000e+00     2.27387e+01     3.03526e-03
500        532     1.00000e+00     2.27384e+01     1.09332e-02
Reached Maximum Number of Iterations

ans =

250     1

ans =

250     1

Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     4.95326e-04     4.14754e+02     1.61957e+03
2          3     1.00000e+00     3.56426e+02     1.67151e+03
3          4     1.00000e+00     3.36207e+02     1.19204e+03
4          5     1.00000e+00     3.06043e+02     3.91437e+02
5          6     1.00000e+00     2.95438e+02     7.53368e+02
6          7     1.00000e+00     2.81971e+02     5.13258e+02
7          8     1.00000e+00     2.58591e+02     5.05433e+02
8          9     1.00000e+00     2.30304e+02     2.67402e+02
9         10     1.00000e+00     2.23343e+02     1.39943e+02
10         11     1.00000e+00     2.19976e+02     1.17064e+02
11         12     1.00000e+00     2.13990e+02     1.53892e+02
12         13     1.00000e+00     2.02760e+02     1.92951e+02
13         14     1.00000e+00     1.90914e+02     1.58021e+02
14         15     1.00000e+00     1.83917e+02     8.03570e+01
15         16     1.00000e+00     1.79728e+02     8.95175e+01
16         17     1.00000e+00     1.76159e+02     1.08320e+02
17         18     1.00000e+00     1.70111e+02     1.21786e+02
18         19     1.00000e+00     1.56963e+02     9.89316e+01
19         20     1.00000e+00     1.46553e+02     1.09969e+02
20         21     1.00000e+00     1.35878e+02     1.47104e+02
21         22     1.00000e+00     1.25182e+02     4.42663e+01
22         23     1.00000e+00     1.18903e+02     6.72439e+01
23         24     1.00000e+00     1.15247e+02     5.73144e+01
24         25     1.00000e+00     1.11739e+02     1.13761e+02
25         26     1.00000e+00     1.09110e+02     8.20114e+01
26         27     1.00000e+00     1.06807e+02     6.08763e+01
27         28     1.00000e+00     1.04876e+02     3.18232e+01
28         29     1.00000e+00     1.04155e+02     3.32192e+01
29         30     1.00000e+00     1.03617e+02     3.44535e+01
30         31     1.00000e+00     1.03045e+02     3.45866e+01
31         32     1.00000e+00     1.01982e+02     3.65316e+01
32         33     1.00000e+00     1.00041e+02     4.01535e+01
33         34     1.00000e+00     9.70583e+01     2.47913e+01
34         35     1.00000e+00     9.47439e+01     1.82661e+01
35         36     1.00000e+00     9.30508e+01     3.67281e+01
36         37     1.00000e+00     9.18658e+01     2.42113e+01
37         38     1.00000e+00     9.10988e+01     2.14335e+01
38         39     1.00000e+00     8.98613e+01     2.06537e+01
39         40     1.00000e+00     8.88808e+01     1.80946e+01
40         41     1.00000e+00     8.77843e+01     3.17394e+01
41         42     1.00000e+00     8.68548e+01     3.18080e+01
42         43     1.00000e+00     8.57145e+01     2.41548e+01
43         44     1.00000e+00     8.48271e+01     1.16200e+01
44         45     1.00000e+00     8.41132e+01     1.91924e+01
45         46     1.00000e+00     8.31875e+01     2.76160e+01
46         47     1.00000e+00     8.19785e+01     3.13311e+01
47         48     1.00000e+00     8.00798e+01     2.67228e+01
48         49     1.00000e+00     7.84550e+01     2.35401e+01
49         50     1.00000e+00     7.76431e+01     1.29071e+01
50         51     1.00000e+00     7.73034e+01     8.45101e+00
51         52     1.00000e+00     7.69885e+01     2.05713e+01
52         53     1.00000e+00     7.64413e+01     3.60059e+01
53         54     1.00000e+00     7.59750e+01     3.20636e+01
54         55     1.00000e+00     7.52687e+01     9.25764e+00
55         56     1.00000e+00     7.47837e+01     1.72501e+01
56         57     1.00000e+00     7.44123e+01     2.24122e+01
57         58     1.00000e+00     7.37758e+01     2.40236e+01
58         59     1.00000e+00     7.27080e+01     3.02029e+01
59         60     1.00000e+00     7.11823e+01     3.49535e+01
60         61     1.00000e+00     6.96410e+01     3.83668e+01
61         62     1.00000e+00     6.81612e+01     1.80875e+01
62         63     1.00000e+00     6.72560e+01     1.93715e+01
63         64     1.00000e+00     6.68913e+01     1.53838e+01
64         65     1.00000e+00     6.65676e+01     3.03215e+01
65         66     1.00000e+00     6.62205e+01     2.78809e+01
66         67     1.00000e+00     6.56664e+01     1.38443e+01
67         68     1.00000e+00     6.50261e+01     2.54167e+01
68         69     1.00000e+00     6.45483e+01     4.68627e+01
69         70     1.00000e+00     6.40170e+01     6.21956e+01
70         71     1.00000e+00     6.34461e+01     6.88977e+01
71         72     1.00000e+00     6.26691e+01     6.80799e+01
72         73     1.00000e+00     6.16516e+01     5.65556e+01
73         74     1.00000e+00     6.05219e+01     2.77899e+01
74         75     1.00000e+00     5.99923e+01     1.70344e+01
75         76     1.00000e+00     5.95684e+01     1.20098e+01
76         77     1.00000e+00     5.91300e+01     1.72347e+01
77         78     1.00000e+00     5.87014e+01     2.69104e+01
78         79     1.00000e+00     5.83724e+01     2.96716e+01
79         80     1.00000e+00     5.80703e+01     2.57501e+01
80         81     1.00000e+00     5.76934e+01     2.30180e+01
81         82     1.00000e+00     5.72863e+01     2.21201e+01
82         83     1.00000e+00     5.66472e+01     2.76538e+01
83         84     1.00000e+00     5.57283e+01     3.52021e+01
84         85     1.00000e+00     5.43995e+01     3.60621e+01
85         86     1.00000e+00     5.28790e+01     2.01920e+01
86         87     1.00000e+00     5.20576e+01     4.38487e+00
87         88     1.00000e+00     5.18123e+01     8.49740e+00
88         89     1.00000e+00     5.17055e+01     8.62655e+00
89         90     1.00000e+00     5.15723e+01     5.77641e+00
90         91     1.00000e+00     5.14626e+01     9.25767e+00
91         92     1.00000e+00     5.13832e+01     1.38009e+01
92         93     1.00000e+00     5.12915e+01     1.59360e+01
93         94     1.00000e+00     5.11971e+01     1.22116e+01
94         95     1.00000e+00     5.10980e+01     7.10873e+00
95         96     1.00000e+00     5.09902e+01     3.40927e+00
96         97     1.00000e+00     5.07901e+01     1.17853e+01
97         98     1.00000e+00     5.04495e+01     2.31557e+01
98         99     1.00000e+00     4.98427e+01     3.57057e+01
99        100     1.00000e+00     4.90216e+01     4.16413e+01
100        101     1.00000e+00     4.81607e+01     3.56512e+01
101        102     1.00000e+00     4.72626e+01     1.45926e+01
102        103     1.00000e+00     4.70846e+01     7.88166e+00
103        104     1.00000e+00     4.69956e+01     4.87191e+00
104        105     1.00000e+00     4.69507e+01     3.22198e+00
105        106     1.00000e+00     4.69162e+01     8.09681e+00
106        107     1.00000e+00     4.68818e+01     2.53300e+00
107        108     1.00000e+00     4.68501e+01     9.43511e+00
108        109     1.00000e+00     4.68011e+01     5.25747e+00
109        110     1.00000e+00     4.66884e+01     6.20122e+00
110        111     1.00000e+00     4.64491e+01     1.24893e+01
111        112     1.00000e+00     4.60793e+01     1.16336e+01
112        113     1.00000e+00     4.53511e+01     1.80182e+01
113        114     1.00000e+00     4.37453e+01     1.34056e+01
114        115     1.00000e+00     4.27735e+01     5.37748e+01
115        116     1.00000e+00     4.23030e+01     1.28953e+01
116        117     1.00000e+00     4.19269e+01     4.22004e+01
117        118     1.00000e+00     4.14477e+01     4.54762e+01
118        119     1.00000e+00     4.13729e+01     2.64740e+01
119        120     1.00000e+00     4.09800e+01     1.34432e+01
120        121     1.00000e+00     4.09307e+01     5.21012e+00
121        122     1.00000e+00     4.09028e+01     3.95834e+00
122        123     1.00000e+00     4.08930e+01     4.32013e+00
123        124     1.00000e+00     4.08865e+01     1.75179e+00
124        125     1.00000e+00     4.08821e+01     1.27368e+00
125        126     1.00000e+00     4.08791e+01     1.39728e+00
126        127     1.00000e+00     4.08753e+01     1.04765e+00
127        128     1.00000e+00     4.08636e+01     3.21494e+00
128        129     1.00000e+00     4.08473e+01     5.38778e+00
129        130     1.00000e+00     4.08141e+01     7.29220e+00
130        131     1.00000e+00     4.07636e+01     7.20268e+00
131        132     1.00000e+00     4.06679e+01     4.23810e+00
132        133     1.00000e+00     4.04925e+01     4.68193e+00
133        134     1.00000e+00     4.01828e+01     1.62878e+01
134        135     1.00000e+00     3.97891e+01     2.49668e+01
135        136     1.00000e+00     3.96011e+01     1.29646e+01
136        137     1.00000e+00     3.95133e+01     3.79964e+00
137        138     1.00000e+00     3.94893e+01     2.74573e+00
138        139     1.00000e+00     3.94797e+01     1.77869e+00
139        140     1.00000e+00     3.94743e+01     1.09483e+00
140        141     1.00000e+00     3.94697e+01     8.09017e-01
141        142     1.00000e+00     3.94664e+01     9.77093e-01
142        143     1.00000e+00     3.94614e+01     1.58935e+00
143        144     1.00000e+00     3.94546e+01     2.03134e+00
144        145     1.00000e+00     3.94450e+01     2.79685e+00
145        146     1.00000e+00     3.94340e+01     2.00357e+00
146        147     1.00000e+00     3.94246e+01     1.86606e+00
147        148     1.00000e+00     3.94157e+01     1.49920e+00
148        149     1.00000e+00     3.94074e+01     2.39104e+00
149        150     1.00000e+00     3.93884e+01     4.68522e+00
150        151     1.00000e+00     3.93676e+01     6.01503e+00
151        152     1.00000e+00     3.92953e+01     8.31609e+00
152        153     1.00000e+00     3.91652e+01     9.28238e+00
153        154     1.00000e+00     3.89409e+01     1.03650e+01
Cubic Backtracking
154        156     3.61701e-01     3.88495e+01     7.66182e+00
155        157     1.00000e+00     3.87578e+01     1.92221e+01
156        158     1.00000e+00     3.85470e+01     2.75533e+00
157        159     1.00000e+00     3.85176e+01     1.82391e+00
158        160     1.00000e+00     3.85088e+01     2.75127e+00
159        161     1.00000e+00     3.85058e+01     1.16158e+00
160        162     1.00000e+00     3.85045e+01     4.08239e-01
161        163     1.00000e+00     3.85039e+01     7.21748e-01
162        164     1.00000e+00     3.85032e+01     1.29159e+00
163        165     1.00000e+00     3.85018e+01     1.76088e+00
164        166     1.00000e+00     3.84993e+01     1.65749e+00
165        167     1.00000e+00     3.84931e+01     4.30330e+00
166        168     1.00000e+00     3.84837e+01     3.57073e+00
167        169     1.00000e+00     3.84565e+01     2.08236e+00
168        170     1.00000e+00     3.84214e+01     2.46173e+00
169        171     1.00000e+00     3.83414e+01     5.29836e+00
170        172     1.00000e+00     3.81998e+01     1.11993e+01
171        173     1.00000e+00     3.79270e+01     9.67111e+00
Cubic Backtracking
172        175     5.14302e-01     3.76695e+01     2.00813e+01
173        176     1.00000e+00     3.72911e+01     8.73439e+00
174        177     1.00000e+00     3.70035e+01     1.13620e+01
175        178     1.00000e+00     3.69409e+01     4.35823e+00
Cubic Backtracking
176        180     4.02733e-01     3.69149e+01     2.22771e+00
Cubic Backtracking
177        182     3.49373e-01     3.69043e+01     1.16683e+00
178        183     1.00000e+00     3.69015e+01     2.39658e+00
179        184     1.00000e+00     3.68991e+01     1.03009e+00
180        185     1.00000e+00     3.68984e+01     4.45865e-01
181        186     1.00000e+00     3.68973e+01     7.50619e-01
182        187     1.00000e+00     3.68952e+01     2.08606e+00
183        188     1.00000e+00     3.68907e+01     4.21239e+00
184        189     1.00000e+00     3.68801e+01     7.49473e+00
185        190     1.00000e+00     3.68546e+01     1.24622e+01
186        191     1.00000e+00     3.67933e+01     1.96201e+01
187        192     1.00000e+00     3.66543e+01     2.87125e+01
188        193     1.00000e+00     3.63857e+01     3.63326e+01
189        194     1.00000e+00     3.60008e+01     3.33977e+01
190        195     1.00000e+00     3.57709e+01     1.80371e+01
191        196     1.00000e+00     3.56986e+01     8.52648e+00
192        197     1.00000e+00     3.55945e+01     1.82507e+00
193        198     1.00000e+00     3.55476e+01     3.25718e+00
194        199     1.00000e+00     3.54968e+01     7.29407e-01
195        200     1.00000e+00     3.54876e+01     4.19090e-01
196        201     1.00000e+00     3.54871e+01     1.09819e+00
197        202     1.00000e+00     3.54827e+01     5.15087e-01
198        203     1.00000e+00     3.54817e+01     4.76339e-01
199        204     1.00000e+00     3.54773e+01     2.15709e+00
200        205     1.00000e+00     3.54744e+01     1.67220e+00
201        206     1.00000e+00     3.54706e+01     7.36873e-01
202        207     1.00000e+00     3.54638e+01     1.19891e+00
203        208     1.00000e+00     3.54518e+01     3.10195e+00
204        209     1.00000e+00     3.54286e+01     5.18506e+00
205        210     1.00000e+00     3.53910e+01     6.44785e+00
206        211     1.00000e+00     3.53421e+01     5.62661e+00
Cubic Backtracking
207        213     4.14287e-01     3.53284e+01     5.85326e+00
208        214     1.00000e+00     3.52998e+01     2.55959e+00
209        215     1.00000e+00     3.52677e+01     1.39932e+00
210        216     1.00000e+00     3.52445e+01     3.04385e+00
211        217     1.00000e+00     3.52106e+01     3.02110e+00
Cubic Backtracking
212        219     2.47601e-01     3.51915e+01     2.70407e+00
213        220     1.00000e+00     3.51649e+01     1.18019e+00
214        221     1.00000e+00     3.51607e+01     1.07630e+00
215        222     1.00000e+00     3.51571e+01     7.31499e-01
216        223     1.00000e+00     3.51521e+01     7.20430e-01
217        224     1.00000e+00     3.51434e+01     3.09821e+00
218        225     1.00000e+00     3.51166e+01     3.98925e+00
Cubic Backtracking
219        227     1.59383e-01     3.51017e+01     1.17152e+01
220        228     1.00000e+00     3.50315e+01     1.12917e+01
221        229     1.00000e+00     3.47430e+01     1.22976e+01
222        230     1.00000e+00     3.45315e+01     1.65496e+01
223        231     1.00000e+00     3.41820e+01     3.29961e+00
224        232     1.00000e+00     3.39614e+01     1.42738e+01
225        233     1.00000e+00     3.38631e+01     6.83530e+00
226        234     1.00000e+00     3.38364e+01     1.68396e+00
227        235     1.00000e+00     3.38233e+01     2.35392e+00
228        236     1.00000e+00     3.38129e+01     3.74178e+00
229        237     1.00000e+00     3.38073e+01     2.67827e+00
230        238     1.00000e+00     3.38020e+01     1.15128e+00
231        239     1.00000e+00     3.37997e+01     5.81783e-01
232        240     1.00000e+00     3.37990e+01     6.06349e-01
233        241     1.00000e+00     3.37983e+01     4.57007e-01
234        242     1.00000e+00     3.37965e+01     6.39988e-01
235        243     1.00000e+00     3.37942e+01     1.61257e+00
236        244     1.00000e+00     3.37886e+01     3.15287e+00
237        245     1.00000e+00     3.37732e+01     5.59916e+00
238        246     1.00000e+00     3.37420e+01     8.02828e+00
239        247     1.00000e+00     3.36645e+01     1.05009e+01
240        248     1.00000e+00     3.35001e+01     1.32090e+01
Cubic Backtracking
241        250     4.68393e-01     3.34188e+01     7.63911e+00
242        251     1.00000e+00     3.31804e+01     5.62168e+00
Cubic Backtracking
243        253     2.66278e-01     3.31469e+01     1.87864e+01
244        254     1.00000e+00     3.28962e+01     7.48853e+00
Cubic Backtracking
245        256     3.04875e-01     3.28361e+01     4.13573e+00
Cubic Backtracking
246        258     2.71917e-01     3.28145e+01     4.35747e+00
Cubic Backtracking
247        260     3.75557e-01     3.27891e+01     1.48042e+00
248        261     1.00000e+00     3.27709e+01     1.24956e+00
249        262     1.00000e+00     3.27597e+01     1.75535e+00
250        263     1.00000e+00     3.27548e+01     1.10692e+00
251        264     1.00000e+00     3.27495e+01     6.48352e-01
252        265     1.00000e+00     3.27387e+01     1.03373e+00
253        266     1.00000e+00     3.27210e+01     1.65052e+00
254        267     1.00000e+00     3.26888e+01     1.95816e+00
Cubic Backtracking
255        269     1.84281e-01     3.26852e+01     2.59428e+00
256        270     1.00000e+00     3.26725e+01     9.19563e-01
257        271     1.00000e+00     3.26679e+01     5.94338e-01
258        272     1.00000e+00     3.26664e+01     2.11847e-01
259        273     1.00000e+00     3.26659e+01     2.80356e-01
260        274     1.00000e+00     3.26656e+01     2.49121e-01
261        275     1.00000e+00     3.26653e+01     2.37447e-01
262        276     1.00000e+00     3.26646e+01     3.59666e-01
263        277     1.00000e+00     3.26632e+01     8.14548e-01
264        278     1.00000e+00     3.26592e+01     1.41390e+00
265        279     1.00000e+00     3.26494e+01     2.45434e+00
266        280     1.00000e+00     3.26284e+01     3.20563e+00
267        281     1.00000e+00     3.25796e+01     6.04733e+00
268        282     1.00000e+00     3.24797e+01     5.35195e+00
Cubic Backtracking
269        284     4.03979e-02     3.24492e+01     5.17246e+00
Halving Step Size
Cubic Backtracking
270        287     2.35678e-02     3.24183e+01     5.02509e+00
Halving Step Size
Cubic Backtracking
271        290     1.06775e-02     3.24026e+01     5.02368e+00
Halving Step Size
Cubic Backtracking
272        293     8.75930e-03     3.23885e+01     4.85792e+00
Halving Step Size
Cubic Backtracking
273        296     8.64847e-03     3.23754e+01     4.82113e+00
Halving Step Size
Cubic Backtracking
274        299     8.48495e-03     3.23626e+01     4.79578e+00
Halving Step Size
Cubic Backtracking
275        302     6.61829e-03     3.23498e+01     4.46895e+00
Halving Step Size
Cubic Backtracking
276        305     7.68426e-03     3.23386e+01     4.43754e+00
Halving Step Size
Cubic Backtracking
277        308     7.81298e-03     3.23274e+01     4.44386e+00
Halving Step Size
Cubic Backtracking
278        311     7.17260e-03     3.23171e+01     4.33599e+00
Halving Step Size
Cubic Backtracking
279        314     7.68182e-03     3.23065e+01     4.32466e+00
Halving Step Size
Cubic Backtracking
280        317     7.71695e-03     3.22943e+01     3.87005e+00
Halving Step Size
Cubic Backtracking
281        320     8.10289e-03     3.22841e+01     3.84298e+00
Cubic Backtracking
282        322     9.79256e-03     3.22724e+01     3.78237e+00
Cubic Backtracking
283        324     1.17060e-02     3.22575e+01     3.74932e+00
Halving Step Size
Cubic Backtracking
284        327     9.68386e-03     3.22429e+01     3.65198e+00
Cubic Backtracking
285        329     1.89846e-02     3.22231e+01     3.56988e+00
Cubic Backtracking
286        331     1.15832e-01     3.21300e+01     2.96703e+00
Cubic Backtracking
287        333     5.09732e-01     3.19430e+01     2.86055e+00
288        334     1.00000e+00     3.19154e+01     4.38107e+00
Cubic Backtracking
289        336     5.89807e-01     3.18386e+01     3.11372e+00
290        337     1.00000e+00     3.18208e+01     1.55565e+00
Cubic Backtracking
291        339     1.66635e-01     3.18086e+01     3.11696e+00
292        340     1.00000e+00     3.17757e+01     3.18632e+00
293        341     1.00000e+00     3.17619e+01     6.40450e-01
294        342     1.00000e+00     3.17557e+01     1.09718e+00
295        343     1.00000e+00     3.17509e+01     8.20982e-01
Cubic Backtracking
296        345     2.92566e-01     3.17495e+01     9.46843e-01
297        346     1.00000e+00     3.17475e+01     3.32476e-01
298        347     1.00000e+00     3.17472e+01     1.76763e-01
299        348     1.00000e+00     3.17471e+01     7.57184e-02
300        349     1.00000e+00     3.17470e+01     5.94572e-02
301        350     1.00000e+00     3.17470e+01     9.44698e-02
302        351     1.00000e+00     3.17468e+01     1.32735e-01
303        352     1.00000e+00     3.17465e+01     1.28331e-01
Cubic Backtracking
304        354     2.42799e-01     3.17463e+01     5.56811e-01
305        355     1.00000e+00     3.17453e+01     3.96231e-01
306        356     1.00000e+00     3.17420e+01     2.76898e-01
307        357     1.00000e+00     3.17346e+01     5.49474e-01
308        358     1.00000e+00     3.17157e+01     1.79599e+00
309        359     1.00000e+00     3.17091e+01     1.77513e+01
310        360     1.00000e+00     3.16568e+01     3.72509e+00
311        361     1.00000e+00     3.16097e+01     5.82722e+00
312        362     1.00000e+00     3.15805e+01     6.63709e+00
Cubic Backtracking
313        364     2.48522e-01     3.15781e+01     4.05377e+00
314        365     1.00000e+00     3.15717e+01     2.82682e-01
315        366     1.00000e+00     3.15705e+01     3.52940e-01
316        367     1.00000e+00     3.15700e+01     7.22268e-01
317        368     1.00000e+00     3.15699e+01     2.00866e-01
318        369     1.00000e+00     3.15698e+01     1.14195e-01
319        370     1.00000e+00     3.15698e+01     1.33759e-01
320        371     1.00000e+00     3.15698e+01     8.59354e-02
321        372     1.00000e+00     3.15697e+01     4.95021e-01
322        373     1.00000e+00     3.15695e+01     1.19025e-01
323        374     1.00000e+00     3.15692e+01     3.98859e-01
324        375     1.00000e+00     3.15682e+01     1.17773e+00
325        376     1.00000e+00     3.15661e+01     2.05229e+00
326        377     1.00000e+00     3.15609e+01     3.29520e+00
327        378     1.00000e+00     3.15491e+01     4.89311e+00
328        379     1.00000e+00     3.15231e+01     6.30076e+00
329        380     1.00000e+00     3.14842e+01     1.51262e+01
330        381     1.00000e+00     3.14234e+01     1.00023e+01
331        382     1.00000e+00     3.13572e+01     2.51240e+00
Cubic Backtracking
332        384     3.74550e-01     3.13385e+01     1.59041e+00
333        385     1.00000e+00     3.13327e+01     4.16123e+00
334        386     1.00000e+00     3.13232e+01     6.35067e-01
335        387     1.00000e+00     3.13205e+01     6.34061e-01
336        388     1.00000e+00     3.13184e+01     6.72524e-01
337        389     1.00000e+00     3.13169e+01     5.32616e-01
338        390     1.00000e+00     3.13136e+01     1.12510e+00
339        391     1.00000e+00     3.13085e+01     8.00750e-01
340        392     1.00000e+00     3.13002e+01     1.18025e+00
341        393     1.00000e+00     3.12971e+01     8.84212e-01
342        394     1.00000e+00     3.12964e+01     1.28956e-01
343        395     1.00000e+00     3.12963e+01     1.16082e-01
344        396     1.00000e+00     3.12962e+01     1.61351e-01
345        397     1.00000e+00     3.12960e+01     4.37006e-01
346        398     1.00000e+00     3.12957e+01     1.88843e-01
347        399     1.00000e+00     3.12948e+01     3.61698e-01
348        400     1.00000e+00     3.12931e+01     4.54863e-01
Cubic Backtracking
349        402     3.59254e-01     3.12920e+01     1.30738e+00
350        403     1.00000e+00     3.12876e+01     1.04277e+00
351        404     1.00000e+00     3.12721e+01     7.02117e-01
352        405     1.00000e+00     3.12406e+01     3.14311e+00
353        406     1.00000e+00     3.11707e+01     4.21645e+00
354        407     1.00000e+00     3.08783e+01     1.32348e+01
355        408     1.00000e+00     3.08000e+01     1.44335e+01
Cubic Backtracking
356        410     8.44908e-02     3.07515e+01     4.60054e+01
Cubic Backtracking
357        412     3.97596e-01     3.03163e+01     6.34538e+00
358        413     1.00000e+00     3.00511e+01     2.74914e+00
359        414     1.00000e+00     2.98543e+01     5.14810e+00
360        415     1.00000e+00     2.98000e+01     6.01401e+00
Cubic Backtracking
361        417     3.34427e-01     2.97790e+01     3.81520e+00
362        418     1.00000e+00     2.97473e+01     1.49947e+00
363        419     1.00000e+00     2.97178e+01     1.31986e+00
364        420     1.00000e+00     2.96823e+01     2.62300e+00
365        421     1.00000e+00     2.96561e+01     1.63761e+00
366        422     1.00000e+00     2.96470e+01     2.36013e+00
367        423     1.00000e+00     2.96440e+01     2.09950e+00
368        424     1.00000e+00     2.96425e+01     6.98349e-01
369        425     1.00000e+00     2.96423e+01     9.86017e-02
370        426     1.00000e+00     2.96421e+01     2.52698e-01
371        427     1.00000e+00     2.96420e+01     2.11325e-01
372        428     1.00000e+00     2.96420e+01     8.16760e-02
373        429     1.00000e+00     2.96419e+01     6.16555e-02
374        430     1.00000e+00     2.96419e+01     9.64174e-02
375        431     1.00000e+00     2.96419e+01     1.22986e-01
376        432     1.00000e+00     2.96418e+01     1.99052e-01
377        433     1.00000e+00     2.96417e+01     1.36791e-01
378        434     1.00000e+00     2.96416e+01     2.52362e-01
379        435     1.00000e+00     2.96414e+01     9.48728e-02
380        436     1.00000e+00     2.96411e+01     1.45948e-01
381        437     1.00000e+00     2.96406e+01     1.88371e-01
Cubic Backtracking
382        439     9.12161e-02     2.96405e+01     4.81745e-01
383        440     1.00000e+00     2.96400e+01     3.32081e-01
384        441     1.00000e+00     2.96393e+01     1.12679e-01
385        442     1.00000e+00     2.96387e+01     2.37074e-01
386        443     1.00000e+00     2.96380e+01     5.39816e-01
387        444     1.00000e+00     2.96368e+01     9.00243e-01
388        445     1.00000e+00     2.96347e+01     1.31573e+00
389        446     1.00000e+00     2.96311e+01     1.64989e+00
390        447     1.00000e+00     2.96242e+01     1.66699e+00
391        448     1.00000e+00     2.96087e+01     1.84037e+00
392        449     1.00000e+00     2.95736e+01     1.38667e+00
393        450     1.00000e+00     2.95105e+01     2.57109e+00
394        451     1.00000e+00     2.93946e+01     3.23500e+00
395        452     1.00000e+00     2.93211e+01     8.28270e+00
396        453     1.00000e+00     2.91742e+01     3.01079e+00
397        454     1.00000e+00     2.91179e+01     2.28757e+00
398        455     1.00000e+00     2.90911e+01     1.38059e+00
399        456     1.00000e+00     2.90835e+01     1.05466e+00
400        457     1.00000e+00     2.90819e+01     1.87667e+00
401        458     1.00000e+00     2.90774e+01     2.80874e-01
402        459     1.00000e+00     2.90764e+01     1.35869e-01
403        460     1.00000e+00     2.90747e+01     4.88821e-01
404        461     1.00000e+00     2.90737e+01     3.76585e-01
Cubic Backtracking
405        463     4.54085e-01     2.90733e+01     9.34373e-01
406        464     1.00000e+00     2.90729e+01     3.66487e-01
407        465     1.00000e+00     2.90728e+01     8.96300e-02
408        466     1.00000e+00     2.90727e+01     5.18939e-02
409        467     1.00000e+00     2.90727e+01     3.05101e-02
410        468     1.00000e+00     2.90727e+01     4.06064e-02
411        469     1.00000e+00     2.90727e+01     1.28930e-01
412        470     1.00000e+00     2.90726e+01     2.13296e-01
413        471     1.00000e+00     2.90726e+01     9.68276e-02
414        472     1.00000e+00     2.90724e+01     1.78460e-01
415        473     1.00000e+00     2.90721e+01     3.04569e-01
416        474     1.00000e+00     2.90715e+01     3.96365e-01
417        475     1.00000e+00     2.90706e+01     9.94284e-01
418        476     1.00000e+00     2.90687e+01     8.11553e-01
419        477     1.00000e+00     2.90638e+01     1.42913e+00
420        478     1.00000e+00     2.90433e+01     4.14509e+00
421        479     1.00000e+00     2.90082e+01     6.91957e+00
422        480     1.00000e+00     2.89322e+01     1.01340e+01
423        481     1.00000e+00     2.87979e+01     1.09846e+01
Cubic Backtracking
424        483     4.57692e-01     2.87151e+01     2.06038e+01
Cubic Backtracking
425        485     5.28357e-01     2.86564e+01     1.46396e+01
Cubic Backtracking
426        487     4.67009e-01     2.85846e+01     1.94174e+01
427        488     1.00000e+00     2.84744e+01     1.21767e+01
428        489     1.00000e+00     2.84041e+01     6.56336e+00
429        490     1.00000e+00     2.83504e+01     7.88567e+00
Cubic Backtracking
430        492     4.97337e-01     2.83242e+01     6.39655e+00
431        493     1.00000e+00     2.83147e+01     1.88986e+00
432        494     1.00000e+00     2.83063e+01     2.67096e+00
433        495     1.00000e+00     2.82998e+01     6.48956e-01
434        496     1.00000e+00     2.82968e+01     8.11180e-01
435        497     1.00000e+00     2.82928e+01     4.19737e-01
436        498     1.00000e+00     2.82911e+01     8.94852e-01
437        499     1.00000e+00     2.82895e+01     7.35128e-01
438        500     1.00000e+00     2.82884e+01     7.86002e-01
439        501     1.00000e+00     2.82882e+01     4.15919e-01
Function Evaluations exceeds maxIter
Test error of kernel multinomial logistic regression: 0.116000 (distance = 0.140000)
Test error of kernel ordinal logistic regression: 0.152000 (distance = 0.176000)
```

## Graphical LASSO

Solve min_{W positive-definite} logdet(sigma+W), s.t. W_ij <= lambda

```% Load data
docs = full(documents)';
[nSamples,nVars] = size(docs);
mu = mean(docs);
centered = docs-repmat(mu,16242,1);
sigma = (1/nSamples)*(centered'*centered);

% Set up and solve problem
lambda = .01;
funObj = @(W)logdetFunction(W,sigma);
LB = -lambda*ones(nVars);
UB = lambda*ones(nVars);
W = eye(nVars);
W(:) = minConf_TMP(funObj,W(:),LB(:),UB(:));
K = inv(sigma+W);
K(abs(W) < lambda) = 0;
clf;
drawGraph(K~=0,'labels',wordlist);
pause
```
``` Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     3.22718e-04     3.24233e+02     2.51681e+03
2          3     1.00000e+00     3.22501e+02     2.89437e+03
3          4     1.00000e+00     3.21744e+02     2.00445e+03
4          5     1.00000e+00     3.21430e+02     1.67003e+03
5          6     1.00000e+00     3.20927e+02     1.36135e+03
6          7     1.00000e+00     3.20824e+02     1.41744e+03
7          8     1.00000e+00     3.20591e+02     1.07992e+03
8          9     1.00000e+00     3.20494e+02     8.84046e+02
9         10     1.00000e+00     3.20371e+02     7.68824e+02
10         11     1.00000e+00     3.20265e+02     6.64347e+02
Cubic Backtracking
11         13     4.53484e-01     3.20210e+02     6.00605e+02
12         14     1.00000e+00     3.20159e+02     4.70352e+02
13         15     1.00000e+00     3.20136e+02     3.89314e+02
14         16     1.00000e+00     3.20107e+02     3.24526e+02
15         17     1.00000e+00     3.20102e+02     3.33957e+02
16         18     1.00000e+00     3.20084e+02     2.22999e+02
17         19     1.00000e+00     3.20079e+02     1.74439e+02
18         20     1.00000e+00     3.20073e+02     1.54548e+02
19         21     1.00000e+00     3.20068e+02     1.34781e+02
20         22     1.00000e+00     3.20064e+02     1.23284e+02
21         23     1.00000e+00     3.20063e+02     8.72360e+01
22         24     1.00000e+00     3.20061e+02     6.62358e+01
23         25     1.00000e+00     3.20060e+02     6.05389e+01
24         26     1.00000e+00     3.20060e+02     5.08497e+01
25         27     1.00000e+00     3.20059e+02     3.77442e+01
26         28     1.00000e+00     3.20059e+02     3.05205e+01
27         29     1.00000e+00     3.20059e+02     2.48642e+01
28         30     1.00000e+00     3.20059e+02     2.78130e+01
29         31     1.00000e+00     3.20059e+02     1.82057e+01
30         32     1.00000e+00     3.20059e+02     1.41699e+01
31         33     1.00000e+00     3.20059e+02     1.19657e+01
32         34     1.00000e+00     3.20059e+02     1.13690e+01
33         35     1.00000e+00     3.20059e+02     9.48376e+00
34         36     1.00000e+00     3.20059e+02     7.25901e+00
35         37     1.00000e+00     3.20059e+02     6.32951e+00
36         38     1.00000e+00     3.20059e+02     4.71239e+00
Cubic Backtracking
37         40     3.64475e-01     3.20059e+02     4.35473e+00
38         41     1.00000e+00     3.20059e+02     3.39223e+00
39         42     1.00000e+00     3.20059e+02     2.78117e+00
40         43     1.00000e+00     3.20059e+02     2.15063e+00
Cubic Backtracking
41         45     4.45570e-01     3.20059e+02     1.92475e+00
Function value changing by less than optTol
neato - Graphviz version 2.20.2 (Tue Mar  2 19:03:41 UTC 2010)
```

## Associative Conditional Random Fields (trained with pseudo-likelihood)

Optimize pseudo-likelihood in CRF with Ising potentials, subject to constraint that edges are sub-modular (and hence the optimal MAP can be found using graph cuts)

```% Load Data
y = int32(1+X);
X = X + randn(size(X))/2;
[nRows,nCols] = size(X);
nNodes = nRows*nCols;
nStates = 2;
y = reshape(y,[1 1 nNodes]);
X = reshape(X,1,1,nNodes);

% Set up problem in UGM
tied = 1;
Xnode = [ones(1,1,nNodes) UGM_standardizeCols(X,tied)];
sharedFeatures = [1 0];
Xedge = UGM_makeEdgeFeaturesInvAbsDif(Xnode,edgeStruct.edgeEnds,sharedFeatures);
ising = 1;
[nodeMap,edgeMap,w] = UGM_makeCRFmaps(Xnode,Xedge,edgeStruct,ising,tied);
nParams = length(w);
funObj = @(w)UGM_CRF_PseudoNLL(w,Xnode,Xedge,y,nodeMap,edgeMap,edgeStruct);
UB = inf(nParams,1);
LB = [-inf;-inf;0;0];
w = minConf_TMP(funObj,w,LB,UB)

[nodePot,edgePot] = UGM_CRF_makePotentials(w,Xnode,Xedge,nodeMap,edgeMap,edgeStruct,1);
MAP = UGM_Decode_GraphCut(nodePot,edgePot,edgeStruct);
imagesc(reshape(X,nRows,nCols));colormap gray
figure
imagesc(reshape(MAP,nRows,nCols));colormap gray
```
``` Iteration   FunEvals     Step Length    Function Val        Opt Cond
L-BFGS
1          2     3.09646e-04     1.47402e+02     4.44701e+02
2          3     1.00000e+00     1.27291e+02     3.25422e+02
3          4     1.00000e+00     9.54447e+01     1.55947e+02
4          5     1.00000e+00     7.87347e+01     8.84294e+01
5          6     1.00000e+00     6.43963e+01     4.44543e+01
6          7     1.00000e+00     5.64459e+01     2.15631e+01
7          8     1.00000e+00     5.30260e+01     8.65324e+00
8          9     1.00000e+00     5.21021e+01     3.18210e+00
9         10     1.00000e+00     5.19146e+01     1.13304e+00
10         11     1.00000e+00     5.18572e+01     9.32031e-01
11         12     1.00000e+00     5.18103e+01     1.02205e+00
12         13     1.00000e+00     5.17777e+01     8.63162e-01
13         14     1.00000e+00     5.17662e+01     2.54473e-01
14         15     1.00000e+00     5.17651e+01     1.68942e-02
15         16     1.00000e+00     5.17651e+01     3.46060e-03
Directional Derivative below optTol

w =

-0.7214
-2.1634
1.5427
0.9625

```

Mark Schmidt > Software > minConf > Examples of using minConf_TMP