Contents
Description of demo_regression_ARD.m
Regression using Automatic Relevance Determination to encourage sparsity in learned weight vector with high-dimensional data where most features are uninformative. L2 regression with and without regularization and L1 regression are presented for comparisons
clear all close all generateData_irrelevFeatures
usage of ARD regression
options_ard = []; options_ard.addBias = 1; options_ard.variance = 0.5; model_ard = ml_regression_ARD(Xtrain, ytrain, options_ard); yhat_ard = model_ard.predict(model_ard, Xtest); testError_ard = mean(abs(yhat_ard - ytest)); fprintf('Averaged absolute test error with %s and Var %.3f is: %.3f\n', ... model_ard.name, options_ard.variance, testError_ard); fprintf('Number of features used is is: %.3f\n', ... sum(abs(model_ard.w(2:end)) > 0.05));
Averaged absolute test error with ARD and Var 0.500 is: 0.089 Number of features used is is: 2.000
usage of L2 regression
options_l2_1 = []; options_l2_1.addBias = 1; model_l2_1 = ml_regression_L2(Xtrain, ytrain, options_l2_1); yhat_l2_1 = model_l2_1.predict(model_l2_1, Xtest); testError_l2_1 = mean(abs(yhat_l2_1 - ytest)); fprintf('Averaged absolute test error with %s is: %.3f\n', ... model_l2_1.name, testError_l2_1); fprintf('Number of features used is is: %.3f\n', ... sum(abs(model_l2_1.w(2:end)) > 0.05));
Averaged absolute test error with Squared Loss Linear Regression is: 0.089 Number of features used is is: 10.000
usage of L2 regression with L2 regularization
options_l2_2 = []; options_l2_2.addBias = 1; options_l2_2.lambdaL2 = 5; model_l2_2 = ml_regression_L2(Xtrain, ytrain, options_l2_2); yhat_l2_2 = model_l2_2.predict(model_l2_2, Xtest); testError_l2_2 = mean(abs(yhat_l2_2 - ytest)); fprintf('Averaged absolute test error with %s and L2 Reg is: %.3f\n', ... model_l2_2.name, testError_l2_2); fprintf('Number of features used is is: %.3f\n', ... sum(abs(model_l2_2.w(2:end)) > 0.05));
Averaged absolute test error with Squared Loss Linear Regression and L2 Reg is: 0.127 Number of features used is is: 2.000
usage of L1 regression
options_l1 = []; options_l1.addBias = 1; model_l1 = ml_regression_L1(Xtrain, ytrain, options_l1); yhat_l1 = model_l1.predict(model_l1, Xtest); testError_l1 = mean(abs(yhat_l1 - ytest)); fprintf('Averaged absolute test error with %s is: %.3f\n', ... model_l1.name, testError_l1); fprintf('Number of features used is is: %.3f\n', ... sum(abs(model_l1.w(2:end)) > 0.05));
Averaged absolute test error with Absolute Loss Linear Regression is: 0.134 Number of features used is is: 2.000