Math 479/579 Homework #2

Due February 17th

1. Let ~x = (x1 , x2 , . . . , xn )T and f (~x) = f (x1 , x2 , . . . , xn ) : Rn ! R denote any dierentiable
function of n variables.
(a) Given any ~y = (y1 , y2 , . . . , yn )T 2 Rn , show that

d
f (~x + ~y ) |=0 = h(rf )(~x), ~y i
d
where (rf )(~x) = (@f (~x)/@x1 , @f (~x)/@x2 , . . . , @f (~x)/@xn )T denotes the gradient of f evaluated at the point ~x 2 Rn .

(b) Suppose that we can find a vector ~v 2 Rn satisfying the following property: for all ~y 2 Rn ,
the equality

d
f (~x + ~y ) |=0 = h~v , ~y i
d
holds. Show that ~v = (rf )(~x) is necessarily the gradient of f .

2. Let X denote an m n matrix, A denote a d m matrix and B denote a d n matrix.

Consider the function
1
f (X) = kAX
2

1
Bk2F = hAX
2

B, AX

1
Bi := trace((AX
2

B)T (AX

B)),

viewed as a function of the mn elements of X.

(a) Given any m n matrix Y, show that

d
f (X + Y ) |=0 = trace((AY )T (AX
d

B))

(You may use linearity of the trace, i.e. that trace(aX + Y ) = a trace(X) + trace(Y ) for
any scalar a and matrices X, Y without proof.)
(b) Use question 1 above and problem 2 from the first homework to conclude that
(rf )(X) = AT (AX

B),

where here we view the gradient as an m n matrix (by reshaping, etc.) instead of as a
vector in Rmn .
3. In the least-squares approach to binary classification, we encoded the class assignment for
our training points ~xi as a vector ~y 2 Rn with yi 2 {0, 1}. Here we show that we could use
any distinct real numbers a and b with a < b (not just zero and one) in this process without
changing the classification.
(a) Let ~z = a + (b
yi = 1.

a)~y . Show that zi = a if and only if yi = 0 and that zi = b if and only if

(b) Let

y =

1X
yi
n i=1

z =

1X
zi
n i=1

yi = yi

zi = z

:= z

h~ , xi.

and let ~ 2 Rn denote the unique solution of the system

TX
+ I) ~ = X
T y.
(X
Find the unique solution ~ 2 Rn of the system
TX
+ I)~ = X
T z
(X
in terms of (b

a) and ~ .

(c) Let

x :=

1X
~xi
n i=1

:= y

h ~ , xi

We classify a point ~x as class b if

+ h~ , ~xi

a+b
.
2

Show that this occurs if and only if

+ h ~ , ~xi

1
,
2

or in other words if and only if we classify a point ~x as class 1 using the original {0, 1}
labels.
4. (Coding I)
(a) Turn the skeleton code LSClassify.m (available on BeachBoard) into a workable MATLAB
function that takes the parameters (Xtrain,y,lambda) as input and returns [alpha,beta]
as output.
(b) Use the loadCOLOR.m file along with the LSClassify.m function you wrote to perform
the training phase for binary classification of the image pixels. Use the values
=
.005, .5, 10, 250, 800 in your computation.
(c) For each value of = .005, .5, 10, 250, 800, compute the estimated class (foreground or
background) of all pixels in the original image. Then use viewCLASSES.m to view the
corresponding foreground/background estimation of the image.
(d) Summarize your results.
5. (Coding II) In this problem, you will use the LSClassify.m you wrote for problem Coding I
above to perform binary classification of handwritten digits. If you wrote it correctly, you
shouldnt need to modify your LSClassify.m file to complete this problem.
(a) Use the loadMNIST.m file to load the ones and sixes from training portion of the data
set

1
2

[img,lab] = loadMNIST([1,6],'train');
y = double(lab == 6);

The last line converts the output lab into a vector y of ones and zeros, where yi = 1 if
the ith image is a six and is zero otherwise.
(b) Use your LSClassify.m file
1

[alpha,beta] = LSClassify(img,y,20.0);

to compute the corresponding and ~ for

= 20.

(c) Use the loadMNIST.m file to load the ones and sixes from the testing portion of the data
set
1
2

[img t,lab t] = loadMNIST([1,6],'test');

binary = double(lab t == 6);

For each row (or, image) in the testing portion, compute its predicted class label
1

binary est = double( alpha + img t *beta > .5 );

(d) Use the ACCURACY.m file (on BeachBoard) to compute the percentage of data points
that were correctly classified
1

pct = ACCURACY( binary est + 1, binary + 1 , 2);

and report the result.

(e) Repeat parts (a-d) using the images of fours and nines instead of the images of ones and
sixes.