CS 559: Homework Set 2

Due: October 9, 6:15pm

Philippos Mordohai
Department of Computer Science
Stevens Institute of Technology
Philippos.Mordohai@stevens.edu

Collaboration Policy. Homeworks will be done individually: each student must hand in their
own answers. Use of complete or entire solutions obtained from others or online is strictly prohibited.
Late Policy. No late submissions will be allowed without consent from the instructor. If urgent
or unusual circumstances prohibit you from submitting a homework assignment in time, please
e-mail me explaining the situation.
Submission Format. Electronic submission by e-mail is the only acceptable format. Please
name your file LastnameFirstname-hw2.pdf.
Problem 1. DHS problem 3.2(b). (20 points) Let x have an exponential density
 x
e
x0
p(x|) =
0
otherwise.
Suppose that n samples x1 , ..., xn are drawn independently according to p(x|). Show that the
maximum-likelihood estimate for is given by
=

1
n

1
Pn

k=1

xk

Problem 2. DHS problem 3.7. (30 points) Show that if our model is poor, the maximumlikelihood classifier we derive is not the best even among our (poor) model set by exploring the
following example. Suppose we have two equally probable categories (i.e. P (1 ) = P (2 ) = 0.5).
Furthermore, we know that p(x|1 ) N (0, 1) but assume that p(x|2 ) N (, 1). That is, the
parameter we seek by maximum-likelihood techniques is the mean of the second distribution.)
Imagine, however, that the true underlying distribution is p(x|2 ) N (1, 104 )
(a) What is the value of the maximum-likelihood estimate
in our poor model given a large
amount of data?
(b) What is the decision boundary arising from this maximum-likelihood estimate in this poor
model?
1

(c) Ignore for the moment the maximum-likelihood approach and use the methods from Chapter
2 to derive the Bayes optimal decision boundary given the true underlying distributions
p(x|1 ) N (0, 1) and p(x|2 ) N (1, 104 ). Be careful to include all portions of the
decision boundary.
(d) Now consider again classifiers based on the (poor) model assumption of p(x|2 ) N (, 1).
Using your result immediately above, find a new value of that will give lower error than
the maximum-likelihood classifier.
(e) Discuss these results with particular attention to the role of knowledge of the underlying
model.
Hints:
a2 = b2 |a| = |b|.
Use the straightforward decision rule based on posterior probabilities for part (c).
The decision regions may be disconnected.
Use actual sentences for part (e).
Use 104 for the variance of the true distribution (not 106 as in the textbook).
Problem 3. (15 points) 8 observations from a 1D signal X = {4, 3, 2, 1, 1, 2, 3, 4} are
available for non-parametric density estimation. Use square Parzen windows to estimate the probability density at 0, 1 and 2.

1 |u| < 1/2
(u) =
0 otherwise
Try the following values for the size of the window hn = 1, 3, and 5. What are your conclusions
based on these different estimates both about the underlying density and about the estimation
method?
Problem 4. (35 points) Download the Pima Indians Diabetes Database from
http://archive.ics.uci.edu/ml/datasets/Pima+Indians+Diabetes.
(a) Implement a classifier using Maximum-Likelihood Estimation that takes into account features 4 to 6, among the 8 available features.
(b) Implement a Naive Bayes classifier using the same features.
(c) Train both classifiers on the same samples and run them 10 or more times. Record the mean
and standard deviation of the accuracy for each classifier. Initially use 50% of the data for
training and then repeat the experiments using 75% of the data for training.
2

(d) Submit code, but not data, and discuss the performance of the classifiers using the data you
collected for part (c). Ideally, we will see the results in your report without having to run
your code.
Hints:
The cov() command in Matlab can be used to compute the necessary covariance matrices.
You can set active f eat = 4 : 6 to select the necessary features if you follow the example
in the relevant set of notes.
If you are not using Matlab, you will need to be able to compute the inverse and the determinant of 3 3 matrices. You will also need to randomly split into training and test sets
multiple times. Contact me to discuss potential solutions.