CS
678 - Advanced Machine Learning and Neural Networks

__Assignments
– Winter 2015__

Note: All written assignments (except
homework) are to be done with a word processor and be neat and professional.
You will hand in a hard copy at the beginning of class on the due date. Good writing, grammar, punctuation,
organization, etc. are important and will affect your grade.

**Paper Readings and Presentations**

Due: January 20

Do a
write-up of approximately ½ single spaced page each on three papers of
your choosing in the area of machine learning which you will hand in at the
beginning of class on the due date. These papers should be ones you have not
read before the semester (and areas we will not directly cover this semester)
and the papers should cover three different areas in the field. This is a chance for us to explore. The majority of the papers should focus
on innovative technical aspects of machine learning and not just applications
of machine learning. As much as
possible, choose papers which are relatively
recent. In your half page, you
should a) briefly describe the main approach and contribution of the paper and
b) give your opinion on the specific strengths, weaknesses, and contributions
of the approach. Grading will be
based on the perceived effort and insights you have gained.

In your
hardcopy give me a priority order on the three papers regarding which you would
most like to present. Include the
full reference for each paper. Also
e-mail me PDF files for each paper with the preferred ranking stated. I will let you know which paper you will
present and will most likely have you present the highest priority paper, but
may choose another if more appropriate.
You will give a 15 minute presentation on the paper to the class
including an explanation of the approach or concept proposed in the paper and
also the most interesting aspects and insights of the paper. The goal is to
educate the audience on the most interesting aspects of the research in the
limited time you have. Be prepared
to answer questions. We will do one
presentation at the beginning of each class starting early February. The quality of the presentation will
also be part of the grading. You
are encouraged to use slides or any other appropriate teaching aids. I
especially like when in addition to reviewing the main points of the paper, you
also go beyond and add some or your own insights into interesting aspects of
the approach.

Goals
of this assignment include a) an opportunity to better learn how to read and
analyze research papers, b) an opportunity to practice your oral presentation
skills, and c) an opportunity through the presentations for all of us to gain
new and diverse insights into machine learning areas.

Some
good sources for potential papers include *Proceedings
of the International Conference on Machine Learning*, *Proceedings of Neural Information Processing Systems*, *Machine Learning Journal*, *Journal of Artificial Intelligence Research,*
*Journal of Machine Learning Research*,
*Neural Networks*, *IEEE Transactions on Neural Networks*, *Neural Computation, *among others.

**Projects**: The three following projects
are due as a hardcopy at the beginning of class on the due date. Do not attach
your source code. The write-up
would typically be about 4-5 pages.

**Recurrent Neural Networks
Project - BPTT**

Due at
the beginning of class on Feb. 19

1)
Implement
BPTT on the platform of your choice.

2)
Test your BPTT on the *Delayed
Parity Synthetic Task.* You can
create you own *DParity*
generator or here is
an app you can use. This task has a
single time series input of random bits.
The output label is the parity (even) of *n* arbitrarily delayed (but consistent) previous inputs. For example, for *DParity*(0,2,3) the label of each instance would be set to the
parity of the current input, the input 2 steps back, and the input 3 steps
back. Make sure your BPTT first
works for *DParity*
(0,1) which is just the exclusive-or of the current
and previous input. Then
experiment with some different *DParity*(*x*,*y*,*z*) tasks, *k* values, number of hidden nodes, and
other learning parameters, and analyze and discuss how it does. Begin with the parameters suggested in
the slides, and alter those a bit.
Use tables/graphs or whatever best explains your findings. Your *k
*should usually be *z*+1. What
happens if *k* < *z*+1, equal *z*+1, or greater than *z*+1? Of course in an actual application you
would not know *z* apriori.

3)
Test BPTT on another time-series/recurrent task of your
choice. Explain your exact network
structure (i.e. a) # of input, output, and hidden nodes and b) exactly how you
represented the inputs and outputs).
Analyze and discuss your results with appropriate graphs and
tables. Note that a number of time
series problems can be found in the UC Irvine Database. Try to choose one that really needs
recurrence to do well (not all do). This part can be challenging, but do your
best to get reasonable results. You
may also work with a fellow classmate(s) on this part if you wish.

**Model of your choice Project and
Presentation**

Due at
the beginning of class on the day you give your oral presentation of the
model. DonŐt wait until you are
done with deep learning to start this.
This should be started earlier in the semester.

1)
Implement
an ML model of your choice. This
will be a model not previously implemented for another class. You may choose one of the models
discussed in the class, or one that you have an interest in. I will have you propose a ranked list of
3 possible models and e-mail it to me no later than **Feb. 3** (this list can overlap with the models you choose for the
paper presentation above). You can
give it to me sooner and those who get it in soonest will have highest priority
if there are multiple people wanting to do the same model. I will then give you the OK to proceed
forward. This will allow me to give
you feedback on your choice, and allow some more diversity on the models we
consider.

2)
Test the model on at least 2 different data sets and for a
reasonable spread of the learning algorithm parameters. Analyze and discuss your results.

3)
Do a creative experiment with the model and discuss your
findings.

4)
In addition to your written report, you will prepare a slide
presentation to be given to the class (10-12 minutes) at the end of the
semester. The talk should motivate
your model, review how it works, discuss results on your tasks, and give your
overall analysis and recommendation for potential improvements of the model.

**Deep Learning Project**

Due at
the beginning of class on April 7

1)
You
will solve the MNIST task using
deep learning. We have the data in
ARFF format here. There are lots of papers regarding MNIST which you can review if you would like. For consistency and comparison with each
other, we will just use the given MNIST input features without initial
pre-processing, though you are welcome to also try some pre-processing of the
features to see what kind of results that gives. We will just use their
given training set and test set for training, testing, and comparison.

2)
Train on a standard single layer MLP with 200 hidden nodes
and report test results as a baseline to compare with.

3)
Implement an initial unsupervised deep network (Deep belief
net or stacked sparse autoencoders) for preprocessing. If using stacked autoencoders make
sure you include mechanisms to encourage sparsity (e.g. denoising,
weight decay, etc). Use the
produced features as input to an MLP with 200 hidden nodes. Give and discuss your results. Also, try to refine the weights of the
entire network with Backpropagation after your initial learning and discuss
improvements. Try some variations on your deep network to try to get the best
accuracy possible.

4)
Analyze and discuss your results.