CS 678 - Advanced Machine Learning and Neural Networks

Assignments – Winter 2017


Note:  All written assignments (except homework) are to be done with a word processor and be neat and professional. You will hand in a hard copy at the beginning of class on the due date.  Good writing, grammar, punctuation, organization, etc. are important and will affect your grade.


Your Model Readings
Due: January 24


This assignment will allow you to give me a ranked list of three possible models for which you will be doing an in-depth implementation and presentation (see below).  It will also give you an opportunity to explore the field of machine learning a little more.  Do a write-up of 1 single spaced page each on three papers of your choosing in the area of machine learning which you will hand in at the beginning of class on the due date. At least two of the papers should be ones you have not read before the semester (and areas we will not directly cover this semester) and the papers should each cover a different area in machine learning.  These papers should focus on machine learning models that are new to you and interesting to you.  The papers should focus on innovative technical aspects of machine learning and not just applications of machine learning.  As much as possible, choose papers that are relatively recent.  For each model, you should a) briefly describe the main approach and contribution of the paper and b) give your opinion on the specific strengths, weaknesses, and contributions of the approach.  Grading will be based on the perceived effort and insights you have gained.  Note that we allow one of the papers to not be new for you.  That allows you to potentially implement a model that you have been considering for your research, etc.  But, if so, it MUST be a model that you have not already started developing or using in your research.


In your hardcopy give me a priority order on the three papers regarding which you would most like to present.  Include the full reference for each paper.  Also e-mail me PDF files for each paper with the preferred ranking stated.  I will let you know which of the three models you will implement and present.  I will try to follow your ranking, but may choose another if more appropriate.  By appropriate I will consider:


      Is this a doable and reasonable model for a one semester project.

      I prefer that no model is presented by more than one person, since the presentations are an important part of our learning experience as a class.  It will be first-come first-serve if multiple people want to do the same model, so you can hand this in as early as you like.

      In some cases, if you request, you may do your model with a team of no greater than 2 members.  In that case, we will expect a product that exceeds that being done by one person.


Some good sources for potential papers include Proceedings of the International Conference on Machine Learning, Proceedings of Neural Information Processing Systems, Machine Learning Journal, Journal of Artificial Intelligence Research, Journal of Machine Learning Research, Neural Networks, IEEE Transactions on Neural Networks, Neural Computation, among others.


Model of your choice Project and Presentation

Hard copy due at the beginning of class on the day you give your oral presentation of the model near the end of the semester.  This project should be started early in the semester and go on concurrently with the deep learning project below.  Your write-up should be formatted like a conference paper (5 page limit).  You will usually have a small bibliography with more than the initial paper from the readings assignment above.


1)    Implement an ML model of your choice. This will be a model not previously implemented for another class.  You may choose one of the models discussed in the class, or one that you have an interest in.  The model will come from the ranked list of 3 possible models you give me in the initial assignment above.  You can give it to me sooner and those who get it in soonest will have highest priority if there are multiple people wanting to do the same model.  I will then give you the OK to proceed forward.  This will allow me to give you feedback on your choice, and allow more diversity on the models we consider, so we can all learn from the presentations.

2)    Test the model on at least 2 different data sets and for a reasonable spread of the learning algorithm parameters.  Analyze and discuss your results.

3)    Do a creative experiment with the model and discuss your findings.

4)    In addition to your written conference style report, you will prepare a conference style talk (slide presentation) to be given to the class (~10-15 minutes) at the end of the semester.  The talk should motivate your model, explain how it works, discuss results on your tasks, and give your overall analysis including recommendations for potential improvements of the model.


You will hand in an initial project outline proposal shortly after you have the OK, and then a progress report half way through the semester.


Project Outline Proposal:  (1 page) Due Jan 31

1.     What other papers you may include in the bibliography

2.     Brief description of the implementation you will do

3.     What data sets will you test it on

4.     What creative experiment(s) are you considering

5.     Proposed timeline for completion


Project Progress Report:  (1 page) Due Feb 23


Deep Learning Project 

Hard copy due at the beginning of class on the day we all present (~March 30).


1)    You will solve the MNIST task using deep learning.  You may make a special request to solve a different problem if you have a good reason. We have the MNIST data in ARFF format here.  There are lots of papers regarding MNIST that you can review if you would like.  For consistency and comparison with each other, we will just use the given MNIST input features without initial pre-processing, though you are welcome to also try some pre-processing of the features to see what kind of results that gives.  We will just use their given training set and test set for training, testing, and comparison.

2)    Train on a standard single layer MLP with 200 hidden nodes and report test results as a baseline to compare with.

3)    You may choose from any of the approaches we discuss in class or any other deep learning approaches you would like.  This includes convolution net approaches, deep belief nets, stacked auto-encoders, LSTMs, latest supervised approaches, etc.  You CANNOT just use standard BP with multiple hidden layers. Use the produced features as input to an MLP with 200 hidden nodes as at least one comparison with the baseline.  Try some variations on your deep network to try to get the best accuracy possible.  Give and discuss your results.  If using an unsupervised approach, try to refine the weights of the entire network with Backpropagation after your initial learning and discuss improvements.  If using stacked autoencoders make sure you include mechanisms to encourage sparsity (e.g. denoising, weight decay, etc). 

4)    Analyze and discuss your results in a written paper (4 page limit).

5)    We will set aside one day to do fast (3-5 minutes each) presentations of your approaches and results so that we can all learn from each other.