Backprop Tutorial




Installing JavaNNS

 Go to http://www-ra.informatik.uni-tuebingen.de/SNNS/
Click on the link to "JavaNNS" which is at the bottom of the page.
Download the appropriate version for your computer. (I recommend the windows version since this is the only one I have tested.)
Unzip/untar the file to a directory

Install Java if it isn't already installed.
I installed "Java 2 SDK, Standard Edition, v 1.4 Beta 2" from http://java.sun.com/j2se/
Java is already installed on the windows open labs.

Note that JavaNNS comes with a manual and some examples. However, the manual only describes how the basics work and not the underworkings of the mathematical kernel. More details can be found in the original SNNS manual .

Running JavaNNS

 From a command prompt, go to the directory you uncompressed JavaNNS to.
Type:
java -jar JavaNNS.jar
OR:
JavaNNS.bat
(if in windows)

READ THE MANUAL!!

 I highly recommend skimming sections 4 - 9 (only 11 pages) in the manual. It will get you more familiar with all the parts of the simulator and answer a lot of your initial questions.

Example: Training a Network

 1. Go to "File | Open"
2. Select letters.pat and letters.net from the JavaNNS\examples folder (this can be done by ctrl-left clicking on the files). This loads the network and training set.
3. Press OK when a dialog box appears saying "Load corresponding configuration file?".
4. A view of the network should appear.
5. Open the Log window ("View | Log") and the Error Graph ("View | Error Graph").
6. Open the Control Panel ("Tools | Control Panel") and click on the Learning tab.



7. Press the Init button to intialize the weights to random weights.
8. Change learning parameters if desired. (n is the learning rate, dmax is the max nonpropagated error, and Cycles is the number of epochs)
9. Check the shuffle box if you'd like the patterns to be shuffled between epochs.
10. Press the Learn All button and training will begin.
11. An error curve should start to appear on the Error Graph and these errors should also be printed to the log.

Try changing the learning function to Backprop-Momentum (where momentum is also a parameter) or Backprop-Weight Decay (where the amount of decay is a parameter).

Example: Building a Network

 To prepare to create a network:

1. Go to "File | New" to ensure there is no network loaded.
2. Open the Network window ("View | Network"), so you can see the network as it is being built.
3. Open the Create Layers dialog box ("Tools | Create | Layers").



To create a 2 x 10 x 1 network that could be used for xor.pat:

1. Type 1 in the width edit box.
2. Type 2 in the height edit box.
3. Type (1, 1, 1) in the top left position edit boxes.
4. Change the unit type to Input.
5. Make sure the activation function is Act_Logistic and the output function is Out_Identity.
6. Press the create button. This creates the input layer of the network.
7. The hidden layer is created the same way, but use the following values:
width = 1, height = 10, top left position = (3, 1, 1), and unit type = Hidden
Press create.
8. The output layer uses the following values:
width = 1, height = 1, top left position = (5, 1, 1), and unit type = Output
Press create.
9. Press the Close button. The layers of the network have been created. Now they need to be connected.
10. Open the Create Connections dialog box ("Tools | Create | Connections").
11. Choose Connect Feed-forward and then then press the Connect button. Links should appear on the network.
12. Press the Close button.
13. SAVE YOUR NETWORK (see below).
14. Now you're ready to train the network as described above.

Saving the network:

1. Give the network a name using "File | Save" or "File | Save As".
2. Make sure to put something in the edit box labeled "network name"
otherwise javanns will not be able to load it unless you do the following:
a. Open the file in a text editor.
b. On the network name line (near the top) add a name like:
"network name : bob"
c. Don't change the returns, spaces, or other formatting... or you'll be sorry!
d. Save the file as a text file.

Editing the network:

If you'd like to change a link's weight:

1. Select the two nodes that are connected by the link.
2. Right click and pick Edit Links from the popup menu.
3. The above dialog box should appear.
4. The edit box contains the weight value and can be changed. It will take effect once enter is pressed.
The x deletes the link. The source and target can be changed using the arrows, so multiple links can be edited before closing the dialog.

One or more units can be edited by selecting them, right clicking, and choosing Edit Units. To select more than one unit either click on each unit individually or use a rectangular drag select. To unselect a group of selected units, click somewhere on the background.

ExpandOutputNodes: A Tool for Dealing with Multiple Output Classes

 Takes an SNNS file that contain a dataset with one output and multiple output classes and creates an output for each class.

Example: expandoutputnodes Iris.pat Iris_expanded.pat

Download it here.

Accuracy: A Tool that prints the accuracy of a neural network on a given dataset

 Download it here.

USAGE: accuracy patFilename netFilename outputFilename [-v]

patFilename - name of the .pat file with the dataset
netFilename - name of the .net file with the neural network
outputFilename - name of a file to append the results to (Accuracy\tTSSE\tMSSE)
v - verbose mode (prints the neural net and the dataset)

What the tool does
It builds a neural network with the weights described in the given .net file. It then loads the dataset and runs these examples through the network. A pattern is considered classified correctly if the output node corresponding to that class has the highest activation. The accuracy on the dataset is calculated using this definition and printed out.

There are a few things you have to do to ensure that it works:
1. The neural network must have the layers created in this order: input layer, hidden layer 1, hidden layer 2, etc., output layer. To keep my program from being insanely hard to write, I assumed that the nodes are allocated in order by layer. If you delete layers or add a layer in the middle, the program will not work!
2. The network cannot have links that connect beyond the next layer in the network.
3. Pat file must conform to the standards of SNNS V1.0. That means if you have a file of a later version you may have to remove the #comments.
4. The pat file must have binary outputs, one output for each class. The expandoutputnodes.exe tool above can be used to do this.

IMPORTANT NOTE: The values for MSE and TSSE that I print out are not normalized and assume dmax = 0, so the values differ sometimes greatly from the values in SNNS. You are welcome to use either values, but tell me which values you are using.

Example use:
1. Expand the output nodes of iris.pat using expandoutputnodes.exe (described above)
2. Split iris_expanded.pat into a training, validation, and test set using split.exe (see the Delta Rule hints page)
3. Open up three command prompts
4. At one prompt type: accuracy iris_train.pat iris.net iris_train.txt (but don't press return)
5. At another prompt type: accuracy iris_valid.pat iris.net iris_valid.txt (but don't press return)
6. At another prompt type: accuracy iris_test.pat iris.net iris_test.txt (but don't press return)
7. Create a network for iris according to the above stipulations
8. Train the network and save out the weights to iris.net using Save As (type network name as "iris").
9. To test accuracy on one of the datasets, go the appropriate prompt and press return.
10. Train the network some more, save the weights out, and test the accuracy on any datasets you want by rerunning the appropriate command at the appropriate prompt.
11. Repeat this process for every accuracy point you'd like.

If you find an error with this program, contact me immediately and I will correct it. Also, if there are any corrections made they will appear in this section in red. So, check back to see if anything has changed. Thanks. :)

Accuracy: A Tool that prints the accuracy of a neural network on a dataset given it's output activations.

 Download it here.

USAGE: accuracy2 resFilename

How to use the tool:

1. In the Control panel's Patterns tab, change the training set to the dataset you want to test the accuracy on. (If it's not already in the list open it using "File | Open").
2. Save out the output activations. This is done using "File | Save data".

3. Fill out the dialog box as shown in the example above. Make start pattern 1 and end pattern the number of patterns in the dataset. The include output files MUST be checked and the include input files MUST NOT be checked. Choose the create radio button and give the file a name.
4. Run accuracy.exe on the outputed .res file.
5. Make sure you change the training set choice in the control panel back to your training set before you continue training.

The network must have binary outputs, one output for each class. The expandoutputnodes.exe tool above can be used to do this.

This can be used for finding accuracy on the training set, validation set, or test set.

If you find an error with this program, contact me immediately and I will correct it. Also, if there are any corrections made they will appear in this section in red. So, check back to see if anything has changed. Thanks. :)

Norm: A Tool that normalizes the inputs and outputs of a dataset to between 0 and 1.

 Download it here.

USAGE: norm patFilename outputFilename