On-line Appendix

Introduction

Wilson, D. Randall and Tony R. Martinez, "The General Inefficiency of Batch Training for Gradient Descent Learning," to appear in Neural Networks, 2003.

1. Machine Learning Database (MLDB) Experiments to test batch vs. on-line training.
2. Digit Speech Recognition Experiments to test mini-batch training.

For each task, 60% of the data was used for training, 20% for a hold-out set, and the remaining 20% as a final test set. The hold-out set was used to test the generalization accuracy after each epoch of training, and the final test set was used to measure accuracy at the epoch where generalization was the highest on the hold-out set.

Each task was trained using both on-line and batch training methods, and in each case learning rates of 0.1, 0.01, 0.001, and 0.0001 were used. Ten trials were run for each of the four learning rates for both methods on each of the 26 tasks, for a total of 10*4*2*26 = 2,080 runs.

More details about the neural network architecture and training are available in the paper.

Each neural network was trained for 1,000 epochs for learning rates 0.1 and 0.01; 5,000 epochs for 0.001; and 10,000 epochs for learning rate 0.0001. Each neural network was then tested on the hold-out set and the results for each epoch over the ten trials were averaged. A few tasks had generalization accuracy that was still rising after the number of epochs listed above. In those cases, additional training epochs were used in order to determine how long batch and on-line training took to train to a maximum accuracy in each case.

Complete, Raw MLDB Results

A csv file can be opened by Microsoft Excel, but is also easy to parse. Each line consists of the same number of fields (values), with a comma used to delimit each field.

Each of the raw results files contains the following columns:

• The name of the database (e.g., "australian") at the top of the first column.
• Itr: Number of training iterations (epochs) up to and including the row. (If any files contain a "0.1" instead of an integer, this is to indicate what the accuracy was with the random initial weights, and a "0.1" was used instead of "0" to avoid divide-by-zero errors in Microsoft Excel graphs).
• Hold-out set accuracy for each method. Each column has a heading to indicate which run is represented in the column. The format of the column heading is
  • c = continuous = on-line; or b = batch.
  • A number 0..9 immediately after c or b is which of the ten trials it is. (No number means it is the average, which is what is displayed in the leftmost columns).
  • A dash (sort of takes the place of the decimal point in the learning rate).
  • The learning rate (w/o the decimal point)
  • A "t" if the accuracy is on the test set instead of the hold-out set.
  For example:
  • c-1 is the average on the hold-out set over all 10 runs of on-line training using a learning rate of 0.1.
  • b0-01t is the accuracy on the test set on run 0 (out of 0..9) of batch training using a learning rate of 0.01.

Average Raw MLDB Results

Selected Charts

• australian.xls.
• sonar.xls.

MLDB Results Overview

It is not trivial to boil down thousands of experiments into a single chart, but these charts attempt to do just that.

Each row contains the name of one of the MLDB datasets, along with the following information:

• Classes: Number of output classes in the dataset.
• Instances: Number of instances in the dataset (used for training, hold-out and testing).
• MacC: Maximum generalization accuracy achieved by on-line ("continuous") training.
• MaxB: Maximum generalization accuracy achieved by batch training.
• MaxAcc: Maximum generalization accuracy of on-line or batch
• Diff: Difference between on-line and batch (i.e., batch - on-line)
• CSafeLR: Fastest learning rate that allowed on-line ("continuous") training to get within 0.5% of its highest accuracy.
• BSafeLR: Fastest learning rate that allowed batch training to get within 0.5% of its highest accuracy.
• CSafe: Number of training epochs needed to get within 0.5% of on-line's highest accuracy using the learning rate shown under "CSafeLR".
• BSafe: Largest learning rate that allowed batch training to get within 0.5% of its highest accuracy.
• xSlower: Number of times slower batch training is than on-line training, when both are allowed to use their "best" learning rate, i.e., the one that gets them within 0.5% of the highest accuracy in the fewest number of epochs.
• xSlow by hand: Same as xSlower, but has a few entries adjusted by hand. For example, for the letter-recognition dataset, on-line got to an accuracy of 83.53% after 4968 epochs. Batch training's highest accuracy was 73.25% after 9915 epochs, but only because training stopped after 10000 epochs (and these epochs took a long time due to the size of the dataset). Examining the accuracy trends on a log scale made it clear that batch training was lagging by 100 times behind on-line, so the "xSlow" column contains this value as a better estimate of how much slower batch was progressing on that dataset.

The first table defines "safe" as coming within 0.5% of the max accuracy achieved by either training method. Thus, if batch training never comes within 0.5% of on-line training, then it must use whatever learning rate gave it the absolute best accuracy. The second table defines "safe" as coming within 0.5% of the max accuracy of any learning rate for just the training method itself.

The rest of the columns contain the following information for each learning rate: