CS 478 Tool Kit documentation

 

We will always use ARFF files for our datasets, and we will make the assumption that all data will fit in RAM. Details on ARFF are found here.   A collection of data sets already in the ARFF format can be found here.

 

A basic tool kit is provided in C++, Java, and Python to help you get started implementing learning algorithms. You are also welcome to code up your own toolkit or modify the source code made available here however you want.  If you do so, here are a few things to keep in mind:

 

• Some of the labs require a high number of computations. Using an interpreted language may result in longer runtimes. (The TAs are not sure exactly how much longer, since we have never tried any of the labs in an interpreted language. Just be forewarned.)
• Advice about dealing with discrete-valued data: A common way to represent each "instance" (a.k.a. "pattern") is to use a vector of numbers. For discrete values, these numbers could be an index to the "name" of that value in the metadata.

 

The CS 478 tool kit is intended as a starting place for working with machine learning algorithms. It provides the following functionality to run your algorithms:

 

• Parses and stores the ARFF file
• Randomizes the instances in the ARFF file
• Provides four evaluation methods (A more detailed description of these methods is found here):
1. Training set method:
   The model is evaluated on the same data set that was used for training
2. Static split test set method:
   Two distinct data sets are made available to the learning algorithm; one for training and one for testing
3. Random split test set method:
   A single data set is made available to the learning algorithm and the data set is split such that x% of the instances are randomly selected for training and the remainder are used for testing, where you supply the value of x.
4. N-fold cross-validation method
   A single data set is made available to the learning algorithm which is partitioned into N equally sized subsets.  Each subset is used once for evaluating the learning algorithm while the remaining instances are used for training.  The results of the N runs are then averaged to provide the final accuracy estimate.
• Parse command-line arguments
• Normalize attributes

 

Build Instructions for the Java version:
Download the zip file here

Build Instructions for Linux:

   1. Unzip the zip file

   2. javac *.java

(We do not use windows, but there is nothing fancy and I imagine that it should work in Microsoft Visual C++. You will need to create a project solution. If you use windows and have trouble, come see us for help.)

 

Build Instructions for the C++ version:

Open the terminal

wget http://axon.cs.byu.edu/~martinez/classes/478/stuff/toolkitc.zip

unzip toolkitc.zip

cd toolkit/src/

make opt

 

Test to make sure that the toolkit works:

mkdir datasets

cd datasets/

wget http://axon.cs.byu.edu/~martinez/classes/478/stuff/iris.arff

cd ..

./bin/MLSystemManager -L baseline -A datasets/iris.arff -E training

 

You should see the results for a baseline classifier (33% accuracy on iris)

Usage Instructions for both version:
MLSystemManager -L [LearningAlgorithm] -A [ARFF_File] -E [EvaluationMethod] {[ExtraParameters]} [-N] [-R seed]

Where the -N will normalize the training and test data sets (Normalization max and min will come from the training set).

The -R allows you pass in a seed for the random number generator. By default each time you run the code, the data set will be shuffled differently. If you wish to produce the same shuffle, provide a seed such as 1 or 2.

Possible evaluation methods are:

• Training (using same data set for training and testing):

./MLSystemManager -L [LearningAlgorithm] -A [ARFF_File] -E training

 

• Static Split (2 distinct datasets/ARFF files; one for training and one for testing:

 

./MLSystemManager -L [LearningAlgorithm] -A [ARFF_File] -E static [TestARFF_File]

 

• Random Split (1 dataset is split randomly providing x% for training and the rest for testing):

 

./MLSystemManager -L [LearningAlgorithm] -A [ARFF_File] -E random [PercentageForTraining]

 

• N-fold Cross-validation (1 dataset is partitioned into N partitions. The learning algorithm is evaluated on each portion and then the average accuracy is returned.:

./MLSystemManager -L [LearningAlgorithm] -A [ARFF_File] -E cross [NumOfFolds]

 

Here is an example of using the C++ ML tookit and the output:

 

./MLSystemManager -L dummy -A ../Research/dataSets/iris.arff -E training -N
Dataset name: iris
Dataset is normalized.
Number of instances: 150
Learning algorithm: dummy
Evaluation method: training

Accuracy on the training set:
Output classes accuracy:
Iris-setosa: 1
Iris-versicolor: 0
Iris-virginica: 0
Set accuracy: 0.333333

Accuracy on the test set:
Output classes accuracy:
Iris-setosa: 1
Iris-versicolor: 0
Iris-virginica: 0
Set accuracy: 0.333333

Time to train: 5.96046e-06 seconds  (Note: If the simulation starts before midnight and ends after, the time will not be accurate)

 

A DummyLearner class is provided that classifies all instances as the majority class (BaselineLearner). This class can be used as a template for creating your own learning algorithms. The instances are stored in a vector of vectors of doubles (c++ version) or an ArrayList of ArrayLists of doubles (java version). For further implementation questions, please see the TAs. When creating a new learning algorithm, you need to add the include line in the MLSystemManager file and it must inherit from the Learner class.