2. Older Tutorial

2.1 Introduction

PLearn is an open source software for machine learning, with numerous features. It can be used as an runnable software or as a library. This tutorial will help you to discover what is PLearn and how to use it. I assume that:

you have basic concepts in machine learning.
you have basic concept in object programming.
you have a PLearn binary that runs.

2.2 A basic classification problem

2.2.1 First steps

We consider the following classification problem. In a 2-D space, we have two classes. The problem is represented on figure 2.1.

**Figure 2.1:** In red, the first class. In green, the second one. In blue, the analytic decision boundary. The train examples are 0, the test ones +
$\includegraphics[width=100mm]{Figures/class}$

The data are generated with a Matlab script, called task1.m^2.1.The script generates boundary.amat, train.amat, test.amat, and space.amat.

Now, let's train a neural network on this task with PLearn.

We create the PLearn script, a kind of configuration file:

#!plearn

PTester(
  # string: Path of this experiment's directory in which to save all experiment results (will be created if it does not already exist)
  expdir = "expdir-nnet";

  # VMat: The dataset to use for training/testing (will be split according to what is specified in the testmethod)
  dataset = AutoVMatrix(specification="UCI_pima-indians-diabetes all" inputsize = 8 targetsize = 1);

  # TVec< string >: A list of global statistics we are interested in.
  # These are strings of the form S1[S2[dataset.cost_name]] where:
  #   - dataset is train or test1 or test2 ... (train being 
  #     the first dataset in a split, test1 the second, ...) 
  #   - cost_name is one of the training or test cost names (depending on dataset) understood 
  #     by the underlying learner (see its getTrainCostNames and getTestCostNames methods) 
  #   - S1 and S2 are a statistic, i.e. one of: E (expectation), V(variance), MIN, MAX, STDDEV, ... 
  #     S2 is computed over the samples of a given dataset split. S1 is over the splits. 
  statnames = [ ]
  # TVec< TVec< string > >: A list of lists of masks. If provided, each of the lists is used to compose the statnames_processed.
  # If not provided the statnames are those in the 'statnames' list. See the class help for an example.
  statmask = [ [ "test#1-2#" ] [ "*.E[stable_cross_entropy]" "*.E[binary_class_error]" ] [ "E[*]" ] ]
  # PP< Splitter >: The splitter to use to generate one or several train/test pairs.
  splitter = TrainTestSplitter(
    test_fraction = .10
    append_train = 1
  ) ;

  # PP< PLearner >: The learner to train/test
  learner =
    NNet( 
      # int:     number of hidden units in first hidden layer (0 means no hidden layer)
      nhidden = 10  ;
      # int:     number of output units. This gives this learner its outputsize.
      #     It is typically of the same dimensionality as the target for regression problems 
      #     But for classification problems where target is just the class number, noutputs is 
      #     usually of dimensionality number of classes (as we want to output a score or probability 
      #     vector, one per class)
      noutputs = 1  ;
      # double:     global weight decay for all layers
      weight_decay = 0.0  ;
      # string:     what transfer function to use for ouput layer? 
      #     one of: tanh, sigmoid, exp, softplus, softmax, log_softmax 
      #     or interval(<minval>,<maxval>), which stands for
      #     <minval>+(<maxval>-<minval>)*sigmoid(.).
      #     An empty string or "none" means no output transfer function 
      output_transfer_func = "sigmoid"  ;
      # Array< string >:     a list of cost functions to use
      #     in the form "[ cf1; cf2; cf3; ... ]" where each function is one of: 
      #       mse (for regression)
      #       mse_onehot (for classification)
      #       NLL (negative log likelihood -log(p[c]) for classification) 
      #       class_error (classification error) 
      #       binary_class_error (classification error for a 0-1 binary classifier)
      #       multiclass_error
      #       cross_entropy (for binary classification)
      #       stable_cross_entropy (more accurate backprop and possible regularization, for binary classification)
      #       margin_perceptron_cost (a hard version of the cross_entropy, uses the 'margin' option)
      #       lift_output (not a real cost function, just the output for lift computation)
      #     The first function of the list will be used as 
      #     the objective function to optimize 
      #     (possibly with an added weight decay penalty) 
      cost_funcs = [ "stable_cross_entropy" "binary_class_error" ]  ;
      # PP< Optimizer >:     specify the optimizer to use
      optimizer = 
        GradientOptimizer( 
          # double:     the initial learning rate
          start_learning_rate = 0.05
          # double:     the learning rate decrease constant 
          decrease_constant = 0.001  ;
        );
      # int:     how many samples to use to estimate the avergage gradient before updating the weights
      #     0 is equivalent to specifying training_set->length() 
      batch_size = 0  ;
      # int: The stage until which train() should train this learner and return.
      # The meaning of 'stage' is learner-dependent, but for learners whose 
      # training is incremental (such as involving incremental optimization), 
      # it is typically synonym with the number of 'epochs', i.e. the number 
      # of passages of the optimization process through the whole training set, 
      # since the last fresh initialisation.
      nstages = 100  ;
      # int: Level of verbosity. If 0 should not write anything on cerr. 
      # If >0 may write some info on the steps performed along the way.
      # The level of details written should depend on this value.
      verbosity = 10  ;
      seed = 12345
    );

)

We call this file NNnet.plearn.

Now we have to specify the dataset. In a classification problem, the dataset is a set of examples and their associated classes. We have already created such a file in Matlab, called train.amat. Now we have to specify to PLearn that the two first columns of “train.amat” contain the data, and the last column the class label, which is in ${-1,1}$ , corresponding to the output of a tanh.

All this tasks are made with the following file, called train.vmat:

AutoVMatrix(
	specification = "train.amat"
	inputsize = 2
	targetsize = 1
	weightsize = 0
)

Now, let's train the network with the folllowing command:

plearn learner train NNet.plearn train.vmat final.psave

Then test on the original space:

plearn learner compute_outputs final.psave space.amat out.pmat

We need two additionnals commands to view the result of the network with matlab:

plearn vmat convert out.pmat out.amat converts from a binary to an ascci file.

tail +3 out.amat > result.mat removes meta-information at the beginning of the file.

Then we execute the result_task1.m script in Matlab to plot the result, as in figure 2.2.

**Figure 2.2:** Analytic and learned boundary
$\includegraphics[width=100mm]{Figures/decbound}$

2.2.2 What have we done?

NNet.plearn contains informations about the neural net. train.vmat contains informations about the the dataset.

PLearn is built with a direct help system. For example:

plearn help NNet

plearn help AutoVMatrix

plearn help GradientOptimizer

This commands gives you an accurate information.

Here are the more important thing about PLearn.

PLearn is an object oriented software. The scripting language is object oriented. “plearn help NameOfTheClass” gives you the corresponding information

Note that a lot of others parameters exist for each classes but they have default values.

2.3 A second example

We now consider a regression problem. We want to train a neural network to predict a sinus.

First, we generate the data with the task2.m matlab file.

Then we perform the task within only one script, the following regression.plearn.

PTester( 

learner = NNet
    (
    nhidden =  10 ;
    noutputs = 1 ;
    output_transfer_func = "";
    hidden_transfer_func = "tanh"  ;
    cost_funcs = 1 [ mse ]  ;
    optimizer = GradientOptimizer(
                    start_learning_rate = .01;
                    decrease_constant = 0;
                    )
    batch_size = 1  ;
    initialization_method = "normal_sqrt"  ;
    nstages = 500 ;
    verbosity = 3;
    );

expdir = "tutorial_task2"  ;

splitter = ExplicitSplitter(splitsets = 1 2 [
    AutoVMatrix(
        specification = "reg_train.amat"
        inputsize = 1
        targetsize = 1
        weightsize = 0
    ) 
    AutoVMatrix(
        specification = "reg_test.amat"
        inputsize = 1
        targetsize = 1
        weightsize = 0
    )
    ]
) ;

statnames = ["E[E[train.mse]]" "E[E[test.mse]]" ];

);

Let's run Plearn on this script: plearn regression.plearn

Then test on the original space: plearn learner compute_outputs tutorial_task2/Split0/final_learner.psave space2.amat out.pmat

We need two additionnals commands to view the result of the network with matlab:

plearn vmat convert out.pmat out.amat

tail +3 out.amat > result.mat

Let's view the result with a matlab script, result_task2.m:

**Figure 2.3:** Analytic and learned function
$\includegraphics[width=100mm]{Figures/reg}$

2.3.1 What have we done?

We encapsulated the experiment in a powerful scriptable class called “PTester”.

The command plearn help PTester gives you the corresponding information. Note that a lot of others parameters exist for PTester but they have default values.

Furthermore, explore all the files that a PTester creates:

cd tutorial_task2/
ls
plearn vmat view split_stats.pmat
plearn vmat view global_stats.pmat
cd Split0
ls
less test1_stats.psave
less final_learner.psave

2.4 Conclusion

This short tutorial shows a small part of PLearn. Continue by yourself and have a nice Plearn time!