Synthetic Dataset

Posted: May 1, 2014 at 5:33 pm

In order to deal with the problem of static dreams Philippe asked me to create a synthetic data set that has particular temporal properties. The idea is that we can use it to get a sense of both the distribution of percepts over time and the resemblance / boundedness of dreaming and mind wandering compared to perception. The data-set is 2000 frames and contains three objects: a blue circle that gets bigger, a red circle that gets smaller, and a green rectangle that moves from the left to the right. The background toggles between white and grey every 200 frames. Additionally, there was a single all black frame at the end of the data-set to mark epochs. Following are the first and last frames of the synthetic data set, not including the trailing black frame:

store-0000001store-0001999

The change in background colour triggers arousal and therefore perception in the system. Thus the data-set was sampled 10 times over the 2000 frames. The max number of percepts was set to 30 to allow each state of each shape to have its own percept where no merging is necessary. The DM system was trained on these frames and repeated over 50 epochs. Following is a plot of the test:

log.debug

All of the dreams and mind-wandering results of this test showed static results. The following image shows all percepts dumped by the system after one epoch stacked on top of one another:

percepts

I extracted the state vectors (presence of clusters) that correspond to training iterations to obtain the following:

states-training

As there were enough clusters, the training samples fed to the predictor are stable over every epoch. I extracted one epoch from this test and wrote a new program to investigate MLP learning of this data, whose results follow. First we’ll look at a number of tests running over 15 epochs where the aim to learn the sequence. In the resulting images below each column is the network output and corresponds to each input column. The output images are offset by one pixel as no t-1 pattern exists. Both thresholded and raw output values and shown below.

Input Sequence 3 Hidden Units
Constant Learning Rate
15 Hidden Units
Constant Learning Rate
15 Hidden Units
Decreasing Learning Rate
states-training-1-epoch 15epochs_3hidden_constantLearningRate_rawFeedback-valIndex15epochs_3hidden_constantLearningRate_rawFeedback-valIndexCont  15epochs_15hidden_constantLearningRate_discretizeFeedback-valIndex 15epochs_15hidden_constantLearningRate_discretizeFeedback-valIndexCont 15epochs_15hidden_decreasingLearningRate_discretizeFeedback-valIndex 15epochs_15hidden_decreasingLearningRate_discretizeFeedback-valIndexCont

The example with three hidden units corresponds to the current implementation of the Dreaming Machine where the number of hidden units is set to a 10th of the number of network inputs. The network with three hidden units did not learn the sequence well no matter how many epochs were used. When the number of hidden units is half the number of inputs, learning over 15 epochs is more successful. The following plot shows the error rates of various tests after each epoch:

plot

For this data-set, over 15 epochs, the difference between using a linearly decreasing learning rate from 1 to 0 and a constant learning rate of 1, was not significant in terms of the quality of results (although the decreasing learning rate does report lower error). As shown above, the sequence is well learned in all cases when 15 hidden units are used. Unfortunately, this sequential learning is not manifest in feedback in the predictor. While this network does produce state t+1 when fed state t, feeding the network state t does not lead to outputting states beyond t+1 (t+2, t+3, etc.) when feeding t+1 back as an input to the network. The following table shows these results:

Constant Learning Rate
Discretized Feedback
Constant Learning Rate
Raw Feedback
Decreasing Learning Rate
Discretized Feedback
Decreasing Learning Rate
Raw Feedback
15epochs_15hidden_constantLearningRate_discretizeFeedback-valFeedback
15epochs_15hidden_constantLearningRate_discretizeFeedback-valFeedbackCont
15epochs_15hidden_constantLearningRate_rawFeedback-valFeedback
15epochs_15hidden_constantLearningRate_rawFeedback-valFeedbackCont
15epochs_15hidden_decreasingLearningRate_discretizeFeedback-valFeedback
15epochs_15hidden_decreasingLearningRate_discretizeFeedback-valFeedbackCont
15epochs_15hidden_decreasingLearningRate_rawFeedback-valFeedback
15epochs_15hidden_decreasingLearningRate_rawFeedback-valFeedbackCont

While the network will produce the correct first state (t+1 from t), if t+1 is fed back as an input, then the learned sequence is not reproduced. In order to augment the feedback process we used two methods: In discretized feedback the continuous output units are thresholded before they are fed back into the network. In raw feedback the continuous output of the network is fed back without modification. In any case, the network stabilizes after a small number of iterations at which point it either produces a periodic or static pattern. The parameters used to train these networks did not appear as significant as the initial network weights to the networks feedback behaviour. For example, the following results are the output of non-discretized feedback in the same network using a constant learning rate, 15 hidden units and 15 epochs where the only difference is the initial network weights:

TMP-15epochs_15hidden_constantLearningRate_rawFeedback-A-valFeedback
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-A-valFeedbackCont
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-B-valFeedback
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-B-valFeedbackCont
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-C-valFeedback
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-C-valFeedbackCont
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-D-valFeedback
TMP-15epochs_15hidden_constantLearningRate_rawFeedback-D-valFeedbackCont

From what we have learned here we have two concrete implications for the Dreaming Machine: (a) the number of hidden units may need to be increased considering the number of inputs in the full system (4000 percepts); and (b) even if the network is learning predictions, feedback will not manifest those predictions. The periodic and static dreams of the system shown to date are caused by the lack of predictions being manifest in network feedback. This is a significant problem, and likely means a different machine learning method for the predictor is needed. Following is the code used to generate these experiments:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
#include "floatfann.h"
#include "fann_cpp.h"

#include <ios>
#include <iostream>
#include <fstream>
#include <sstream>
#include <sys/time.h>

#include "opencv2/highgui/highgui.hpp"
#include "opencv2/core/core.hpp"
#include "opencv2/imgproc/imgproc.hpp"

#define maxIterations 50

using namespace std;
using namespace cv;

// get the current time (for rough profiling)
double getTime() {
    timeval rawtime;
    double time;

    gettimeofday(&rawtime, NULL);
    time = rawtime.tv_sec+(rawtime.tv_usec/1000000.0);
    return(time);
}

// Test function that demonstrates usage of the fann C++ wrapper
void run(string inputData, string networkFilename, string imageFilename)
{
    double t1, t2;
    const unsigned int num_layers = 3;
    //const float desired_error = 0.01f;
    //const unsigned int max_iterations = 1000;
    const unsigned int iterations_between_reports = 1;

    string imageBasename = imageFilename.substr(0, imageFilename.rfind("."));

    // load datafile
    FANN::training_data data;
    if (data.read_train_from_file(inputData)) {

        FANN::neural_net net;
        net.create_standard(3, data.num_input_train_data(), int(data.num_input_train_data()/2), data.num_input_train_data());

        net.set_learning_rate(1); // was 2, 0.7

        // steepness of activation function (slope of middle of sigmoid)
        net.set_activation_steepness_hidden(1.0);
        net.set_activation_steepness_output(1.0);
       
        // Types of activation functions:
        // FANN::SIGMOID is e.g. in the 0 - 1 range while FANN::SIGMOID_SYMMETRIC is in the -1 - 1 range and FANN::LINEAR is unbound.
        net.set_activation_function_hidden(FANN::SIGMOID_SYMMETRIC_STEPWISE);
        net.set_activation_function_output(FANN::SIGMOID_SYMMETRIC_STEPWISE);

        //net.print_parameters();

        // scale data from 0-1 to -1 to 1
        //t1 = getTime();
        data.scale_train_data(-1, 1); // approximately 1.35359seconds for 41,886 inputs
        //t2 = getTime();
        //cout << "data scale time: " << t2-t1 << endl;

        // Initialize and train the network with the data
        // TODO try changing this.
        net.randomize_weights(-1, 1);

        // Train network
        int epochs = 15;
        double learningRate = 1.0;
        for (int i=0; i<epochs; i++) {
            // epoch training
            //t1 = getTime();
            //net.set_learning_rate(learningRate-((1.0/epochs)*i));
            cout << "15epochs_15hidden_constantLearningRate_rawFeedback learningRate " << i << " " << learningRate-((1.0/epochs)*i) << endl;
            cout << "15epochs_15hidden_constantLearningRate_rawFeedback error " << i << " " << net.train_epoch(data) << endl;
            //t2 = getTime();
            //cout << "canonical_100000_hidden/5_clampFeedback time " << i << " " << t2-t1 <<endl;
        }

        // automatic
        //net.train_on_data(data, 1.3840422E9, 1000, 0.001f);

        // Generate image from network
        Mat masterImageFeedback = Mat(data.num_input_train_data(), maxIterations, CV_8UC1);
        Mat masterImageFeedbackCont = Mat(data.num_input_train_data(), maxIterations, CV_8UC1);
        Mat masterImageIndex = Mat(data.num_input_train_data(), data.length_train_data(), CV_8UC1);
        Mat masterImageIndexCont = Mat(data.num_input_train_data(), data.length_train_data(), CV_8UC1);

        // Validate by each index
        // for each data point (row)
        for (unsigned int i = 0; i < data.length_train_data(); i++)
        {
            fann_type *calc_out = net.run(data.get_input()[i]);

            // for each input dimention (col)
            for (unsigned int j = 0; j < data.num_input_train_data(); j++) {

                float value = calc_out[j];
                //float value = data.get_input()[i][j];

                // populate image
                // continuous
                masterImageIndexCont.at<uchar>(j,i) = char( (value+1)*128 );
               
                // discretized
                if (value > 0)
                    masterImageIndex.at<uchar>(j,i) = char(255);
                else
                    masterImageIndex.at<uchar>(j,i) = char(0);
            }
        }
       
        // Feedback validation
        // set initial input to first training sample
        fann_type *previousOutput = data.get_input()[0];

        for (unsigned int i = 0; i < maxIterations; i++) {

            fann_type *tmpOutput = net.run(previousOutput);

            // TODO discretize values to -1 to 1
            fann_type newInput[data.num_input_train_data()];
            for (unsigned int j = 0; j < data.num_input_train_data(); j++) {
                newInput[j] = tmpOutput[j];

                // populate image
                // continuous
                masterImageFeedbackCont.at<uchar>(j,i) = char( (tmpOutput[j]+1)*128 );
               
                // discretized
                if (tmpOutput[j] > 0) {
                    masterImageFeedback.at<uchar>(j,i) = char(255);
                    //newInput[j] = 1; // discretized feedback
                } else {
                    masterImageFeedback.at<uchar>(j,i) = char(0);
                    //newInput[j] = -1; // discretized feedback
                }
                //cout << int(masterImage.at<uchar>(j,i)) << endl;
            }
            previousOutput = newInput; // feedback loop.

        }

        // Write masterImage to file
        imwrite(imageBasename+"-valFeedback.png", masterImageFeedback);
        imwrite(imageBasename+"-valFeedbackCont.png", masterImageFeedbackCont);
        imwrite(imageBasename+"-valIndex.png", masterImageIndex);
        imwrite(imageBasename+"-valIndexCont.png", masterImageIndexCont);

        // Save the network in floating point and fixed point
        net.save(networkFilename);
        //unsigned int decimal_point = net.save_to_fixed("xor_fixed.net");
        //data.save_train_to_fixed("learn_sequence.data", decimal_point);

    } else
        cout << "Data file could not be loaded" << endl;
}

/* Startup function. Syncronizes C and C++ output, calls the test function
   and reports any exceptions */

int main(int argc, char* argv[])
{
    if (argc != 4) {
        cout << "Usage: " << argv[0] << " [input data] [network filename] [output image basename]..." << endl;
        cout << "Exiting..." << endl;
        exit(1);
    } else {
        try
        {
            std::ios::sync_with_stdio(); // Syncronize cout and printf output
            run(argv[1], argv[2], argv[3]);
        }
        catch (...)
        {
            cerr << endl << "Abnormal exception." << endl;
        }
    }
    exit(0);    
}

/******************************************************************************/