After some experimentation with LSTM topologies, I ended up with a 8 layer network with 32 LSTM units per hidden layer. These networks take a lot longer to train and the MSE for my 10,000 iteration test was 0.0201 (worse than other topologies). The amazing part is that using the feedback mechanism to reconstruct the sequence, scene transitions are preserved! In my previous single and 4 layer LSTM tests, the scene changes were not reconstructed using feedback in the model. The image below shows the results.
The left panel is the original sequence and the right panel is the reconstruction due to feedback. The X axis is a flattened (19×8) position histogram for a subset of clusters and the Y axis is time over the 500 time-step training data. The darkness reflects the number of instances of a particular cluster in a particular cell of the histogram. Interesting observations include that
- the scene structure is preserved (but curiously, delayed by 11 time-steps),
- noise in the original sequence is obviously smoothed out, and
- (not visible in this visualization) the reconstruction tends to be more dense, meaning that more percepts are activated in dreaming than in the original “perception”
The next step is to use these cluster probabilities to generate a LUT of clusters and positions I can use to generate what the dream sequence would look like.