Philippe has encouraged me to work through the design process of developing the system through writing a paper for Computational Creativity. After taking a day to write an introduction and a section on theories of dreaming and mental imagery, I started working through the diagram from the PhD proposal keeping in mind new insights regarding long term memory that came out during New Forms and the proposal defence:
The figure is quite similar to that previously posted, except I’ve collapsed foreground and background into one module, reflected the fact that percepts in memory are merged with patches in perception and added boxes for short and long term memory (STM / LTM). In making the diagram it occurred to me that the addition of long-term memory causes some unforeseen issues. LTM is required because it’s not feasible to compare every segmented patch from every input frame with every percept in memory. This either leads to exponential processing time per frame, or too few percepts stored in memory.
I have been working under the assumption that dreams are composed of long-term memory units because of the complexity and richness of their content. The quality of dreams is not conducive to them being limited to only short-term memory. According to the “integrative interpretation” proposed in the PhD proposal, dreams are initiated by latent activation in the same set of representations as used during perception. The latent activation would presumably be in short-term memory, so how could activation in short-term memory effect activation in LTM? Not to mention at all how memories move from STM to LTM. Until this point I’ve largely avoided the memory literature because its is so massive and complex that it would take a significant commitment to even figure out whether it would help. Of course a good entry-point is that sleep (and perhaps dreams) have been associated with memory consolidation (movement from short-term to long term memory).
One of the issues Philippe has with the project is my interest in neurological mechanisms rather than high level cognitive ones. One of the reasons for this is that there does not seem to be a cognitive theory of dreams, at least I have not found one. Theories of dreaming tend to be neurologically oriented.
The reason for the introduction of the LTM was based on technical needs of the system, but now the impact on the theoretical side is highly significant, and the answer is not clear. LIDA looked promising, but its a proprietary system and there don’t seem to be publications describing the nitty gritty details of how it deals with STM and LTM. It seems I’m left needing to answer some major questions:
- How do memories move from STM to LTM?
- How is the activation in the LTM related to perception, and how is it initiated?