Dreaming Machine #3 Rough Notes
Rough Ideas
TODO: Go through this, structure and keep significant points, move bad/old ideas to a deprecated heading.
It could be interesting to have the system
have a more accurate model of consciousness, meaning differing levels
of consciousness between waking and dreaming. The transitions between
these states could be very interesting. (See REM sleep and dreaming: towards a theory of protoconsciousness.)
- The machine has no consciousness, but what about the transition between waking and dreaming? Dreaming and Hallucinating?
Create a developmental
protoconscious state for the system. It contains a predetermined model
of its role in the world (the movement of the pan/tilt, the taking of
pictures, basically every feature of the system controls in the outside
world (actuators, what about sensors??) maybe it also has the idea of
an image. In the protoconsciousness these elements are played with in
REM sleep. The mapping between sensor images and camera movements are
determined by this process. Perhaps a machine learning algorithm. But
without sensor data what would the machine learn from? How would this
initial structure be determined? what happens if it was all based on
noise, RF in the context?
- Out of scope of resolved by latent activation.
How could the machine generate 80%
non remembered content? Perhaps the colour and spacial relations
between pixels in small groups could be used. A green pixel next to a
red pixel seen in the world could be included in a new synthesized
image. How to encode these relations?
- This
is more about the atoms of experience, which is dependent on the
particular cognition of this machine. What about hierarchy? What about
reproducing a particular arrangement of objects? What about location?
It may be a problem for a memory system to be divorced from a survival relation to its environment (See the evolution of multiple memory systems). What is the purpose of DM's memory?
- survival perspectives are out of scope, a machine has no need for survival. Memory exists to represent the world.
Perception:
Colour is important
spacial structure (edge detection, FFT)
how to use Perception backwards in synthesis? (see below)
Godel Escher Bach
- Stimulation as inherent reward
- dreams as self-stimulation
- Perhaps all the data can be stored in the
system if we store only the abstractions and not the images. Then maybe
we can use a learning technique like the SOM on all the stored data,
and not have to worry about online learning? Not very
neurobiological... Philippe suggests we capture a small set (3) images
and do the analysis on all three.
- Perhaps possible, the patterns for each visual prototype will be stored.
- Slow
processing means that images will have to be processed by in groups.
(based on background subtraction? (for fast change detection?)
It would be really interesting to
include genetics somehow. Like an SNN that creates a population of
enrematic memories. These genetic sequences result in the various
representations of the system.
What if all the constants in the
system (ie the starting point for the free-running circadian) were
encoded in a genetic plan. The DM system could then be used in a
genetic algorithm. What would the fitness be?
- Hippocampus & Time: What about the
storage of the time of events? A window with a fixed number of time
steps. How could this be used in the association cortex? What about
locations of prototypes for that matter?
- Time
and space should be treated by the same mechanism, but its not clear
how this would be. What is the "location" of a moving object? The
average of all its positions? the start? the end?
Image-Schemas are actually representations
of movement and location in space! If we go with a video camera, then
consistency of an object across multiple frames can be used and
representations of the movement of objects is possible. What features
of the movement would make sense to extract? Just because the sensor is
time-based does that require that the object analysis is time-based?
- Moving away from Mandler.
Create the dissertation
that is based on this development of the system. Each stage is like a
different artwork, an has its own chapter.
- Moving away from strict conceptions of development.
Perceptions change fidelity (from
the POV of consciousness) fuzzy while dreaming, specific while awake
due to reinforcement from external stimulus.
- It seems unlikely that prototypes would change their characteristics depending on sleep/waking states.
- An
approach to the representation is to think of the viewer as
consciousness (homunculus) on which cognitive processes project. Is it
a contradiction treating the audience as both consciousness and other
subjects in the world?
- Going this direction. An interesting twist on self-consciousness.
- Maybe the high degree of activation from the world drowns out the "dream" state that is always happening.
- dreaming as a method of developing concepts. (memory integration, development of self, offline)
- I have not revisited this.
If the systems only measure of
emotion (and survival) includes audience attention. and then associates
dark images (when there are no people) with a negative emotion, then
the lack of people could trigger dark images. As soon as someone gives
it attention, then its emotional state would switch and brighter images
may be seen. maybe this is implemented as a greater probability of dark
images with no face stimulation.
Hemispherical Asymmetry
REM dreams are negative, right-brained and associated with the creative side.
NREM dreams are positive, left-brained and associated with the analytic side.
- Steven thinks this is all bunk.
Association Network
- Major References: TODO: Change to reflect current state? (physiology?)
- Homoeostasis and cognitive coherence
- Paul Taggart, unclear to me.
- How to determine the shape of the propagation of activation? (why propagate to red, and not to squareness?)
- 1. The system has an innate set of features, this
limits the dimensionality of connections. Therefore each connection
between neurons happens for each of these dimensions (A is connected to
B in terms of redness, C to B in terms of shape).
- 2. In order to constrain on which
channels/features/dimensions activation is propagated, the connection
strength could be queried first. On what dimension is A and B most well
connected? Propagate in that direction.
- Since each dimension has its own weight, its
possible that kind of constraint would simply result from the system.
Low connection weights would not allow signal propagation. In cases
where multiple channels are well connected, how would this be
constrained?
- Attention: Does the VQ system allow
the machine to "ignore" habituated stimulus it has already seen? How
could the VQ cause the computer vision system to ignore certain things?
It would be great if such a mechanism could be used to constrain
processing to use less computational resources (ignore regions of the
input).
- Habituation: Would a large difference between
a new stimulus and a prototype lead to high activation? Or rather would
a small difference leading to low activation be habituation?
- Piagetian Differentiation? Could
the subsequent activation of two prototypes increase the correlation
between those prototypes? The dream could be a result of the
association network training on prototypes. Why
would we assume that two objects placed nearby in time would have
anything to do with one and other? By what mechanism can links be made
between prototypes?
- correlation (distance) of particular dimensions of features. What are the features?
Figure intention (need of attention) into this. It is the agent's intention that connects signifier from signified after all.
- The system sees a blue ball and a blue truck, they are separate prototypes and not seen in close succession. How is the link made between them? Its
not that a blue ball and red truck are seen in sequence, but that they
are seen in the same image (the correspondent to a subsequent saccades.)
- A blue prototype will be linked with other blue prototypes (regardless of shape)
- Could each prototype manifest a projection for each
dimension (shape, colour, location) into the association network (or a
different association network for each dimension). The blue ball
projects into shape colour and location. Then (maybe later) the blue
truck projects its properties.
- This is now manifest with connections based on feature.
Each projection has a
line of sight, if it sees another projection within that region of
sight it links to it. this link starts off as the distance between the
points, but for each activation of blueness (from any prototype) the
blue points are more connected. The activation of any blue prototype
then activates the other blue prototypes. The concept of blue exists,
but it can still only be activated by a present object. How does an infant group a set of objects by similarity?
- The association along one dimension (first order?) seems to happen before its possible to associate on other dimensions (second order?). Which dimension is first? The one most different than the other prototypes?
- The priority of dimentions is discussed in "Neuron Design" in Dreaming Machine #3 System Design. The mapping to first and second order operations is likely problematic. How associations cross feature boundaries is crucial.
Perhaps the choice of what direction an association can be followed is a function of the second layer in the network
(that deals with relations of relations). For example two groups of two
classes of objects are presented. In terms of spatial location they
could be one large cluster, or two smaller ones. Knowing that there are
two groups could be informed by the fact that there is a clear feature
distinction between the two groups. How would this work?
Eventually the concept of the group would need to be abstracted from
any features that make the groups similar. That is the idea of a group
being based solely on location?
- since development depends on the physical
manipulation of objects it could be that associating objects by spatial
location is a dead end and only applies in the local situation of
playing with objects.
- Once the network is complex enough almost anything will activate many regions. Maybe a stochastic system
is worthwhile here that only activates in a direction that was most
stimulated last (latent stimulation) or are the connection strengths
sufficient for limitation? How to decrease connection strengths?
Dreams?
(emotional feedback? ie no attention from audience?)
What about expectation? A sequence of
objects in a scene are habituated, what does dishabituation look like
in this system? For prototypes the degree of stimulation is based on
difference between new stimulus and previous stimulus. What about the
difference between a associative pattern and a sequence of
associations? You habituate a baby to seeing a red ball and then a
green ball. Then you show a red ball without the green ball, this
causes dishabituation, how to make a machine dishabituate? And how to express that?
- Out of scope.
What happens If we separate exafference and
reafference? The system could then look for things it imagines. Where
would the imagined image come from? By definition it would have to be a
differentiated signifier.
- This is one level of abstraction from the
prototypes, it follows that this process should continue at higher and
higher levels of abstraction. From atoms, to associations, to networks
of associations to networks of networks.
- Network to represent concepts: Prototypes are differentiated, training an association network. The patterns of association in this network train a second network.
We have two objects in the same place, they generate prototypes and the
association network associates them based on location. (One has a link
to the other) The strength of this link trains a second network, where
the concept of location is the activation of a particular unit. So
groups of similar features in a particular dimension in one network are
manifest as single units in the second network. Then
association between units i the second network as associations between
concepts? Its unclear how the first network would train the second.
Concepts are intrinsic in this case, objects have features and
locations in space, so there are two dimensions and two concepts. Using
a network to represent that which is
innate implicit? may not make sense. What about a second network dealing with subclasses, objects more close to one and other?
Would this network be linguistic?
What about meta-representation? Clearly not
an aim in this project, but an interesting intellectual thought
regarding abstraction. Best to stick with first and second order
operations (pre-operational)
Accommodation? What if the network of associations could actually change the prototypes?
Up to this point we're really only in the assimilation mode, the world
impacts the system, and it tries to make sense of it. Once the
association network is sufficiently complex it could have emphasis over
the initial prototypes. That is the notion of blueness could transform
the way the blue prototypes are constructed, or even transform the
prototypes directly. This is more like accommodation, where the
concepts in the association network transform the prototypes
themselves. In a way this does refer to the idea of accommodation
generating signs, which are applied back to the SOM-like organization
of prototypes.
- Likely out of scope, but very interesting. (changing prototype generating mechanisms)
Onscreen image or pattern of activation as action-schema: In Piagetian
theory a particular action schema is associated with a cue, where the
schema satisfies some need of the agent. When the cue is available
(preoperational) the schema is executed. So we have the cue
(perception) the need/desire, and the action scheme. For DM3 the action
scheme could be the visual representation on the screen. This is the
way DM2 works really, for each activation of a prototype (cue) a
corresponding schema (image) is presented onscreen. Since the image is
an artwork, a logical need/desire for the machine is the attention of
the audience. Perhaps a full Piagetian approach is possible!
In this case a differentiated signifier is
when the prototypes does not engage a prescripted schema. Seeing a
particular thing in the world does not initiate the presentation of the
corresponding prototype. An interesting idea is a stochastic system
where a pool of schema are selected for a cue. This pool starts with
one schema, and grows through differentiation. The more schemas in the
pool the more differentiated the signifier. In terms of desire, if one
schema does not illicit attention, then its probability of being called
is decreased. So where does the pool of schema come from? It could be
the set of prototypes in the system, or it could be some processing or
juxtaposition of those prototypes. Maybe a schema is a particular
pattern of activation in the association network. Activation patterns
that are successful are enforced, while those that are unsuccessful are
not.
Imitation as base reflex response:
The camera sees a red ball, and on the screen the viewer sees what the
system thinks a red ball is on screen (a fuzzy redish blob).
For percepts to be objectified the same
object should be associated with multiple schemes. Does this mean that
multiple stochastic patterns grow as the network of objects increases?
It is not clear how the differentiation of object and subject would
mean if the outward image is the schema, as where would the pool of
different arrangements of images come from?
It is possible that having a single
desire makes the desire moot. What does it matter if one has free-will
if there is only one option from which to choose?
- Moving away from development, the image is not an action in the world (it does not transform perception in an immediate way).
- Restricted experiment to see what concepts develop from a limited set of stimulus.
Mandler: Image-Schema
Movement and interaction of object as basis of conceptual thought.
Connection with action movie analysis project.
Sperry/Bogen split-brain:
Perhaps one hemisphere is the representational (conceptual) and the
other based on low level features? This does not make sense as there
does not seem to be differences in architecture abstraction. So what is
the difference between left and right thinking? The spread of
activation is the same, the access to low level and high level visual
representations is the same... so what is different? Oh and if the
frontal lobes are "central executive" then how are two managed?
right hemisphere specializes in "global, parallel and holistic processes", (association?, low-level features?)
left specializes in "sequential and analytic processes". (analytic?, representational)
What if analytic thought is not a fine
activation, but an entirely different way of thinking that requires an
independent system?
- Steven thinks this is bunk.
Architecture
- Perception no longer separate from synthesis.
- microfeatures
- mirrored in synthesis, using NN?
- Can this system be reversed? Reconstructing images from features? Maybe the images don't exist in the human mind, just the features that are extracted?
Part of the reason why the
perceptual system is reconstructive is because the visual data received
from the retina is of low quality. When using a camera we have good
images, these good images may lessen the difference between the
original and reconstruction, or perhaps it will throw away data in the
original image in an interesting way.
- Perception
could model the retinal-geniculate-striate -> PVC -> Secondary VC. and ignore the dorsal and ventral streams (that don't see well understood) See Biopsychology Root, Biopsychology Chapter 6 (The Visual System)
FFT deconstruction of repeating patterns? Somewhat easy to reconstruct, biologically relevant?
- Use of feedback to reinforce features
What about the role of attention? Just another variable that must be specified? See A
neural model of selective attention and object segmentation in the
visual scene: An approach based on partial synchronization and
star-like architecture of connections
- While dreaming the gating of the visual field could be done (to a degree) by reducing the camera exposure to give a dark image.
Then only a very bright stimulus would wake the machine, like a
flashlight in the eye. This may require a video camera, or images would
have to be taken continuously from the camera while the machine is
asleep.
- Dream output as perceptual input while sleeping?
and perceptual input should always both happen at the same time (How
should they be blended? different channels? Perceptually does not make
sense.. if the camera is static then perhaps hallucinations would not
be too messy a signal. This would not be apparent to the viewer, it
would simply be a feedback loop.) sleep or waking would determine the
weighting between imaginative or perceptual input.
-
- Synthesis
mirrored in Perception
- consolidation: conflate the input image with the
results synthesis and feed back into memory: Results of raw image
(before segmentation) and Dream generation? (sounds more like
hallucination).
all memories are reconstructions (loftus)
see an image, abstract image (hist +shape?), adding image to input image, then abstract again, until ?
attach degree of mix of new and synthesis to last stimulus based on the degree of memory recollection?
Combination of thousands of images to create a map of objects in context
- Dreaming Machine: Perception - Synthesis (IAT 888 project)
- Memory
RSOM?
SOM for clustering (perceptual buffer)
SNN for topographic memory?
something more continuous? Incremental learning
- Attention /
Sensorimotor (connect pan/tilt to received images?)
We have a concept of presence now. So the
digital camera could be the fovea, it gets moved to the area of gross
interest. The periphery could then be a second fixed wide-angle video
camera. The video video camera would be low res, but get data quickly.
If an object of presence appears, it could direct the foveal camera to
attend to that area. Mapping X-Y to pan/tilt and size of blob to zoom.
- Ok if segmentation is realtime.
The above attention model
systematically ignores regions that do not change over time. The
architecture never changes, and so there would be no memory of it. Or
perhaps there are two memory systems, one for the background and one
for the foreground. This is the case since the adaptive background is a
memory of everything but moving objects. An alternate idea is to
flood-fill the background (assuming high enough contrast, or detecting
contrast in the centre vs the edges, and accumulate what is left over.
Rather than moving objects, this model of attention focuses on objects
surrounded by low contrast areas. These are likely rare in a real physical environment.
- Dream Update this with the new discussion regarding tononi
- decoupling of sensorimotor system from 'brain'
- change of memory recall/writing
- circadian rhythm (based on stimulation)
- waking level of stimulation
stages of sleeping (waking, NREM -> REM)
DM should dream more like a child than like an adult. See Dreaming and the brain: from phenomenology to neurophysiology
- The dream is a stochastic pattern of activations of memory, whose behaviour is influenced by the recent waking activity.
Dreams can be static even if the child can only make sense of the world through motion.
- Circadian Sleep Model
entrained by light sensor w/ extreme low-pass filter (to match retinal ganglion cells) and degree of stimulation (distance between perceptual buffer and input image?).
- or should sleep entrainment be based on the degree of stimulation in the association network?
What if a model of sleep included a
continuum from SWS to REM sleep? This could be neuropysiologcally
oriented, SWS sleep would be the synchronous activation of neurons,
with REM sleep would be the unsynchronized (chaotic) neuron activation.
Emotion?
degree of stimulation
Summation of all aspects of stimulation, number of contours detected, degree of activation of memory, etc..
The machine could be "disconnected" from the
environment simply by multiplying the input signals by a factor, eg
0.1. only extremely high contrast would be integrated into the dream,
or wake-up the system. Seems like sound would be required, at least as
a great way of stimulating the system in a quick way, (quicker than
taking a photograph.) Or a video camera could be used, perhaps closing
the iris during sleep.
What does a machine desire? What about an artwork? Attention
Use face detection to provide a measure of how much attention the installation receives
in order to differentiate we may need a negative emotion: being ignored vs being attended to.
A face object is rewarding, but remembering (showing) any object that causes attention in the audience also gets rewarded.
Consciousness
Initially we have a genetic protoconsciousness, its purpose is to generate prototypes.
Once higher levels (sufficient number of
prototypes? representatons?) form then a greater consciousness takes
control of how prototypes are constructed.
One concrete thing consciousness could do
is change the degree a new image effects prototypes. Starting with 50%
it could decrease later on. Initiation prototypes could be random
pixels (random and gaussian blured?)
Development?
brain / sensorimotor system develops in installation from contextual stimulation