Dreaming Machine #3

Rough Ideas

• It could be interesting to have the system have a more accurate model of consciousness, meaning differing levels of consciousness between waking and dreaming. The transitions between these states could be very interesting. (See REM sleep and dreaming: towards a theory of protoconsciousness.)
• Create a developmental protoconscious state for the system. It contains a predetermined model of its role in the world (the movement of the pan/tilt, the taking of pictures, basically every feature of the system controls in the outside world (actuators, what about sensors??) maybe it also has the idea of an image. In the protoconsciousness these elements are played with in REM sleep. The mapping between sensor images and camera movements are determined by this process. Perhaps a machine learning algorithm. But without sensor data what would the machine learn from? How would this initial structure be determined? what happens if it was all based on noise, RF in the context?
• How could the machine generate 80% non remembered content? Perhaps the colour and spacial relations between pixels in small groups could be used. A green pixel next to a red pixel seen in the world could be included in a new synthesized image. How to encode these relations?
• It may be a problem for a memory system to be divorced from a survival relation to its environment (See the evolution of multiple memory systems). What is the purpose of DM's memory?
• Perception:
  • Colour is important
  • spacial structure (edge detection, FFT)
  • how to use perception backwards in synthesis? (see below)
  • Godel Escher Bach
• Stimulation as inherent reward
• dreams as self-stimulation
• Perhaps all the data can be stored in the system if we store only the abstractions and not the images. Then maybe we can use a learning technique like the SOM on all the stored data, and not have to worry about online learning? Not very neurobiological... Philippe suggests we capture a small set (3) images and do the analysis on all three.
• It would be really interesting to include genetics somehow. Like an SNN that creates a population of enrematic memories. These genetic sequences result in the various representations of the system.
• What if all the constants in the system (ie the starting point for the free-running circadian) were encoded in a genetic plan. The DM system could then be used in a genetic algorithm. What would the fitness be?

Architecture

• Perception (see Machine Perception) (no longer separate from synthesis.
  • microfeatures
  • mirrored in synthesis, using NN?
    • Can this system be reversed? Reconstructing images from features? Maybe the images don't exist in the human mind, just the features that are extracted?
  • Part of the reason why the perceptual system is reconstructive is because the visual data received from the retina is of low quality. When using a camera we have good images, these good images may lessen the difference between the original and reconstruction, or perhaps it will throw away data in the original image in an interesting way.
  • Perception could model the retinal-geniculate-striate -> PVC -> Secondary VC. and ignore the dorsal and ventral streams (that don't see well understood) See Biopsychology, Chapter 6 (Visual System)
    • iLab Neuromorphic Vision C++ Toolkit: http://ilab.usc.edu/toolkit/downloads.shtml
    • Quad tree to break image into chunks, analysis on each chunk, take ranking for regions for more analysis?
    • CERN (Markam, blueBrain)
  • FFT deconstruction of repeating patterns? Somewhat easy to reconstruct, biologically relevant?
  • Use of feedback to reinforce features
  • What about the role of attention? Just another variable that must be specified? See A neural model of selective attention and object segmentation in the visual scene:
  • While dreaming the gating of the visual field could be done (to a degree) by reducing the camera exposure to give a dark image. Then only a very bright stimulus would wake the machine, like a flashlight in the eye. This may require a video camera, or images would have to be taken continuously from the camera while the machine is asleep.
  • Saliency Map toolkit to extract
• Synthesis
  • mirrored in perception
  • consolidation: conflate the input image with the results synthesis and feed back into memory
  • all memories are reconstructions (loftus)
    • see an image, abstract image (hist +shape?), adding image to input image, then abstract again, until ?
    • attach degree of mix of new and synthesis to last stimulus based on the degree of memory recollection?
  • Combination of thousands of images to create a map of objects in context
    • only the common objects in different contexts are visible, and uncommon objects invisible, and visa versa.
    • The role of context in object recognition, see steven's email.
• Perception - Synthesis Process
  • Define ROI (background subtraction) fixed aspect ratio (Object-centric approach)
    • A non object-centric approach could be to train the map on statistical properties of the scene, and not focus on objects. This would be a kind of "gist" of an image for recognition. See The role of context in object recognition and Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention
  • A non-object centric approach would be simply to feed some abstraction of the context into the SOM, for example a hist of the whole image, or some other statistical summary of the whole image) Is this important? The objects already contain a small amount of context from the cropping, is this sufficient?
  • Which image stored has the most similar edge detection & histogram. (Should the position of object (CoM?) also be fed into the SOM?, see above object-centrism vs statistical approaches) From dream theory (Tononi), The impossible does appear in dreams, so the position of these objects in dreams need not be related to their position as captured.
  • Children do not have complex visual imagination skills, resulting in their dreams being largely static, they do not transform. (but maybe images are constructed, just not moving images.) Look into the visual system of preschoolers
  • What is the best distance measure for hist + edge images?
  • Try a k-means to see cluster data? Would require too much time to calculate all differences for each new input.
  • Next steps could be executed:
    • add new image (to a degree) to old image in prototype, record the position of the camera, and histogram (and edge?) of latest image.
    • how to deal with the categories? (RSOM, HSOM (requires storage of all patterns))
    • SNN: these differences between these images train the weights SNN
    • the dream is a stochastic pattern of activations, whose behaviour is influences by the recent waking activity
    • what is the design of the network?
    • machine keeps getting stimulated by the same pattern: dreaming being an over stimulation or under stimulation
      • if the mean activation level is too high or low then sleep.
• Memory
  • SOM? RSOM?
  • something more continuous? Incremental learning
• Attention / Sensorimotor (connect pan/tilt to received images?)
  • We have a concept of presence now. So the digital camera could be the fovea, it gets moved to the area of gross interest. The periphery could then be a second fixed wide-angle video camera. The video video camera would be low res, but get data quickly. If an object of presence appears, it could direct the foveal camera to attend to that area. Mapping X-Y to pan/tilt and size of blob to zoom.
  • The above attention model systematically ignores regions that do not change over time. The architecture never changes, and so there would be no memory of it. Or perhaps there are two memory systems, one for the background and one for the foreground. This is the case since the adaptive background is a memory of everything but moving objects. An alternate idea is to flood-fill the background (assuming high enough contrast, or detecting contrast in the centre vs the edges, and accumulate what is left over. Rather than moving objects, this model of attention focuses on objects surrounded by low contrast areas. These are likely rare in a real physical environment.
• Dream Update this with the new discussion regarding tononi
  • decoupling of sensorimotor system from 'brain'
  • change of memory recall/writing
  • circadian rhythm (based on stimulation)
  • waking level of stimulation
  • stages of sleeping (waking, NREM -> REM)
  • DM should dream more like a child than like an adult. See Dreaming and the brain: from phenomenology to neurophysiology
• Emotion?
  • degree of stimulation
    • Summation of all aspects of stimulation, number of contours detected, degree of activation of memory, etc..
    • The machine could be "disconnected" from the environment simply by multiplying the input signals by a factor, eg 0.1. only extreamly high contrast would be integrated into the dream, or wake-up the system. Seems like sound would be required, at least as a great way of stimulating the system in a quick way, (quicker than taking a photograph.) Or a video camera could be used, perhaps closing the iris during sleep.
• Development?
  • brain / sensorimotor system develops in installation from contextual stimulation

REM sleep and dreaming: towards a theory of protoconsciousness

Hobson Core Ideas

• Fill this in!

Primary consciousness (p803)

• Perception
• Emotion (must be involved), why? (William James)
  • emotion as degree of stimulation?

Secondary consciousness

• depends on language
• self-reflective awareness (awareness of awareness)
• abstract thinking
• volition
• metacognition (abstract thinking)

Dreams involve features of primary consciousness, and not so much secondary. See the following features:
http://www.nature.com/nrn/journal/v10/n11/extref/nrn2716-s1.pdf

• • Perception: vision and motion
  • bizarre cognition: plot incongruity (plot special here, or are is he talking about narrative?)
  • Fantasy: dreams often contain chimeric characters but not fantasies. (character that is two people at once)
  • Children start "adult" dreaming at age ~5.
  • Emotion: anxiety, elation and anger are common.
  • plot: loss of orientation gradually within "scene", radically between "scenes"
  • splicing: lack of continuity between scenes.
  • REM reports 7x longer than NREM. (description reports?)
  • Sensation of movement more common in REM than NREM
  • "Character recognition: Unreliable in REM but dreamer does not notice errors"
  • Thinking: highest in waking, lowest in REM. Reciprocal with hallucination across states.
  • Memory Theory source: 20% recognizable, 80% synthesis. (full "memories", microfeatures of Memory Theory?) See foot 58
    • Fosse, M. J., Fosse, R., Hobson, J. A. & Stickgold, R. J. Dreaming and episodic memory: a functional dissociation? J. Cogn. Neurosci. 15, 1–9 (2003).
  • theory of mind: dreamer can recognize mental process of other dream characters.
• How do we measure Waking, REM (emergent I), NREM (II III IV) states from brains? lower frequency = lower sleep state is higher number (ie IV)
• Type of consciousness in the pontine brainstem based on the reciprocal relation between: (nothing to do with thalamus matrix)
  • aminergic inhibitory neurons (activated during waking, inhibited during REM sleep, increasingly inhibited during NREM sleep)
  • cholinergic excitory neurons (inhibited during waking, activated during REM sleep, increasing activation during NREM sleep)
  • Q: Does this mean that directed neural activity is high in waking state and low in REM? whereas REM sleep has a high degree of undirected neural activity?
• Waking consciousness:
  • Awareness of world, bodies and selves (awareness of awareness)
• Sleeping consciousness;
  • integration of desperate data into a seamless "scenario"
  • lack of perception of its own condition (we often can't tell we're dreaming)
    • incoherence
    • limitation of thought
    • impoverishment of memory how?
  • No separation of "background and foreground processing" p803 c2
• narrative organization of subjected experience is required for "adult" dreaming during REM. p2c2
  • as you develop at some point narrative structure of sense data is possible. Steven does not buy this since children do organize into narrative.
  • Children's dreams.
• Oscillators and clocks: neural structures that hardwired to generate temporal patterns.
• in REM sleep the brain is minimally inhibited (ie activated!)
  • In waking the brain is largely inhibited. During REM this inhibition is removed, allowing greater activation.
  • fetus "autoactivation" could set the stage for waking state inhibition of the brain.
• It is assumed that conscious arises from the corticothalamic and limbic systems.
• Dreaming as a preparatory and recovery process. p805
• Dreaming as a side-effect (right word?) of REM sleep mechanisms.
• function of REM sleep and dreaming may have different identities (in human development the fetus has REM sleep, but not dreams. A fetus has a waking state)
• Possible functions of REM sleep & dreaming:
  • Sleep and energy regulation: REM sleep deprived rats lost central body temperature and lost weight p806 c2
    • Cognition only possible in a narrow range of temperatures
  • Psychological Equilibrium: Cognitive deterioration in humans deprived of REM sleep for 3-5 days
    • The more REM sleep was interrupted the more the body subject attempted to get.
    • Cognitive deterioration also happens with NREM sleep deprivation
    • neurotransmitters serotonin, noradrenaline, histamine (not a monoamine) and dopamine are significant in the regulation of sleep, mood, learning and temperature control.
  • Sleep and Learning: Learning changes REM sleep and REM sleep changes learning ability.
    • small effects
    • "The brain changes the status of its information as it sleeps" p807 c1
    • semantic memory is not strongly enhanced by sleep.
    • procedural learning is weakly enhanced by sleep
    • memory is not impaired with REM and SWS (III IV) deprivation
    • inhibition of REM sleep can increase learning
  • Dreams as auto-creative, hyperassociative, perhaps to imagine ways to organize waking experience? p807 c2

Dream Consciousness and protoconsciousness

• Development of consciousness is a gradual development built upon the "innate virtual reality generator" whose properties are defined in dreams.
  • Not a reality generator, just something something that makes sense of PGO (baseline activation?) stimulus.
• binding problem (foot 59): REM sleep enhanced sensorimotor integration. REM sleep could enrich connections through over-stimulation.
• The developing REM sleeping brain has built in predictions of space and time, eventually adjusted by the outside world.
  • space from surfaces of grey matter, time as a the time based behaviour of neurons.
• protoself:
  • forms in early developing brain REM sleep.
  • takes responsibility for automatic acts
• dream as the manifestation of automatic processes.
• volition in dreams as much an illusion as the sense of conscious will while awake. foot: 63 64
  • Steven does not buy this as stated.
  • conscious is taken as an illusion, is this the case? Is it just the state of the www.eatrawvegan.com
  • If all animals are assumed to be conscious how would things change.
• dreamless developmental REM sleep provides a virtual world, complete with imaginary self with strong emotions, but without awareness. p808 c2
• "Although our dreams appear to be agent driven, they are not volitional, nor do they contain the self-reflection, insight, judgement of abstract thought that constitute secondary consciousness." p808 c2

The neurobiology of conscious state control (AIM) p808 foot 66

• Activation
  • Brain is active during waking, inactive during NREM, and reactivated during REM.
  • Brain regulates its own activation
  • Sleepiness occurs after the destruction of the dopamine neurons in the substantia nigra.
  • reticular activating system turns gating on and off p.362
• Input-output gating
  • In REM sleep the forebrain can be activated, but disconnected from the sensorimotor system.
  • Is this on off or to a degree? What is the quality of the change of activation? (certain channels shut off one by one, or all decreasing in signal level? )
    • Steven is not sure about this, he thinks it is on or off.
    • Circadian rhythms are trained by light levels and stimulus.
  • Plastic Activation Signals (PGO) p809 c2
    • signals recorded in the pontine brainstem, the lateral geniculate body of the thalamus, and the occipital cortex.
    • PGO signals originate from within the brain and are treated like external signals.
    • How do these signals originate? Steven is not sure.
    • supports dreams as sensorimotor integration
    • "Activating the visual system from the motor side up" p809 c2
      • eye movements generate content
• Modulation
  • animgergic neurons switch on and off Memory of experience? "neural circuits to keep records" little/no memory of dreams.
    • memory changes in waking and dreaming experience.
    • frequency of recall in dreams.
  • What causes the switch from waking to REM sleep?
    • REM-off cells are active during waking and inactive during REM. and REM-on cells which do the opposite.
    • During a waking state acetylcholine, serotonin, noradrenaline, histamine and dopamine are activated
    • During NREM these neurotransmitters are all activated to a lesser degree, but not shut off.
    • During REM acetylcholine and dopamine are activated, serotonin, noradrenaline and histamine are shut off. foot 104, 105
    • Circadian rhythms and stimulation
• AIM Model
  • What causes the A, I and M changes?
  • What are other effects of the lack of waking transmitters in REM sleep?

Imaging, Lesion and EEG Studies

• Limbic system and temporal lobes could be important in primary consciousness. p810 c2
• synchronization leading to temporal binding essential to waking consciousness? (foot 59)

The Evolution of Multiple Memory Systems

Sherry

• Memory is a function that allows us to acquire, retain and retrieve information. p1
• functions based on Memory:
  • Recognize the familiar
  • predict events
  • return to particular places
  • assess the consequences of behaviour
• Perhaps one system is not sufficient for all functions of Memory (functional incompatibility) p1-2
• What is a memory system?
  • Interaction between acquisition, retention and retrieval characterized by rules of operation. p2 c1
  • Different systems us fundamentally different rules of operation.
  • Memory systems can be strong or weak.
    • strong: The system is independent of other systems. (multiple systems do not share any processes)
    • weak: The system is composed of processes that are shard with other systems.
  • The same information may be fed into different Memory systems, and different information may be fed into the same memory system.
  • Different kinds of information is handled by different processes, but its unclear if these processes need independent Memory systems.
  • Multiple Memory systems occur where these different information processes each uses its own Memory system with its own rules of operation that do not overlap with Memory systems corresponding to other information processes.
• An evolutionary view of Memory systems p3
  • Memory should evolve to match the requirements of the particular environment.
• Heritable Variation in Memory: p3
  • Memory traits are known to be genetic. (The the gene's effect of nervous system development)
  • Changes of a single gene in fruitfly's change their ability to learn dramatically.
• Memory and reproductive success. p3 c2
  • Obviously memory (identification of individuals or locations) can contribute to reproductive success. Some examples include;
    • Avoiding predation (recognizing predators and mobbing) p4 c1
    • Animal contests (remembering previous won and lost interactions saved repeating the risk of contest)
    • Cooperation and altruism (recognition of genetic lineage, or social grouping)
    • Foraging (recognition of food abundant locations)
• Adaptive Specialization p4 c2
  • Memory system adapts to the survival requirements of environment.
  • How to rats associate the taste of food eaten hours ago to an illness felt currently?
  • General Purpose Learning Theory
  • Different memory systems for different survival problems. p5 c1
• Functional Incompatibility
  • Compound eyes are fine for pattern recognition, but slow. Ocelli have poor pattern recognition and resolution, but are fast. (fast = less to process)
• Multiple Memory Systems in Animals p5 c2
  • Song Learning
    • Songs are used to attract mates and defend territory.
    • Songs may have local dialects and individuals may have personalized repertoire.
    • There may be a different memory system for remembering songs to be sung, vs songs to be recognized.
    • Restrictions on what is learned:
      • Some kinds of data (songs) may not be learnable.
      • The ideal learned data structure (song) is called a template.
    • Time of learning p6 c1: Some birds will only learn during a certain period of their development.
    • Memory and song production: Some birds only sing learned songs months after hearing them.
    • Neural basis of song learning: in some birds brain structure increases in volume with the increase of song repertoire.
    • Songs not sung: Birds may continue to learn (recognize) neighbour's songs long after they stop learning new repertoire.
  • Imprinting and Orientation p6 c2
    • Imprinting: Attachment of young to parent birds and effects attraction to potential mates.
      • Restricted period of time where imprinting occurs, long delay between imprinting and its effects on mating.
      • Imprinting is not the only method of recognizing other individuals.
    • Stellar Orientation: Migratory birds learn the stellar configuration to help with navigation.
      • These patterns are learned independent of their rotation about the north star, but not independent of axis of rotation.
    • Olfactory Orientation: Fish return to their spawning stream by unique olfactory cues.
      • As little as four hours of exposure to the olfactory signature is required for the fish to return years later.
      • This olfactory sensitivity only manifests during spawning.
• Functional Incompatibility: song learning and food caching p7
  • The song-learning memory system does not function well for the task of food caching.
• Multiple memory systems in humans and primates p8 c1
  • Possible memory systems:
    • procedural (I) vs declarative (II)
    • implicit (I) vs explicit (II)
    • semantic vs episodic
    • semantic vs cognitive
    • habit vs memory
    • dispositional (I) vs representational (II)
    • taxon vs locale
    • early vs late
  • Memory systems I and II
    • Gradual, incremental learning (I) vs single-trail memory construction (II)
    • Amnesia patients tend to have problems with system II and no impairment with system I memory
    • Non-matching sample task
      • recognition of object seen once, after some delay.
  • Functional incompatibility between systems I and II
    • System I is about feature extraction (preserving invariance across episodes) p10 c1 (Remember what is the same?)
    • System II about about representations (preserving variance) (Remember what is different?)
    • "primary memory" (memory of up to seconds) may have its own memory system.
    • language may have its own memory system. (What is the difference between a symbol and stimulus?)
• Exaptation and generality in memory systems. p10 c2 (Gould, spandrel: left overs from adapted traits)
  • Selected traits may have many cause many different effects in the environment (fox's feet adapted to make paths in snow)
  • exaptation, specialized solutions may allow solutions to more general problems.

Machine Perception

• Multiple channel abstraction -> Reduction of dimensionality -> Classification MRC
  • Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention.
  • Action Recognition Using a Bio-Inspired Feedforward Spiking Network
• Biological model of attention and basic abstraction:
  • A neural model of selective attention and object segmentation in the visual scene:
• Significance of Context:
  • The role of context in object recognition

Biopsychology

• Chapter 3 ()
• Chapter 5 (Methods)
  • Brain Imaging 5.1 p102
    • Contrast X-Ray: Inject X-ray absorbing material into area where contrast is needed.
    • CT (Computed Tomography): Sections produced by combining map individual Xray images. shows 3D structure
    • MRI: Images the resonance of hydrogen atoms (no X-rays!) Combined into computer sections, or 3D structure
    • PET: (Positron Emission Tomography) imaging of cells who consume an injected radioactive dye. Shows cell activity due to cell consumption.
    • fMRI: Imaging the oxygenation of blood in the brain, showing brain activity where cells require increased blood flow.
    • EEG frequency (change of voltage in head) reflects states of consciousness (Waking, REM, NREM)
      • Q: does voltage correlate with fMRI or PET data on neuron activity?
        • Correlation was based on post-synaptic activity, not APs
• Chapter 4 (Neuron Function)
• Chapter 6 (Visual System)
• Chapter 7 (Mechanisms of Perception)
  • Perceptual system is hierarchical and functionally separated. Signals are not only abstracted, but abstractions can effect sensory neurons.
    Engel, Fries & Singer, 2001; Gao & Suga, 2000; Sillito, Cudeiro & Jones, 2006)
  • Interactions between sensory systems happen at many levels of hierarchy (not just high association levels)
  • The Auditory System 7.2 (page 164) Skimmed
  • The Somatosensory System 7.3 (page 171) Skimmed
    • Three Paradox's of Pain:
      • Adaptiveness: survival requirement
      • Lack of Cortical Representation: Pain seems to exist at a lower level and its perception is not tied a particular cortical area.
      • Decending Pain Control: Can be suppressed by cognitive and emotional factors.
    • Q: How is a pain stimulus different than any other free-nerve sensor?
      • pain is a different pathway into the mind
      • phantom limb pain means that the perception is not the type of nerve.
  • The Chemical Senses 7.4 (page 179) Skipped
  • Selective Attention 7.5 (page 183)
    • Endogenous Attention (Cognitive, top-down)
    • Exogenous Attention (External, bottom-up)
    • Neural Mechanisms p184
      • Neurons in the area of intention are excited with the non-attended neurons are inhibited?
      • The neurons are tuned to focus on a part of an image by a change in their receptive fields. This means that the pathways that feed the higher level functions are manipulated to alter attention.
      • Q: What process is the cause of these possible changes? Appears to relate to consciousness.
        • neural pathways.
• Chapter 14 (Sleep, Dreaming, and Circadian Rhythms)

Chapter 6 (Visual System)

• The "inherent creativity in your own visual system" p130
• The Eye (6.1) p131 skipped
• The Retina (6.2) p134 skimmed
• From Retina to Primary Visual Cortex (6.3) p140
  • retina-geniculate-striate pathways
  • Left and right eyes are interlaced and distributed to PVC in both hemispheres.
  • Retinotopic Organization
    • The retina is topologically mapped to the retina-geniculate-striate: Signals at nearby retinal cells activate nearby neurons.
  • The retina-geniculate-striate is composed of two channels
    • parvocellular layers (P): small cell bodies
      • Sensitive to colour, details and stationary and slow objects
      • more cone connections
    • magnocellular layers (M): large cell bodies
      • Sensitive to movement.
      • More rod connections
    • These channels project to slightly different locations.
  • Edge detection
    • Contrast enhancement increases our ability to see edges.
    • In Horseshoe Crab: (Ratliff, 1972)
      • The rate of a neuron firing is proportional to the intensity of the light.
      • Lateral neighbours are inhibited during this firing.
      • Make a computer model of this for experimentation
      • This actually does sound similar to the effects of an edge-detection convolution. The size of the kernel is the degree of lateral inhibition, summing values over space to produce the result of a single pixel.
      • Q: The structure of the lateral network is needed to understand this.
      • Q: Does this happen at the retina, or later in the system?
        • Will get more research about it.
        • Moo-Wing Poo
  • Receptive Fields
    • The field effects of retinal neurons do not change from retina to PVC. This implied little lateral connection, and straight connections (Hubel and Wiesel, 1979, 2004)
    • Receptive fields are smaller in foveal neurons
    • Receptive fields are circular
    • Receptive fields are monocular
    • Many neurons have some lateral connection. Concentric circles with excitatory area surrounded by an inhibitory boundary.
    • On and Off cells are more activated when the centre is activated, but not the margin, or the opposite. Q: So they respond best to contrast, but contrast at what resolution? enabled by this, simple small lateral inhibitory connections? What is the distribution of on and off cells?
    • Q: When we dream of an image, is that image imprinted on the PVC?
      • Steven says maybe, but he is not sure, this is not a fact.
  • Simple Cortical Cells
    • These cells have similar on-off effects as lower striate cells, except they are organized into lines (rectangles) rather than circles.
    • These cells are then sensitive to contrasting lines at particular orientations.
    • Q: how does this work at the NN level?
    • Q: how are these lines organized?
    • Q: How many cells make up a line?
      • Need a model to explore
  • Complex Cortical Cells
    • These are the cells that do the higher level image abstraction. There are more of these than the simple cells.
    • They have much of the same features of simple cells: rectangular receptive fields, sensitive to contrasting lines at certain orientations.
    • Receptive fields are larger than in simple cells.
    • Cells respond to lines at particular orientations independently of their position in the field.
    • Binocular
  • Q: Any diagrams of how this works, the description is too high-level.
  • Columnar Organization of the PVC (Hubel, Wiesel & Stryker, 1977; Martinez & Alonso, 2003)
    • The signals from simple striated cells are combined at a higher level at the complex cells.
• Seeing Colour 6.5 p149 (Skipped)
  • Retinex Theory
• Vision and Conscious Awareness 6.6 p153
  • PVC -> Secondary VC -> Visual Association
  • As we move up this abstraction hierarchy receptive fields increase in size and respond to more and more specific stimulus.
  • blindsight: The possibility for a person to respond to visual stimulus that they are not consciously aware of.
  • Visual system of the Macaque is well mapped (Grill-Spector & Mallach, 2004)
  • Once visual data reaches the PVC it passes to the higher level areas via two streams: (p156)
    • Dorsal: Spacial stimuli through the location and motion of objects.
      • Where (Ungerleider and Mishkin 1982)
      • Behavioural action with objects "Control of Behaviour" (Goodale, 2004; Goodale & Milner, 1992; Logothetis & Sheinberg, 1996)
    • Ventral: Objects characteristics through colour and shape.
      • evidence that groups of neurons in this area respond to classes of objects (faces, letters, animals etc..)
        (Haxby, 2006; Reddy & Kanwisher, 2006)
      • What (Ungerleider and Mishkin 1982)
      • "Conscious perception" of objects (Goodale, 2004; Goodale & Milner, 1992; Logothetis & Sheinberg, 1996)

A neural model of selective attention and object segmentation in the visual scene:

An approach based on partial synchronization and star-like architecture of connections Roman Borisyuka, Yakov Kazanovichb, David Chika, Vadim Tikhanoffa, Angelo Cangelosia Core Ideas

• Attention model based on a spiking NN which is biologically inspired.
• It is not clear how attention is controlled.
• Realization that edge detection is a multi-scale problem and not easily implemented using an NN (hence the use of traditional methods here)

Introduction:

• • Object segmentation requires context, previous experience, and agent goals (attention?)
  • Attention correlated with synchronicity of spikes:
    • Fell, Fernandez, Klaver, Elger, & Fries, 2003;
    • Fries, Schroeder, Roelfsema, Singer, & Engel, 2002;
    • Singer, 1999;
    • Steinmetz et al., 2000
    • Temporal Correlation Hypothesis (TCH): Gray, 1999; Malsburg van der, 1981; Malsburg van der, 2001
    • Synchrony of spikes for reinforcement, Review: Ritz and Sejnowski (1997) and Wang (2005)

Module A (Selective Attention) Section 2

• • Hodgkin–Huxley neuron model: Chik et al. (2009).
  • Attention is the synchrony between the central and peripheral neurons that correspond to areas of the image. inspired by the "Central Executive" concept.
  • Central neuron C2 inhibits connections from areas of the image not attended to. The driving oscillations are endogenous.
    • C2 receives external input from "areas within the brain that are not directly involved in the functioning of the attention system". p709-710
    • How does this inhibition work? by various degrees of connectivity between C2 and periphery neurons.
  • Peripheral neurons are frequency coded so their firing rates are coded for their various types of stimulus (R/G,B/Y).
    • Does this mean that there is a pair of low level neurons, one encoding R/G and the other B/Y? what about brightness? No the colour (in this system) is encoded as frequency, so red has a low frequency, blue a high frequency. The brightness is the amplitude of the signal, and the frequency is the colour. Or is the colour amplitude coded? (explaining why blue is attended to first)
  • Discussion of plasticity on page 710 is unclear. Is this the varying degrees of inhibition from C2 to Pn?
  • fastest peripheral neurons then get attention first. (useful for reactive generative attention) p710
    • The higher the frequency of the inputs, the more likely those inputs would be attended to. Does this correlate with the degree of stimulation? A more stimulating entity is more likely to elicit attention?
    • How do higher frequencies in Pn result in more inhibition of lower frequencies? Is it that the lower frequencies are simply inhibited by the higher frequencies directly or by C1?
  • The background is hard-coded to be ignored by the system (giving little stimulation) This seems unlikely.
  • How is this scalable to multiple abstractions of data? (beyond just colour?) or does attention happen only at this low level which feed the higher level processes? This does not sound very parallel.
  • Not quite clear how the scanning process works. How does C2 scan through regions, what is its input in the test case? p711
  • Local excitatory connections between PNs are needed for select whole colour regions. But does this only work with a background that causes no excitation in the corresponding PNs?
  • Local inhibitory connections are required to tell the objects apart. (eg pear vs cloth) So we have both local inhibition and excitation?
  • How could this model of attention be applied to camera movement?
  • Example case (Four colour balls) p712
    • Use of 24 local excitatory connections. (8 neighbours, and all of their neighbours) No inhibition

Module B (Contour Extraction) Section 3

• • Edge Detection:
    • What is a spacial derivative? Sounds something like a linear interpolation between pixels.
    • What is a second derivative?
    • Page 713 has a mathematical method for getting an edge: Test if pixel P is on an edge:
      • F is a function that defines the image itself?
      • The absolute difference between pixels in the neighbourhood of p is greater than a threshold. (eq 5)
      • second derivative should change sign at p. (what does that mean?)
      • third derivative should be negative and below a threshold.
      • The detection of an edge vary greatly depending on scale, calculations are then often done at multiple scales.
      • This method is very complex, requires smoothing, and does not seem to have a biological basis?
  • Gabor filters with derivatives along the gradient direction for different scales: (just like in other models)
    • Broussard et al., 1999; Huang, Jiao, & Jia, 2008; Lindeberg, 1998; Petkov & Subramanian, 2007; Sumengen & Manjunath, 2005
    • There is no mention of this in the section!! Maybe its used in Module C?

Module C (Object Segmentation) Section 4 p713

• • oscillatory network used to find edge boundaries, suppress noise and segment. (4.1)
  • This network uses data from Module B (the computed contours) not Module A.
  • Contours are used to constrain activation to contours, that is because the background is not constant?

The role of context in object recognition

Oliva & Torralba, 2007

• Introduction
  • Background elements provide useful information in object recognition.
• Contextual Influences on object recognition
  • Objects in a familiar background are detected more accurately and quickly than objects in inconsistent scenes.
  • "Average images aligned on a single object can reveal additional regions beyond the boundaries of the object that have a meaningful structure." p520
• The effects of context
  • Components of context meaningful in object recognition:
    • semantics, spacial organization, and pose (orientation)
  • Context most significant in "glance" (<200ms exposure) object recognition.
  • Framework of contextual influences: refs 11, 18-20
  • The presentation of a context scene primes the mind-brain to consider subsequent objects in that context. The recognition of loaf of bread is faster when presented after an image of a kitchen counter, and slower when presented after a bass drum.
  • This can also change the recognition of the background suggesting a "mutual" influence between contexts and objects.
• Implicit learning of contextual cues
  • Human subjects have been shown to learn contextual relations between arbitrarily associated objects.
  • The transfer of contextual cues is not impaired by stretching (ref 28) but significantly impaired by changes in viewpoint.
  • factors effecting speed of recognition:
    • co-occurrence (two objects in one scene) may happen at multiple levels:
      • Kitchen predicts stove (macro)
      • Nightstand predicts alarm clock (local)
    • Associations can be definite or probabilistic
    • Memory search vs Visual search
    • Observers may act on an object in a consistent manner, or not. Q: What does this mean?
• Perception of sets and summary statistics
  • If we consider object-object relations as context then we assume the atoms of perception are objects.
  • An alternative is the statistical properties (ie histogram?) refs 40-44
    • mean size and variance of objects
    • CoM
    • textural descriptors
    • clutter
    • depth / perspective
  • Could these be useful for DM? Is a combination of statistical and object-centric approaches make sense for this project?
  • Ref 48 discusses computer vision based on the statistical approach.
• Global context: insights from computer vision
  • Sliding/scaling window around image and determine if the target is in one of these windows. (ref 49)
  • A major problem is compact representations of context suitable for computation.
  • One method is the use of "aggregated statistics of low-level features" refs 11,18,50.
  • These are "state-of-the-art" features of scene and context-based object recognition.
  • scene recognition: res 50,52-56
• Contextual effects on eye movements
  • Global scene directs the eye to target objects. Is this relevant when the movement is not purposeful (as in DM) or until DM becomes purposeful.
  • Saliency maps are good to predict the chance of a viewer's fixation.

Rapid Biologically-Inspired Scene Classification Using Features Shared with Visual Attention

Christian Siagian,

• Developed for Robotics
• Types of machine vision for scene recognition:
  • Object based: segregation, grouping, object recognition
    • Which objects are meaningful?
    • SIFT: Scale Invariant Feature Transform: Partial recognition, can be better than strict object recognition.
  • Region based: uses segmented image regions rather than landmarks (objects)
    • Similar problems (segmentation) as in object recognition
  • Context based: Use entire image, no segmentation into smaller regions.
    • Features are extracted from the whole image to capture "statistics and/or semantics"
    • Noise tends to be average out over the whole image.
    • Methods:
      • Texture descriptors and hist (Renniger & Malik, ref 14)
      • Colour hists w/ matching (Ulrich & Nourbakhsh, ref 15)
      • 2D FFT of grid components and then PCA (Oliva & Torralba, ref 16) (more recent work uses hierarchical wavelets, Torralba, ref 17)
  • Biologically Plausible:
    • See refs 18-21
    • Humans can get the gist of a scene very quickly and accurately.
    • Gist in humans seems related to spectral components and colour diagnosticity
    • Gist in this context is a vector in feature space
    • Gist and Saliency may be complimentary.
    • "Feature extraction mechanisms, V1 to V4 + MT are shared by attention and gist" (p3, c2)
      • Assuming there are no paralell processes providing different signals in the striate.
      • no definition of "attention" at this point: Attention = Saliency here? p3 c2 pa3
      • So is it plausible if this is the measure of plausibility?
    • Saliency: Using model of striate -> association cortex through the dorsal (where) stream. Ref 35
      • Saliency through spacial competition between low level feature responses over the whole field. p3 c2
      • Emphasizing differences (inhibiting similarly firing neighbours, ie lateral inhibition??)
      • well studied: refs 20, 21, 36, 37, 38 (in bold are the models)
    • Gist: Ventral (what) stream:
      • gist of the whole scene
      • Emphasizing sameness? (no lateral inhibition? Temporal inhibition rather than lateral?)
• Design
  • Visual Cortex Feature Extraction (A)
    • Saliency: (ref 20)
      • Convert input image into 8 (or 9?) versions, all at different scales (using linear interpolation)
      • 5x5 Gaussian smoothing of these outputs:
      • centre on and centre off operations on each output (adds lighting invariability) (what is the algo to do this?)
      • scale difference operator on scales 2,5; 2,6; 3,6; 3,7; 4,7; 4,8 (why this pattern?)
        • interpolation of finer scale with pointwise absolute difference
        • linear interpolate the coarser scale to the same size as the finer scale and calculate the pixel by pixel differences?
        • What does this give you? "gather information in regions at different scales", then why the conflation of the scales?
    • No explanation of how the orientation, colour and luminosity channels are derived.
    • Orientation: Gabor filters at multiple angles and orientations?
    • is the centre-on/off algo the course/fine divider?
    • Gist:
      • break image into a 4x4 grid and figure out the mean value for each region. Resulting in a 16 item signature
      • How to integrate the multiple gist channels?
      • Colour constancy through training?
      • PCA used to reduce the 544 dimentions to be fed into a MLP trained w/ backprop used as classifier (offline)
    • Skipped to conclusion

Dream Theory

Philosophical

• husserl?

Artistic / Literary

• Do androids dream of electric sheep
• does this really make any sense to include?

Psycological

• Freud

Cognitive / Neurological

• Hobson: REM sleep and dreaming: towards a theory of protoconsciousness
• Tononi: Dreaming and the brain: from phenomenology to neurophysiology
• Domhoff: The Scientific Study of Dreams: Neural Networks, Cognitive Development, and Content Analysis
• Foullkes: Dreaming: A Cognitive-Psychological Analysis

Computational

• Memory as garbage collector

Dreaming Machine #3 Memory Theory Machine Consciousness Machine Emotion Machine Perception

Dreaming and the brain: from phenomenology to neurophysiology

Yuval Nir & Giulio Tononi, 2009

• Contemporary Dream Research
  • Dreams occur in both REM and NREM sleep (in all four stages?) Refs 59,66,67
• Phenomenology of dreams and their relation to brain activity.
  • Consciousness can nearly vanish during SWS, despite persistent neural activity.
  • Dreams as "vivid sensorimotor hallucinatory experiences that follow a narrative structure" refs 3,11
    • Q: Has anyone done a deep comparison between narrative theory and what these cogsci people throw around to describe a dream?
  • REM sleep dreams are typical (narratives), implying that the non-typical NREM dreams are non-narrative?
  • PGO waves only occur just before, and during REM sleep? So much for PGO waves causing dreams, since NREM dreams are possible
• Similarities between (REM) dreaming and waking
  • Common features of dreaming and waking:
    • Rich sensory modalities as present in waking.
    • More visual and sonically oriented (More oriented toward their sensory experience of the real world.)
    • Experiences rather than abstractions (like concepts)
    • EEG: REM and quiet waking have very similar features. (refs 15-20)
    • PET: REM and quiet waking are comparable (refs 11, 20)
    • Activation of visual cortex is about the same in REM and quiet waking. (refs, 16,17,19)
    • Dreams match the cognitive ability of the dreamer (eg mental deficit, children, refs 13, 14, 21, 22, 23)
      • This is significant for DM, because it certainly has a different cognition than a human adult.
    • Personality and interests carry from waking to dreams. (refs 12-14)
  • Differences between waking and dreaming
    • Reduced voluntary control and volition in dreams.
    • Reduced self-awareness and altered reflective thought in dreams: Single-mindedness (ref 28), contradictory beliefs, acceptance of the impossible (ref 29)
    • Q: If the DM does not have some of these brain regions that are deactivated during dreaming, does that conflate its dreaming and waking states by minimizing the differences?
      • Does it matter if the states of so different?
    • Increases emotionality in dreams: (eg joy, surprise, anger, fear and anxiety, refs 34-36) sadness, guilt, and depression are rare. (ref 11)
      • These could be useful to map these emotions to (perhaps) the level of stimulation, but I suppose the difference between the forms and later are unlikely to be separable based on level of stimulation/activation.
      • 30% of REM dreams have no emotional content.
    • Altered mnemonic processes: REM dreams are often lost, what about NREM dreams?
      • After waking the memory of dreams fades rapidly.
      • Residue of waking consciousness is integrated into 50% of dreams, but always out of context of the original memory. (refs 39-41)
      • "Finally, many have the impression that the network of associations stored in one’s memory might become looser than in wake [44,45], perhaps favoring creativity, divergent thinking and problem resolution [4,46]." p92
    • Dreams have been considered madness and psychosis, but may be more similar to delirium.
    • Dreams in relation to "daydreams" (ref 51)
  • What determines the level of consciousness in sleep?
    • Consciousness may be higher in REM sleep (refs 56, 57)
    • There does appear to be some level of consciousness in NREM sleep. (refs 58,59)
    • hypnagogic hallucinations are short periods, and in some cases snapshots or images. (ref 60, 11, 47)
      • What would have to change in a network to switch from NREM and REM dream characteristics? More varied activation? Perhaps the low-frequency NREM sleep activates fewer memories more strongly, rather than the high-frequency low voltage REM sleep, where many memories are being ativated, but to a lesser extent?
        • Make this part of the network design.
    • Dream reports in SWS sleep are rare, and tend to have very different qualities:
      • short, thought-like, less vivid, less visual, more conceptual, less animated, under greater control, more plausible, more concerned with current issues, less emotional and less pleasant (last two contradictory?) (refs 9, 11, 63)
      • this seems suspect to me. Steven this this could be relevant.
    • 10-30% of NREM dreams are indistinguishable from REM dreams. (in terms of content, ref 64, 65)
    • lesion studies have shown that you can have dreams with REM, and REM without dreams.
    • Different parts of the brain may be in different states, eg thalamocingulate showing a waking level of activity, with the cerebral cortex being in NREM, during sleepwalking. (ref 68)
    • Level of consciousness as the ability of the brain to integrate information (ref 79)
  • Why is the dreamer disconnected from the environment?
    • Disconnection is not absolute, strong stimulus can get through.
    • Dissociation of V1 of the PVC from high level VC during REM (ref 18)
    • Q: what is the "default-mode" network? The brain regions active regardless of input.
      • Default modes are suspect.
    • "self-related introspective processes rather than by general mind wandering" (refs 31,95,96)
  • Are dreams more like perception or imagination?
    • Perception: Making sense of the sensory data through interpretation and synthesis. (feed forward) (Hobson)
    • Imagination: Abstract thoughts enriched with perceptual and sensory experience. (feed back) (Freud, Tononi)
    • If dreams are PGO waves, then they are more like perception. Where aminergic changes emphasize feed-forward and deemphasize feedback.
    • Area correlated with brain imagery (imagination) (ref 128)
      • Broadman's area 40, temporo-parital,occiptical junction.
        • without this, no dreams, no visual imagination.
    • Evidence shows that dreams are more like imagination (Box 3)
    • lesions in the VC that effect waking perception also occur in dreams. That is if your faceblind, then you don't dream of faces.
    • In children dream recall develops at the same time as the ability to build images in the mind (box 2)
      • Should this mechanism be explored in the perception/synthesis of DM?
      • Children 5-7 tend not to have narrative, but event based dreams.
      • young children cannot imagine continuous visual transformation (only movement?) (ref 118, 21) Unfortunate aesthetically.
    • Human dreams as narrative: (ref 125)
    • Visual imagination does not have fine-grained details, as reported by lucid dreamers (ref 111)