Posted: August 12, 2010 at 4:24 pm
 |
 |
This is the third diptych prototype. I’ve changed the camera mode from auto white-balance and exposure to manual. The SOM in this case is much stronger, I suppose due to the colour constancy between images. Following are some details:
Posted: August 11, 2010 at 10:52 am
 |
 |
This test is made up of 2401 images. The capture duration was about 2.5 hours, with a 500ms settle between frames. Luckily this worked out from centre, but later I realized that (a) spirals can only do square grids, and (b) that the 0 tilt is not the centre of the tilt range. The auto white-balance and exposure does not work very well in this situation. I could deal with the contrast effects to some degree, but the auto white-balance is problematic. Notice the edge effects around the cabinets. The next step is to use a light meter to do manual exposure and turn off auto white-balance.
One of the most interesting results is the out of focus images that happen in areas with little contrast. I’ve chosen a couple of details from the above images: (more…)
Posted: August 5, 2010 at 4:24 pm
This is the first post in the new SOL production category. This is because the SOL project has moved beyond the Dreaming Machine projects and I’ve decided to look at it as a different artwork.
I’ve had the idea of make SOL diptychs for some time. The diptych will be two collages made up of the same images. One will be arranged by the location of the images in space, captured systematically. The second will be arraged by the SOM. Below is an early prototype. This one is 4:3 rather than 16:9, which is because that is how the registration was done.
Images arranged by their captured location:
Images arranged by SOM trained on RGB histogram:
Its hard to say how successful it is, its only a 9×9 map, whereas SOLs are from 70×70 to 150×150 images. I’ll next capture a 50×50 map and see how long that takes, and how the resulting SOM looks.
Posted: May 26, 2010 at 12:20 pm
As I’m reading on mental imagery, and mental representation I’m torn between the Constructivist and Empiricist positions on development, as described by Muller, Sokol and Overton. On one hand their is the idea that mental representation is consciously constructed in relation to the world in an embodied fashion. The cornerstone of this position is that representation exists to serve the intentionally directed will of the agent. On the other hand there is the notion that the relationship between mental representation and the world is a causal one. The world impacts itself on the agent, who passively receives signals that are transformed into representations. (more…)
Posted: May 13, 2010 at 3:15 pm
Following is the final presentation and paper created for IAT888 (metacreation) for the perception and synthesis system of DM3. Here is a teaser of one of the automated accumulations:
Posted: March 29, 2010 at 9:22 am
It appears that all the SOM weights fed by edge-detection are very similar:
Corresponding to these resulting accumulations:
I think the poor accumulations are due to poor distance measures considering the edge-detections. The next step is to break the objects into a few Euclidean blocks, and calculate hists from those blocks. They should be much more flexible regarding object edges than the edge-detection. Basically it appears that two edge-detections of the same object in different orientations are as different as edge-detections of two different objects.
Posted: March 22, 2010 at 10:55 am
For this project the major value of the functional distinction, as proposed by Mishkin, Ungerleider, and Macko (1983), between the ventral and dorsal streams of the visual system are:
The idea was that BA40 may integrate from both ventral and dorsal streams, allowing dreams to combine contexts with objects in surprising ways. Unfortunately many studies refute the claims that the dorsal visual stream is correlated with object contexts and locations, and that it actually deals with sensorimotor control in visually guided behaviour (Goodale and Westwood, 2004).
There is evidence that a functional dissociation occurs in the processing of scenes/contexts and objects, but in the ventral stream. The hippocampus is associated with the processing of contexts and locations, while the perirhinal cortex is associated with object recognition (Murray, Bussey & Saksida, 2007). The project will continue with two subsystems, one for objects, and the other for contexts and locations. These ideas have been integrated into the working document: dreaming-machine-perception-synthesis.html.
Posted: March 21, 2010 at 10:38 pm
Since this project is meant for the complex, ever changing real-world, I’ve moved from the simple toy examples (balls and coasters) to outdoor scenes. Unfortunately things are not working very well. (more…)
Posted: March 12, 2010 at 10:12 am
In order to make a greater contribution and situate this work in what is known about the visual system is a separation of the two major streams from the primary visual cortex. The dorsal stream (occipital to parietal lobe) is considered, by one theory, the “where” region of the visual system that is associated with locations and places. The ventral stream (occipital to temporal) is the “what” region associated with particular classes of objects.
The idea is to separate the visual analysis into these two streams. Some rough ideas regarding how this could work is in this document. It is a subset of the “Dreaming Machine #3 Notes” document posted previously and contains some additional ideas for the final paper, in particular a first attempt to map the system to biological processes.
Posted: March 9, 2010 at 11:23 pm
I put a quick SOM into the current test patch to see how well it deals with this idealized object data. The SOM is a 2×2, trained using a constant learning rate of 0.5 and a constant neighbourhood size of 1. I did not keep track of the number of training iterations. As proposed the images were abstracted into a RGB histogram (768 values) and a 40×30 pixel edge-detection before being fed into the SOM.
The following images illustrate how the SOM would accumulate the images. These are not accumulated in PD, but manually, but are layered based on how the SOM would choose to accumulate them: (more…)
Posted: March 8, 2010 at 3:53 pm
Here are some rough ideas for DM3, with particular effort paid to the perception / synthesis system I am currently working on. These notes have been written in tomboy and reflect my research up to this point. All linked notes are also included, which leads to a large html file as exported by tomboy.
Posted: March 8, 2010 at 3:11 pm
Here is the simulated accumulation of the same coaster images discussed in this previous post. This time they are cropped by the single largest contour found. Note how much less emphasis the poorly registered image has.
Posted: March 3, 2010 at 10:02 am
Here is the 3 page proposal I’ve written for the meta-creation class. It describes the system in more detailed terms and discusses the whole system. Philippe’s feedback was that this perception component does not make enough contribution to the field. He suggested that I either choose a more interested method of object segmentation, perhaps a more biologically oriented one, or use a different type of SOM, like a GSOM. A GSOM is a SOM that increases the number of units depending on the quantization error (QE) calculated for each unit. A high QE means too many inputs are associated with a particular unit, and that the map is therefore likely too small.
Next steps are to look into other object segmentation methods, continue working on implementing the system, as currently proposed, in gridflow, and to look into implementing a SOM, and then a GSOM, in gridflow.
Posted: February 23, 2010 at 5:38 pm
The perception/synthesis project is a component of the next “Dreaming Machine” installation. In DM1 and 2 entire images were stored, not components of images. The purpose of this project is to determine a method for extracting components from an image, but without a complex shape recognition system. (more…)
Posted: August 11, 2009 at 1:43 am
Some shots of the installation of Dreaming Machine for PDCON09:
Posted: June 30, 2009 at 10:53 pm
After exhibiting the high-resolution SOMs I’ve become very interested in their aesthetic. I’ve come up with an idea for system that trains a map using the robot camera. The solution to the problem of the finite number of images is solved by having a temporal aspect to the installation. The camera would capture images, and the SOM would be trained. The exhibition would start with a blank slate before the SOM is trained, and over time the SOM would be refined more and more. Once the SOM is trained the exhibition would end, or perhaps remain for a time in a static state.
Something like the dual SOM would likely be needed as the training would work best if the SOM is presented with the same images and in random order. The first SOM would store the frames which the second SOM would randomly iterate over in training. Another idea would be to simply have the SOM trained slowly with the live images from the camera. Even if the camera took an image every 3s, it would take at least 70 days to train a 70×70 SOM. Then act of capturing images would be more closely connected to the training process. As each image is captured it would immediately be presented on the display.
The display could be an array of smaller screens, perhaps 4-9 inches diagonal. Each screen would show one single image at full frame. One problem with this approach is that the frames would take too much visual emphasis and distract from the structural relationships between the images. Controlling as many as 40×40 or 70×70 units would be a technical difficulty.
Another idea would be to have multiple projectors each showing a portion of the whole image, 4×4 projectors (each showing 10×10 or 15×15 units), or 8×8 projectors (each showing 5×5 or 10×10 units ), depending on resolution. All the projectors would be projected on the back of a single rear-projection surface. The strength of this method is that the Gaussian masks could still be used, multiple projectors would blend together very well, and would be more technically simple. The resulting image would be extremely high resolution and still be using live content from the camera. Even the brightness of the space would be less of an issue as the projectors would likely be close to the projection surface and therefore brighter.
Even with cheaper (non-HD) projectors an 8×8 grid could create a resolution as high as 10,240×8,192 pixels which would allow the viewer to really see the details in each individual image as well as the overall structure.
Posted: June 30, 2009 at 8:40 pm
The idea is to take a sequence, video or film. Take the first 100 (n) frames and train a SOM on those frames. That trained SOM is the first frame of the output sequence. Then take the 100 (n) frames starting at frame #2, and so on. The resulting sequence would have 100 (n) frames less than the input sequence. The result would show the structure over the last 100 (n) frames. A new scene would start a new cluster and grow in the field until it shrinks as the scene ends and is displaced by the next scene.
The number of frames used would define the number of clusters in the feild and the number of units in the frame. HD+ resolution would likely be needed as each frame would be composed of many frames from the original footage.
Posted: June 11, 2009 at 6:49 pm
Even though I had to remove the code to save images from the installation, somehow I was able to save the memory field when striking the installation. The sawtooth method for training seems to work quite well, there is certainly a cluster of darker and lighter images. Many local errors though, clear in the number of colours floating around. This may be due to there simply being not enough iterations, this was only a two week exhibition. Here is the whole 40×40 memory field:
A detail:
Posted: May 10, 2009 at 10:26 am
Here is a shot of the system being tested with the 52″ LCD TV provided by Videotage. Whereas the software showed a 3×3 portion of the memory map in Norway, this project will show a 7×7 section of the map and make use of the full 1920×1080 available pixels. Surprisingly the patch has scaled well and driving this large display is no issue, as well the DVI-I to HDMI cable worked just like a native DVI cable, the nvidia card read the EDID of the display fine. Only issue was the contrast seemed to increase when running in proper HD mode, which did not happen at the lower resolutions. For this application the contrast change was not an issue and I did not need to resolve it.
Posted: April 9, 2009 at 1:46 am
It has been a while since I posted due to being very busy working on Dreaming Machine. I spent Monday and Tuesday babysitting the system while it captured images of false-creek and the central branch of the Vancouver Public Library. I’ve yet been unable to train a SOM on these data-sets suitable for printing, as they take over 750,000 iterations. Here are some images of the setup and context of False Creek and the VPL:
