Obsessive Motivation

This approach to motivation is more subtle than the first approach. Rather than fixing camera positions in a grid, and using the histograms to choose which grid position to move to next, this method uses the difference between the middle histogram and the LRTB (left, right, top, bottom) histograms to create a vector for the next move. The more different the edges are, the larger the movement. The result has a rather obsessive quality. The camera’s gaze tends to obsess about the details of a small area, and eventually (after an indeterminate time) move onto another region to obsess about. Here is a plot of the camera’s movement. It starts in red, and ends up in green:

pt7-zoom.png

Notice the clusters where the camera explores the small details of one region. The obvious colour shift in the second from the right cluster indicates the camera spent much more time in that area than in the other clusters. Here is a detail of that area:

pt7_cluster_detail2.png

Upon closer inspection it seems clear that this cluster is actually two clusters, the second of which (in green) is much more dense. The camera spent 2571 iterations in this cluster alone, where the total iterations was only 4274, representing approximately 60%. In the next run I’ll attempt to increase the likelihood of the gaze to escape these clusters by increasing the length of the vector.

This mock-up shows the path of the camera overlayed on the visual field. The gaze is clearly attracted to areas including many edges, and tend to escape when the vector is aligned with edges in the frame:

pt7-overlay.jpg

The First Motivated Camera

I’ve made my first attempt to remove the random control of the camera’s gaze. This approach is based on an analysis of the histograms of middle, top, right, left, bottom regions of the image. The x and y regions that are most different than the middle control the pan/tilt direction. This was done so that the camera moves over a fixed grid, so that locations that have already been visited could not be revisited. Even with this mechanism the gaze of the camera is highly looped and overlapped. It also tends to get stuck in certain areas. The following plot shows 2667 iterations:

density3.png

The plot starts in the red area and ends up in the green area. The density of the camera’s fixation on certain areas is clear:

density3-density.png

The upper right corner has been visited extremely disproportionately. This is even more extreme when the range of the camera (the area in which the camera is able to look) is included in the plot:

density3-full-field.png

density3-density-full-field.png

The next steps will be to give up on this grid-based approach and calculate a vector from the differences between the various histograms in order to point the camera in a new direction. Since this vector will contain some of the complexity of the image I hope it will not be as likely to get stuck in a certain area.

Towards a self-motivated camera

Before starting to impliment the ideas of having the camera control its own path I need to be able to verify what the camera is looking at. To that end I’ have determined a mapping between unit ID and pan/tilt location, in order to vizualize the field of the camera. Following is a montage of 56×11 640×480 images arranges by thier position in pan/tilt space. Future plots of camera paths will be superimposed on this feild.

dm-reference-montage.jpg

The images on the far left edge are blurry because the camera was not given sufficient time, between captures, to move from the far right to the left right.

SOM Structure

Here is a U-Matrix representation of the memory map resulting from the Piksel installation:dm1-umatrix.png

The red Xs are units that were never associated with images. Note that due to a bug the histograms were truncated to 100 elements while being stored. This U-Matrix then only shows the similarity of the first 100 elements of the R channel histogram. A U-Matrix is normally calculated by the mean of the sum of differences between each unit and its neighbours. In this case the measure of similarity is based on the sum of the differences divided by the number of neighbours. The large amount of dark units means that this SOM was highly folded. Compare this to the U-Matrix and memory field below:

dm-test-montage.jpg

dm-sample-umatrix.png

On Machine Experience and Emotion

The following is an excerpt from a discussion with David Jhave Johnston.

david jhave johnston wrote:

one thought i had is that dreams are so personal so energetic street footage doesn’t convey the emotivity. thats a tough ledge to get around.

I responded:

How can you compare a person’s lifetime of experiences with an installations life of 10 days in a shop window? Things could/would only really get interesting with a huge network (I upgraded my machines to 4GB, and am using a different method to make the patch as scalable as possible) so we’ll see how far I can push it. Also there is the aspect of storing components of stimulus, rather than images, which could allow the construction of imaginary memories. No idea how that would work currently. I have spoken briefly to Gabora about the emotional aspect. I’m playing with (the still Piagetian idea) that emotions could all be reduced to low level emotions related to biological state.

If all knowledge is based on sensor data (as in the Piagetian project) then why, Gabora asked, would a gun associate closely with a knife? They have little to do with one and other as sense-data is concerned. My answer was that perhaps they are bound not just by their sensor impressions, but also (and perhaps more importantly) by their effect on our internal state. They associate with one and other because they cause a similar emotional state.

The problem with that line of enquiry (for my phd) is that it would require a model of those low level emotions. Interestingly the character of those emotions would just be another channel of sense data fed into he SOM. I have still not yet figured out how to deal with multiple channels of sense data, the current idea is have a SOM for each channel, which are cross linked by temporal correlation. A free association could then move through the space of similar images, and then cross into a free-association of sound where the sound matches the image. Then cross into emotion… Ideally this should be an nD SOM, with 2 dimensions for every sensor channel. I can’t get my head around this part.

A measure of stimulation would be very interesting, where the cross-link happens where the stimulus is very strong in the SOM. (A strong stimulus could be as simple as a very close similarity between the activated unit and the input stimulus)

Distribution of Memories over Time

Here is a plot of how many memories were captured at which time. The X-axis shows the dates of the installation, the Y-axis the histogram counts. The peak on the first day of the installation is due to the accelerated collection of memories during the first 50% of collected memories.

memory.png

Interesting that there are also peaks on the Friday and Saturday after the opening (on Thursday the 11th). It will be interesting to see these trends in a longer installation.

Piksel Real Code

Here is the memory field of the installation (as of this morning) after running for two days.

piksel08-1-montage.jpg

The context appears extreme in terms of the SOM’s ability to organize it. This context could use more units in the SOM. Talking to Wendy Mansilla I had the idea that it would be nice to generate these montage images in an exhibition so that they can be seen by the audience.  The idea is to generate them at full resolution and have them printed and hung in the gallery around the installation over time as the exhibition continues.

This is the path of the longest dream path so far in the piksel exhibition. It contains 45 memories. It is not the longest dream (which is a loop of 8 highly similar memories) but activates the most unique memories since starting the installation. It will be interesting to see how this will change as the installation continues.

long_dream.png

Another idea I had, when talking to Alex Norman, was a way of making the camera move without random operations. The idea is that the camera will break the image up into 5 regions. The lower, upper, left and right edges; and the centre (where the centre will be larger). For each of these regions a histogram will be taken. The histograms will then be compared using the usual method. The edge histogram that has the largest difference, in comparison to the centre area, will select the direction. Just as the dreams now follow the structure of the memories, the camera will follow the structure of the visual context.

Piksel Installation

Here are some shots of the progress of installing “Dreaming Machine #1 (prototype)”. The installation went well. The folks at Lydgalleriet were great and this will be the first time I’ll get to see the system projected! Tomorrow will be just for fine-tuning and the opening will be at 7pm.

img_0131.JPG

img_0135.JPG

Final Steps Towards Norway

I’m finally happy with the look and behaviour of Dreaming Machine. I managed to make the dream representation much more interesting and dynamic by including the each actiavted units’ nearby neighbours. The result is a much more complex representation. I’ve used the gaussian alpha channel from the montages also in this representation. One thing is clear is that I need to move to a higher resolution image when working with stills like this. Standard def video is just not good enough. This means getting a better camera (Alpha 350?) and a pan/tilt tripod head that will work with any camera.

Here is a video of a single dream

The corresponding memory feild:

Dream montage

The visualization of the dream’s path:

dream.png

More Dreams

Here is a long (134 memory) dream from the system as it currently stands:

dream1.jpg

This is the trajectory of this particular dream: (Note that the trajectory folds back on itself a few times, causing a few complex loops.)

dream1_plot.png

This is the SOM (memory field) at the time of this dream: (Note the region of tree memories in the lower left corresponding to the start of the dream.)

montage1.jpg

Dreaming / Free-Association System

I’ve started work on replacing MAM’s over-complex dynamic patching with a python (pyx) external. The CA-like aspect of MAM is really better suited to an OOP system than doing the work in PD. Here are the first of the system’s dreams.

The first is an early version where loops were very likely (where the same memories are retreived over and over again):

dream4-arrange-scale.jpg

Here is a dream generated by the current version of the system which avoids these endless loops. This particular dream is quite long (30 activations) which is due to the similarity between (dark) memories. In this version the histograms of the stored images are analysed for similarity (using the same method as the SOM) in order to propagate activations to the most similar memory. The signal is degraded proportional to the distance between memories. The more similar images are the longer the dream is, due to less signal degredation. Signal degredation is not visualized in these images.

dream5.jpg

This is the SOM used to generate these dreams. It was a test running over 3 days. Due to a glitch the camera recorded while I was not expecting that resulted in many dark images.

montage.jpg

High-Resolution Montage

I’ve written some python code (using PIL) to create a high-res montage (in the style of the Memory Association Machine visualization) from the stored images. The 30×30 SOM produces a near 10,000 by 7,000 pixel image. Here is a montage that shows the level of detail in the full-resolution version and a lower res version of the whole map.

montage-composition1.jpg

montage-scale.jpg

Tracing the Camera’s Path

ptz.png

This plot illustrates the path the camera takes to examine the visual context. The colours represent time, purple is older, yellow is newer. The zoom level is represented as the thickness of the lines. Thicker is more zoomed in. The units for PTZ are in elmo camera units. Age is specified as the iteration at that moment in time.

900 Units

Now that I have switched to the moving camera the project seems to be steadily improving. The weather was very interesting this weekend, a mix of sun and rain. The result is some beautiful subject matter for the moving camera. I increased the size of the network to 30×30. This is now too big a SOM to display using the MAM visualization method (the image below was generated with ‘montage’ from imagemagick). The image does appear organized, but is still highly complex. I wonder how big a network would be needed for a non-folded SOM of this subject matter. I have to make a u-matrix visualization to see how folded the feature maps actually are.

I’m much happier with the aesthetic now. I’ve gone with a random-walk style of camera movement to give some continuity between subsequent frames. Here is the memory map from the system running over the weekend. This image is results from constant neighbourhood and learning functions (1 and 10) and represents approximatly 8 hours of stimulus. One memory slot was not associated with an image (with a red ‘X’).

sunrain-montage.jpg

The disappointing static camera

I merged the fixes for the histogram method back into the temporal patches. After running the patch overnight it was clear that the static camera is probably not going to work out well. There is just not enough variation in the environment to keep things interesting. The additional problem of the extra overhead (of the high number of sensors) means that an image can only be captured each 2s or so. An approach to revisit would be to use reference frame subtraction to feed a presence image into the SOM. I think part of the lack of interest (and organization) in the raw pixel method is that, with a static camera, the vast majority of the pixels are the same.

I’ll move the histogram patches into trunk and start working with a moving camera. The issue with a moving camera and the dream aesthetic is the relation between one image and another. A way of approaching the smoothness of a dream could be using a slow drunk-walk random movement. That way there would be a portion of the last image in the next image and give some consistency (or at least slow change) of both the images and the histograms. I should store the sequence of the images captured so that a free association could happen both through similarity (the SOM) and through time.

The results of the training using the raw pixel method overnight. There appear to be only two clusters of images. The presence of cars and such do not cluster. (white blocks are units not associated with inputs)

static_camera_raw_pixels.jpg

Histograms

It has been some time since my last post. I was trapped by a number of technical issues that meant I was not even sure my SOM was working properly. Through the debugging process I created two variations of the current system. Both use temporal timing. That is the SOM is not triggered by the motion analysis, but runs at a fixed interval. This was done to simplify the patch as much as possible. The second variation used a histogram in place of the usual pix_dump method (of feeding the raw pixel data to the SOM).

I have learned two things. The first is that the likely cause of the  lack of organization in previous work was due to timing issues. That is the first thing I will go back to fix in other code branches. The second is that the idea of using a fixed camera may be problematic. In my experiments I was able to making a working system using the pix_histo object and passing the histogram to the SOM, rather than raw pixels. This works surprisingly well for a dynamic and moving context (that is it works best if the camera is moving, not a static camera). Of course when using the histogram the system is no longer using the raw pixel data, and therefore not making a pixel by pixel comparison. In orther words the pixel-by-pixel method is most appropriate for the static camera, where the histogram method may be more appropriate for the moving camera. It is clear that the histogram method does not work very well with a static camera.

Histogram method with static camera:

result00000.jpg

Histogram method using a moving camera:

result000001.jpg

Man Rolling Plastic

I’ve just started on the first stages of integrating some of the future work in Thesis into MAM heading towards the “Dreaming Machine” project. At this point I’m working with using frame subtraction to record changes in a scene using a fixed camera. There is no SOM used at this stage.

I’ve included a number of visualization methods based on single buffering. The first is a normal long-exposure where the images build up on top of one and other with high transparency showing just a subtle shift of the background. The method of storing all images in pix_buffers means that the visualization can be decoupled from the processing. There can be many different and simultaneous visualizations of the data now. The idea is that the system would move between them. The following images show another visualization method where the difference between the reference and current frames are stored along with each frame. These are used as alpha channels so that each time-step is less transparent:

Man Rolling Plastic Wrap

Man Rolling Plastic Wrap

I’ll be showing an early version of “Dreaming Machine” for the piksel festival in Norway this December.

Memory Association Machine:
An Account of the Realization and Interpretation of an Autonomous Responsive Site-Specific Artwork.

PDF Document

[B. D. R. Bogart. Memory Association Machine: An Account of the Realization and Interpretation of an Autonomous Responsive Site-Specific Artwork. Master’s thesis, Simon Fraser University, 2008.]

This thesis is an account of the realization and interpretation of the autonomous responsive electronic media artwork “Memory Association Machine” (MAM). Realization and interpretation are components of the creative process that braids conceptual, site-specific, electronic media art and artificial intelligence practises. The meaning of MAM is dependent on its unique location in space and time. MAM relates itself to its context using three primary processes: perception, the integration of sense data into a field of experience, and the free-association through that field. MAM perceives through a video camera, integrates using a Kohonen Self-Organizing Map, and free-associates through an implementation of Liane M. Gabora’s model of memory and creativity. These processes are as important as MAM’s physical appearance, are composed of computational elements, and allow the system to respond to context autonomously.

“Self-Other Organizing Structure #1”
Seizures, Blindness & Short-Term Memory

PDF Document

[B. D. R. Bogart. Self-Other Organizing Structure 1: Seizures, Blindness & Short-Term Memory. In Andrew Brouse, editor, Proceedings of the Second International Pure Data Conference, Montréal; August 21-26, 2007, pages 1–9. Pure Data Conference 2007, 2007.]

“Self-Other Organizing Structure #1” (SOOS1) is the first in a series of site-specific responsive installations. Rather than depending on the artist to define how these works relate to their site, the task is given to the artwork itself. The structure of the artwork changes in response to continuous stimulus from its context. Context is framed as parameters perceivable by the system that make its place in space and time unique.

Read more

Untitled Iterations

[B. D. R. Bogart. untitled iterations. http://www.ekran.org/ben/wp/2007/untitled-iterations-vagueterrain-2006, 2004.]

Musicians create music and visual artists create images. I am becoming increasingly aware of the commonalities I share with musicians rather than other visual artists. The creative process of a sound artist or an electronic musician is very similar to my own. The fundamental basis of both what I do and what a musician does is the creation of structure, or perhaps more specifically the making of a score or program that nurtures the creation of structure. My own practice emphasizes structure (in the form of language, image, sound or even a set of related ideas) regardless of my chosen medium.

Read more

Art in The Face of The Sublime

[unpublished, 2000]

One would imagine that the purpose of an Art education is to learn about Art theory and practice. Over my undergraduate education I have found that the more ideas I am exposed to the more I realize how little I really know. In my first few university years I thought of Art only as a form of expression and communication. I thought all Art had to have a statement, a concrete proposition to make about the world. At this point in time I don’t feel that I know enough to make concrete statements. Answers lead to questions and the more I learn the less sure I am about what truth means. My role as an artist and the purpose of Art itself becomes just as mysterious. It would seem that in order to be an art-maker one must have the answers to these questions. Perhaps the purpose of Art is the pursuit of these questions.

Read more