Self-Organizing Map Using Areas of Movement as to Determine Degree of Reorganization.

The image above was already posted and included here for comparison with current work in progress; before I get into the details, I’ll situate this project in relation to the previous piece. In “As our gaze peers off into the distance, imagination takes over reality…”, the panoramic photograph gets increasingly reorganized from a ‘horizon’ to the top:

What is interesting about it is the interplay between the different degrees of reorganization that gives the colour regions ephemeral smoke-like shapes. Since then I had in mind different ways of manifesting degrees of reorganization (a continuity between sensation and imagination, realism and abstraction, etc.); at the start of the MPCAS project I considered using a depth camera to provide the degree of reorganization where closer objects would be less re-organized than distant objects (a direct manifestation of what the horizon was a proxy for). Since the view from the MPCAS camera does not have a clear place for a horizon it was unclear what should determine the differences in reorganization. Using the areas of movement is interesting because it’s consistent with the emphasis on flow I was interested in with a moving image. So I wrote some code to use the following image to determine the degree of reorganization and have been making a lot of explorations.

The roadways and trees have the most movement with some movement on the sidewalk as well. The street signs, utility poles, mountains and architecture are static. In my first explorations, below, I could not see much effect of the movement image. The degree of reorganization seemed quite consistent. There are some hints of effect though; note the details around the pink area amongst the trees on the right. This is an area where there is lots of movement (on Broadway) visible through the trees.

I thought perhaps the subtlety was due to the parameters and tried a few more variations, as follows. While the trace of the movement image is still visible in small details, the images are too evenly organized overall.

At this point I was starting to think there was a scaling issue and that the movement information was not being interpreted correctly. So I did a test using one of the foreground masks such that the foreground objects in the frame are not reorganized, but background objects are. Sure enough the code is working properly, see image blow. Some background objects are not reorganized because it was a quick test with few training iterations.

Since I was using code from “As our gaze peers off into the distance, imagination takes over reality…” I realized that I had used an exponential function where the degree of reorganization accelerates from the horizon to the top. So I went through another set of variations where the movement information is squared or cubed, see following images. The effect of the movement information is much clearer now and you can see traces of the architecture, street signs etc. in the degree of reorganization.

I’m feeling like these are on the right track, but I was still imagining the road areas would be much less organized. The top right is actually under-trained (8000 iterations) so the detail in the road area is due to those pixels not being changed, not due to the effect of the movement areas. It seems a relatively small number of iterations (50000) works quite well and also a quite large maximum degree of reorganization. The top left is actually way too reorganized with both too many training iterations (300,000) and too much reorganization. The bottom two images are the most successful and I also did a test with a fewer number of iterations and a greater degree of maximal reorganization, see following image. It looks like it has slightly too few training iterations, but is the strongest result so far.

I’m really struggling to imagine how these change over time. There are some precedents in other pieces I’ve made with SOMs learning from images over time, but the combination of the variable degree or reorganization with the frame-rate is novel. For reference, consider the following videos from a video exploration as part of “As our gaze peers off into the distance, imagination takes over reality…” (top) and “Through the haze of a machine’s mind we may glimpse our collective imaginations (Blade Runner)” (2017) (bottom).

I just started a test running these same params for a set of 35 images spanning from night time to early morning to get a sense of how they look in differing lighting conditions. After that I’ll see about changing the params slightly and then see about how to work with change over time. Each of these images takes about 45mins to calculate, so the whole 24h sequence at 10fps would take 70 years! The idea was to optimize since most of the frame is static over time; I’d train a base image that may take 45 mins but then only slowly refine that image frame by frame with an emphasis on the moving objects. The details of this process will have a huge effect on how the image evolves over time and also the aesthetic of each frame.