As previously discussed, I found there was too much variation in the process that is not manifest in the final still image. I’ve thus made a video loop that shows the gradual deconstruction of the photographic image.
As our gaze peers off into the distance, imagination takes over reality…
The relation between the world as we see it and the world as we understand it to be is all wrapped up in abstraction, the act of emphasizing some details in order to dispense with others. My artistic interest in abstraction and computation is an interest in the world as it is versus the world as we read into and conceptualize it. As our gaze peers off into the distance, imagination takes over reality… is site-specific public artwork that uses a photographic image of the surrounding site as raw material. The bottom of the image shows this imperfect photographic illusion, a proxy of objective reality, that becomes increasingly abstracted towards the top. The abstraction is a proxy for the constructive powers of imagination and is the result of a machine learning process that rearranges colour values according to their similarity. The imaginary structure presented in this image is both an emergent composition and an exposition of the implicit structure of reality. The image is a representation of place (as a physical location at the site of installation) and non-place (as an ambiguous and emerging structure under constant renewal). The viewer’s experience of the tension between ambiguity and reality mirrors the mind’s constant attempt to bridge experience and truth.
Above is the final design for the QE Plaza. I’ve removed most of the white padding to preserve the aspect ratio of the image to match that required by the installation location. I’ve included details of the full resolution version below. I’m quite happy with artifacts due to sub-sampling used to progressively smooth the transition between the full resolution pano and the 20px per cell SOM. This leads to a quite nice tilt-shift effect that contributes to the tension between reality and imagination.
The 10px cell test with a small number of iterations (1,000,000) is shown below. While the spires are reduced they are still present, especially at full resolution. In the scaled image below, the most noticeable one is to the left of the leftmost set of red benches.
I also tired using the 7px cell test (below) with very few iterations (top: 500,000 and bottom: 250,000) and larger max neighbourhood (30²) to reduce spires; unfortunately, the lack of training shows a lack of organization compared to the results documented in this post. So I’m going to stick with the previous 20px 5,000,000 training iteration version and spend some additional time working on the transition.
After some time and consideration, I’ve decided although the spires fit with the theme of imagination and emergence they are just too visually dominant. I’ve thus been exploring using the 20px cell size (that avoids spires) with a lower horizon:
I’m still tinkering with the code and the central issue is that I’d like to get better coverage of the area near the horizon, which seems to mean more than 5,000,000 iterations of training, and thus an increase of cell size. The following images show some of these explorations. I’ve also used an even more pointed Gaussian to soften the boundaries.
In order to investigate those spires I tried running 5 million iterations of training with a smaller network (10px per cell rather than 7px previously used). The result is quite interesting, but the neighbourhood size is too large, causing the sky to dominate significantly. There is still a tendency for clusters to grow in the upper left direction (causing spires) that I cannot explain. The light-coloured spires in the previous post originate at pixels with high degrees of brightness surrounded by darkness, but in the image below we can see they are also present in lower contrast areas (e.g. the sky).
The difference between the two following images is a small change in the neighbourhood function (larger on top). I can’t explain the emergence of these beige spires; they emanate from very small areas of the same colour in the original image, but I’m not sure what drives their expansion into the sky. They seem to expand increasingly according to the number of training iterations. I’m doing another run with a larger neighbourhood and see what happens.
I found a bug in ANNetGPGPU that resulted in the neighbourhood function having a hard fall-off rather than a gradual Gaussian decay. The result really effected the final aesthetic even after millions of iterations. I also changed from HSV to RGB colour model, which was causing some artifacts. The image above shows a short training session to make sure things are going in the right direction. Extra thanks to Daniel Frenzel for the very fast turnaround fixing the bug!
The following images show the latest iteration of the work. In this version the SOM is blended with the original pano, and I’ve moved the ‘horizon’ higher up. The transition between the pano and the lower resolution SOM (7px per cell) is quite nice and leads to a bit of a tilt-shift effect at the ‘horizon’. See the detail image below. I’m not entirely keen on the degree to which the blue sky over-powers the buildings, so I’m now rendering a version with a smaller max neighbourhood and we’ll see what that looks like tomorrow.
I’ve been very busy with other projects, including a big exhibition of Watching and Dreaming (2001: A Space Odyssey) at the Digital Art Biennial in Montreal this month. I had already shot two attempts at a “final” panorama, but have been having significant issues stitching the images together. I finally have a suitable result, but it uses the wider images, which limits the resolution to ~60,000 pixels, which is short of the 80,000px target:
The following images show the results of training a SOM using the colour values of the original pano pixels and bypassing the segmentation processes. The result is that performance is improved (a 5px cell is possible), and the images are quite interesting, although the white sky dominates the image, where it was previously ignored as it was seen as a single giant segment and ignored.
Based on the explorations up to this point, I wanted to post some initial designs using the Gaussian renderer and blending with the original panorama at a more carefully specified horizon. The following images show the results with varying multipliers applied to the Gaussian functions used to draw SOM cells. If you look really closely, you may be able to see that one of the red benches dissolves into a plume of red. A person sitting on one of the red benches on the left turns into thick black smoke.
The following images show exponential and linear (respectively) increases of neighbourhood size where the maximum neighbourhood size at the top is the same (1600). They don’t seem all that aesthetically different, nor does there appear to be any smoother a transition at the horizon. The linear version is slightly more interesting (and cosmological).
After making the decision to seed the top row with averaged values I executed a few runs using a larger network (20px for each SOM cell) and a range of maximum neighbourhood sizes (10, 50, 100, 500, and 1000). The last is so large that it nearly covers the entire width of the map (~1400 cells). Following are a few rectangle renderings of the result. Note, I used too large a scaler, so each cell is drawn quite large. I think this explains the large rectangles at the bottom of the image, but that needs more investigation.
I wanted to follow up on some of the conceptual aspects I touched on in my previous post. When I started my Masters degree I was really interested in how I could create a computational process that was not random nor a perfect reflection of my intention nor the outside world. In general, I try to use randomness as sparingly as possible. In my Masters project I, as best as I can recall, did not use any randomness at all, making the system deterministic. Since it used images from its environment (that was constantly changing) it was unpredictable, and yet deterministic.
Thanks to Daniel Frenzel, ANNetGPGPU now supports setting different neighbourhood sizes in a single network. This means I will no longer have to generate a different source and data file for each neighbourhood size. Following is an image visualizing the weights (using the old rectangle renderer for performance reasons).
In the following images (raw on top, original overlap on bottom) I used a smaller range of neighbhourhood sizes compared to the previous post (1-30). I was hoping the buildings would be less obliterated, and I actually prefer some of the horizontality of the previous 1-40 version. The reason why I’m unhappy with these results is that the visually interesting structure only begins on the horizon, and thus the abstraction should only start at that point.
The following is a 4k video of a portion of the colour field sequence previously posted.
The following image is constructed by taking a single row from each of the trained networks such that the lowest (bottom) row has the smallest neighbourhood size (1) and the highest (top) row has the largest (170). Note that very little of the image is readable because the image is abstracted very quickly as the neighbourhood size increases in steps of 1. Note some stability in structure (position of colours) near the bottom of the image. Near the top the increasing horizontality indicates large differences in structure in subsequent neighbourhood sizes.
The following images show a sequence where the neighbourhood size increases linearly from 1 to 33 (skipping even neighbourhood sizes). In networks with neighbourhoods larger than 33, the results look about the same structurally and occasionally show large changes in subsequent neighbourhood sizes. The larger the neighbourhood the more instability there seems to be.
I realized that part of why the SOMs in the previous sequences are inconsistent over time is because a time seeded random number is used to rearrange of order of the segments (inputs) for each SOM, which adds significant random variation. I first tried to use serial training in ANNetGPGPU, but found that it is significantly slower than random training (serial training time: 745.579s; random training time: 12.1s). I also rewrote the code so that the next network actually uses the previous network as a starting point, rather than starting with the original training data for each neighbourhood size. The results, a selection of which follows, certainly have more cohesion, but the use of the previous network reduces some of the colour variation.
I managed to batch generate 170 different SOMs with different neighbourhood sizes (default/170 to default). Unfortunately, they are not stable over time; even though they have the same initial conditions, each result has a different structure. I’m not sure if this is due to the change of neighbourhood size, or some indeterminism in the way the algorithm proceeds on the GPU. Following is a selection of the sequence with increasing neighbourhood sizes. These SOMs are rendered using the new code that renders with Gaussianoids rather than rectangles.
The following images are renderings of the SOM structure trained and visualized using the same methods as previously posted. The only difference here is that much smaller neighbourhood sizes are used (top: default/150 and bottom: default/50)