I’ve finally got things working such that the arrangement of segments reflects their structure as learned by the SOM; the lack of such structure in previous results were due to a bug in ANNetGPGPU. The following images are all the result of a 100,000 iteration training period with random initial conditions. The last three images are details of the second image.
After tweaking some more code, the template matched percepts look even more blocky than the previous centred ones. Due to this, I’m doing one more test using template matching, and if that does not provide better results, I’ll abandon template matching for percept generation. I’m also considering changing the area and aspect ratio features to the width and height of segments. Currently the aspect ratio is over weighted because as its not normalized; normalizing could be based on the the widest and tallest aspects, being 1920 and 800 respectively, but the different would overly weight the wide percepts.
The following images show some very early work using a SOM (implemented in ANNetGPGPU) to arrange the montages of segmented regions. Rather than sorting according to single parameters, e.g. colour, the composition should reflect the similarity of regions across all parameters. In the first image, I used the wrong segment directory where I did not tweak the overexposed sky. The result is a number of very large white regions at the bottom. While all regions are clearly organized according to their colour parameters, the clusters don’t look right. I think I messed up some of the code mapping the 1D BMU IDs with 2D positions in the SOM lattice.
The following collage explorations use the same code used to generate collages during my Banff Centre Residency, where the segments are extracted from the pano previously posted. These images are composed of over 230,000 individual segments, but the very large regions from the monochromatic sky are filtered out. The images without mattes are details of the collages immediately above them. Note that these are quite low resolution as the original segments are so small. I’ve also extracted segments from a more coarse segmentation and I’ll generate some collages from those this afternoon.
In my previous post I claimed that the inclusion of large readable percepts in the output was due to the lack of filtering, but in fact I was filtering out large percepts. It seems the appearance of apparently large percepts is due to tweaks I made to the feature vector for regions. Previously, the area of percepts was very small because it was normalized relative to the largest possible region (1920×800 pixels); as percepts this large are very unlikely, the area feature had less range than the other features. I increased the weight of the area feature ten fold hoping it would increase the diversity of percept size. Instead, it seems this extra sensitivity preserves percepts composed of larger regions, increasing their visual emphasis.
Through the whole Banff Centre residency I was trying to find a midway point between pointillism and the more readable percepts; it seems I stumbled the solution. I’m still not happy with their instability, and I’m now generating new percepts where I use template matching so that regions associated with one cluster are matched according to their visual features (to an extent). I have no idea how this will look, but it could make percepts less likely to be symmetrical, since regions are no longer centred in percepts.