Updates to Segmentation and Clustering
Posted: December 6, 2013 at 7:27 pm
After implementing the predictor in the main DM program, I had the chance to run the and them dump percepts to give some form to the ML generated output previously posted. The results were quite weak. It appears that there is simply too much information to be encapsulated by the small number (~2000) of percepts in the system. The first issue was the way that percepts had a tendency to crawl in from the edges due to the centring of clusters. I resolved this by treating percepts on edges differently which merge while still being anchored to the edges. Additionally, the precept’s averaging of constituent regions were weighted to emphasize the most recent percepts (something like 75-85% weighting of current stimulus). This made percepts appear much more stable (over time) than they actually were. In short, a very unstable cluster was represented by a highly specified image. The idea with this was that the presentation of the percepts would be recognition, and in perception the display would show a reconstruction of sensory data from clusters alone. The result was very little correlation between the reconstruction and the sensory information:
Abstraction and The Singularity
Posted: November 25, 2013 at 10:33 am
It seems that one of the foundational ideas of the singularity is that the machines we build will exceed our intentions and act in ways beyond our will and control. Now, I do see how machines act in ways we do not expect, and in the case of a machine with a lot of power over aspects important to human life, that can lead to problems. This happens all the time due to bugs and human error and even happens in the most non-intelligent machines. The centre of the problem seems to be the power we give over to automated systems. (more…)
150,000 frames, Single epoch sequential training
Posted: November 22, 2013 at 8:23 pm
After the previous tests I’ve gotten a better sense of the prediction problem. We realized there may not be enough data in my previous tests to get sufficient training (corresponding to a few days of the full system processing). Additionally, I found a few issues with the segmentation code that could have changed the behaviour of clusters over time. I took the 6 days necessary to train a new data-set. The full data-set is composed of only the day time periods (including sunset and sunrise), and includes approximately 300,000 frames. In 6 days I processed approximately half the set, 150,000 frames. Note that this is actually significantly more data (for the same number of frames) compared to previous examples due to their inclusion of night frames. Following is the resulting error from same MLP learning procedure as used previously, presenting each pattern only once without repeated epoch training, and reporting error after each iteration:
Dream Simulation (Periodic)
Posted: November 1, 2013 at 4:48 pm
I ran a test training the MLP with data filtered such that at least one cluster must change state between subsequent frames. This time, the result is a periodic dream that does not stabilize over 10,000 iterations:
Data Stability Over Time
Posted: October 31, 2013 at 6:01 pm
After meeting with Philippe, we decided that the static dreams could be due to the variation over time in the data-set being “seen” as noise in the data by the ANN, which is attempting to generalize over time. This would explain dreams being static, because they reflect the stability and ignore the noise.
The first step in examining this issue was to look at the stability of my data, which is actually highly stable over time. This is shown in the following plot, which shows the amount of change (in %) between subsequent frames in the foreground and background data-sets:
Dream Simulation (failure)
Posted: October 24, 2013 at 10:40 am
After training the MLP, I thought I should try running it in feedback mode to see if it would actually be predictive. It is not. There are some shifts in input patterns at the start of the dream, but it takes only 8 iterations to settle into reproducing a stable pattern. The following image shows the results of the first ~1000 iterations of the dream (truncated from 10,000 iterations). The left image is the raw output while the right is the thresholded version. Note the slight changes in output patterns early in the sequence (far left edge of both panels).
Machine Learning Progress? (non-recurrent MLP)
Posted: October 21, 2013 at 4:59 pm
I’ve put my leaking code problems aside for now to continue working on the project, the next phase being the ML stuff. So I’m now using FANN because while OpenNN was nicer, more complete and active, it did not provide functions for online / sequential learning needed for this project.
This is my second attempt to train an MLP with plausible data produced by the system. The input is a set of 41,887 state vectors (representing the presence of clusters at each moment in time) produced by a previous run of the segmentation and clustering system. Each element in the vector is a Boolean value corresponding to each perceptual cluster: 0 when the cluster is not present in the frame and 1 when it is. For training, 0 to 1 values are scaled to -1 to +1. The previous attempt appeared to work because the output resembled the input, but I realized after running prototype feedback (dreaming) code that the network was trained just to reproduce the input pattern, not the next input pattern.
The MLP here is considered a canonical case to compare with future sequential learning and contained three layers (1026 input, 103 hidden, 1026 output), and was presented the whole input set over 50 epochs. The network was presented a single state at each iteration, not a window of states over time. The code is a modified version of the FANN xor_example.cpp and uses the “rProp” learning algorithm where weights are initialized with random values between -1 and +1. (more…)
Clustering Memory Leak (not solved)
Posted: September 26, 2013 at 3:01 pm
Further testing has shown that there is still a leak in the clustering code, but I can’t figure out where it is. It does seem that cv::Mat is somehow at the centre of this issue though. I had previously noticed that I could stop the memory increase just by commenting out the mergeImages() code (that averages images for two percepts). This week I realized that I can also stop the memory increase if I don’t normalize the feature vector before clustering. Now this really does not make sense since I rewrote all the feature-related code to change from a vector of floats to a cv::Mat so I could use the opencv normalization functions. There is no code overlap between the previous version of normalize and this new version, and yet if I run normalization and clustering memory increases (red line), if I run either independently (blue and green) there is no memory increase:
I can’t recall why I thought this memory increase was solved before. I may have not been running the normalization code in that test, i.e. I was testing clustering with only foreground objects, where normalization is not needed due to perceptual distances in CIELuv.
Clustering Memory Leak Solved!
Posted: September 11, 2013 at 9:31 am
After confirming no leaks in the segmentation code I reran the clustering unit test. Even over 1000 frames, this clustering code showed a significant leak: (The black line is a linear model fitted to the data.)
The leak was caused by the way I was replacing percepUnit instances with newly merged percepUnit instances. The solution was to change the mergePerceps() function from returning the merged percepUnit to modifying a percepUnit in place using pass by reference. A 10,000 frame test overnight has shown that there is no longer a leak in the clustering code:
I did notice some strange percepUnits while debugging the memory leak. I need to confirm that the new background segmenter is producing reasonable results before moving on. After that, the next steps are to merge all these changes from unit tests into the trunk code, and then I can begin rewriting the way threading and rendering is done. At which point I will have caught up with where I was at the start of the summer.
What is The Difference Between Human and Non-Human Animals?
Posted: September 6, 2013 at 10:26 am
So what causes a non-human animal to empathize? I’ve been thinking a lot about how humans and non-human animals differ. I’ve come to the conclusion that there are two major differences:
- I believe humans have an unparalleled ability to abstract, that is build hierarchies of mental representations where details are thrown away to encapsulate larger concepts that can be broadly applied. This is what allows us to convince ourselves of untruths, to mistake our expectations for reality, to exploit others by defining away their suffering and to imagine and build technologies that extend our cognition.
- Many of us are not in a day-to-day struggle for survival. I presume that much of the morality, empathy and free will that we consider crucial to our human identity would melt away under constant threats to survival. Consider cannibalism and infanticide amongst chimps, who are genetically closer to us than they are to any other apes.
This seal in the video certainly has little empathy for the penguins, so why the empathy for the photographer? Perhaps all animals have lines that define “us” and “them” where we choose to empathize or to exploit. I further expect that these lines arise from biological survival: If you are below me on the food chain, then you are “them”, if you are equal on the chain then you are “us”.
Clearly this is a little more complex in humans, but I expect only because of our ability to abstract. We create a concept (say, race or gender) and then use that to move where our line of “us” and “them” is.
Memory Leak (Segmentation)
Posted: September 3, 2013 at 11:58 am
A test over this weekend showed that there is no memory leak or CPU time spikes in any of the segmentation code (the y axis is rss in megabytes):
CPU time is constant over this test, and we can see here that although there are spikes in memory usage, that there is no leak as a linear model of the data (black line) shows a decrease in memory usage over the whole test. The next step is to write a unit test for the clustering code to confirm there is no leak there, and then we can move onto writing new threading/rendering code and adding ML to the system!
Mean Shift Segmentation – Revisited
Posted: September 3, 2013 at 11:52 am
In the last post I talked about a different segmentation approach rather than trying to figure out why the FloodFill() operation was using more and more time (Eventually getting to over 200 seconds per frame). A quick look at the Creativity and Cognition (C&C) version did not show any functional differences compared to the unitTest code. Of course the C&C version was crashing after about 24 hours, which likely was not enough to exhibit the problem. I took a few days to rewrite the backgroundSegmentation() function from scratch. In doing so I noticed a nice new function: meanShiftSegmentation(). The code is now much cleaner, but unfortunately it is not any faster than the old flood-fill version. This is partially because the segmentation is happening at the full 1080p resolution (not 1/4 the pixels as in the floodFill version). I should be able to get it under 1s with some optimization. The good news is that this segmenter works much better, there are no more “blind spots” as the segmenter breaks up the whole image and joins small regions into larger ones rather than leaving unsegmented areas. Here is an example of the segmentation results (in improperly mapped HSV colourspace): (more…)
Memory leak solved?
Posted: August 26, 2013 at 3:09 pm
After almost two months, and seeking the help of quite a number of people who know a lot more about C++ than me, the problem I’ve been facing with the memory leak has been (at least partially) resolved, thanks to this post. It turns out that this whole time the way I was measuring ram usage was incorrect (maxrss via rusage vs rss via /proc). Thus all memory plots for the whole life of this project were inaccurate. This does not effect the fact that the system did eventually crash after using too much memory, but at least it means I should be able to actually find the problem. (more…)
What is Language?
Posted: July 10, 2013 at 10:22 am
I think language begins with the ability to perceive two sensory patterns that are different (because they are always different) as the same. I think of this as a clustering process, the carving of chunks from a continuous space. It’s a choice of saying these are the same, when the aren’t. It’s an imposition of a compartmentalized structure (concept) on continuous chaos (reality). This could be seen as a filtering process.
These (prelinguistic) clusters at the atoms of perception. We build the perceptual world from these chunks of assumed sameness. These chunks are also the material from which we learn predictions of the world, we learn constraints and periodicity by predicting the occurrence of these clusters.
linguistic symbols are build up layer by layer of these clusters, forming meta-clusters that hold abstract notions such as “object”. Additionally, the notions such as causality, justice, good and evil all result from abstractions of the predictions (via a similar clustering process) combined with emotional valence. I think this is pretty close to Barsalou.
Post Creativity and Cognition
Posted: July 8, 2013 at 4:38 pm
I am back in town for almost a week and still struggling with jet-lag. The conference went well, and I will post my paper as soon as the proceedings are on the ACM website. The papers in the conference leaned quite a lot to the HCI side, so I felt like my work was quite a bit at odds with the community. Still it was a good opportunity to meet people and see Australia for the first time. (more…)
Attention and Prediction
Posted: July 8, 2013 at 2:12 pm
After watching this video, I was inspired to think about attention in relation to the Integrative Theory that has been developed during this research project. (more…)
Creativity and Cognition Approaching Fast!
Posted: June 10, 2013 at 10:22 am
I have been hurriedly attempting to fix some of these memory leaks before exhibition at the Creativity and Cognition conference happening next week. In lieu of any significant development progress, this is a grab of what the perceptual output currently looks like: (note the white background has been replaced by the background model.)
~150,000 frame test with live camera
Posted: June 5, 2013 at 2:58 pm
In preparing for the exhibition of a prototype of the system for Creativity and Cognition, I’ve been running the system with a live camera input. The good news is that the system is looking aesthetically interesting (sorry no screen-grabs currently available). The bad news is that this testing seems to have uncovered a memory leak somewhere, as shown in the continued increase of memory usage even after the maximum number of clusters have been reached for FG and BG: (more…)
Posted: May 28, 2013 at 10:38 am
After quite some frustration I have managed to move the segmentation and clustering code into a separate thread. This thread runs as fast as possible, and the main rendering process checks in on the thread between frames to see if new data (percepts) are ready to be rendered. The main thread then makes a local copy of the percepts and then renders them. The main reason for this change is that the rendering rate was always expected to be much faster than the expensive segmentation rate, and keeping it separate keeps segmentation from blocking rendering. Following is a plot from a ~90,000 frame test of the threaded code.
Posted: May 28, 2013 at 9:52 am
Following are images that represent the learning of percepts over time. Each pixel represents the mean colour of a percept at a particular time. The identity of the percept is the Y axis, while the X axis is time. These were the results from the last ~90,000 pixel test, with 2000 background and 1000 foreground clusters requested.
The top image shows the background percepts, while the bottom image shows the foreground percepts. The original images are 90,000 pixels wide, and are available here: background and foreground. We only begin tracking these percepts once the max clusters has been reached. The middle of the original images have been removed where the left sides of the images are the start of the clustering process where all new patterns are clustered with the nearest cluster. The right side of the images shows the same clusters at the end of the test. Note how they diverge from the start to the end of the test. It is presumed that the apparent regularity on the Y axis in the background near the start of the test is due to the repetition of similar background percepts in subsequent frames. This also explains the regularity of colour blocks of foreground percepts at the start of the test. As this set of percepts is constantly adjusted to represent a continuous flow of sensory information, their regularity is reduced to increase their density.