Posted: August 26, 2019 at 5:51 pm

Before starting the recent collage explorations, I had been doing more reading and thinking about entropy, see notes following. I also got a chance to watch a lecture that Sarah Dunsiger sent on entropy and emergence, I’ve included my notes on that below as well.

Entropy

• Natural log of the number of states in a system multiplied by a constant
• The log reduces very large numbers and does little for small numbers. (e.g. ln(10e06) = ~16, ln(10e02) = ~7
• The constant (Boltzmann) is a very small number (~1e-23)
• So entropy is a small representation of really large numbers of possible states.
• All possible 640×480 images in 8bit have 5e12 possible states and an ‘entropy’ of 4.04e-22. (Does it make any sense to think of entropy of an image?? An image is not dynamical, entropy is about dynamics, not structure.)
• Second Law of Thermodynamics:
  • Entropy of closed systems never decreases (the number of possible states only increases until equilibrium, maximum entropy)
  • Entropy in open systems may decrease if the environment entropy increases (the number of possible states may decrease if the number of states in the environment increases)
• is entropy about the propagation of energy? Does a system with more energy have more states? If it has more states, it looses that energy to the environment (increasing the number of its states in the environment).
• is there some analogy in ML? Could the energy be the state of excitement of the initial conditions? The rate of learning?
• More entropy means more complexity because more information is needed to represent the potential states of a system.
• This seems more about the constraints of the system than the specific energy states.

Sarah Papers

• order can be introduced from entropy alone
• order from disorder?
• the whole often resembles the part (chiral particles make chiral structure)

Entropy and Emergence (Video Lecture)

• entropy as a measure of what you don’t know about the state of a system
• fewer states means more certainty due to less possibilities.
• a high-entropy system is random / has many states and no constraint.
• entropy as the minimum number of binary questions one must ask to fully determine the system.
• random needs every question whereas a pattern can be compressed
• Key take-away: entropy does not indicate disorder because a system may have more ordered states than disordered states.