Reversing Entropy with Maxwell's Demon (Patreon)

INTRO

Entropy is sometimes described as a measure of disorder or randomness. The second law of thermodynamics – the law that entropy must, on average, increase – has been interpreted as the inevitability of the decay of structure. This is … misleading. As we saw in our episode on the physics of life, structure can develop in one region even as the entropy of the universe rises. Ultimately, entropy is a measure of the availability of free energy – of energy that isn’t hopelessly mixed in thermal equilibrium. Pump energy into a small system and complexity can thrive..

But entropy IS connected to disorder and randomness in a very real way. See, entropy is a measure of ignorance. Entropy very directly measures how much information we’re DON’T have about a system. But with knowledge comes power. Literally. Know a system perfectly and you can, for example, extract its energy. To show you how I’m going to have to introduce you to a friend of mine named Maxwell’s demon. But before I do that, a little recap is in order.

This episode will build heavily on our recent episode, linked here - worth a watch if you haven’t. There we show that entropy is a direct measure hidden information. It’s defined by the number of possible configurations of particles – “microstates” in physics-speak – that could produce the same, observed set of macroscopic observables – or the same macrostate. More precisely, if we only know the thermodynamic properties of a system – temperature, pressure, volume, etc., how much extra information would we need to perfectly describe the current microscopic arrangement? Let’s explore this using the analogy we used last time- the Go Board.

Let’s say we place 180 black stones on random vertices of our Go board. In the vast majority of such arrangements stones are spread evenly over the board. These all correspond to the same smoothly-distributed macrostate, and to maximum entropy. Even if you knew you were in this macrostate, you still wouldn’t know much about which vertex had a stone. There are around 10^108 possible microstates that give this smoothly-distributed macrostate, and you’d need close to 361 bits of information – one for each 19x19 vertex – to fully describe the board.

On the other hand, what if the black stones all ended up on the right? There are only a few configurations that looks like this, which means knowing the macrostate almost tells you the exact microstate – you only need a few more bits of information about that middle column. This is close to minimum entropy. Now, those stones all being on one side could represent particles that all have high or all have low energies, or particles that are all on one side of a room. Either way, this would be far from thermal equilibrium, and particles would quickly flow to fill the available space. You could extract energy from that flow.

Ok, so that was one weird Go configuration. But what about this one, where the stones are arranged in stripes? Again, there’s only one arrangement like this. So it should also be low entropy, no? Well… not quite. The particles are already pretty well mixed. They don’t need to move far to reach equilibrium. How are you going to extract energy? In fact this configuration isn’t low entropy in the thermodynamic sense. And this is the critical distinction: thermodynamic entropy is related to the amount of hidden information based on thermodynamic knowledge only. It defines how far a system is from thermal equilibrium, and so also defines the availability of free energy – energy that can be extracted as a system moves back to equilibrium.

Thermodynamic entropy is low if there are differences in the average thermodynamic properties from one macroscopic region compared to another. But weird configurations like these, where average thermodynamic properties are the same everywhere, are just weird microstates among the many microstates of a very high entropy macrostate.

So much for structure and organization always meaning low entropy, right? Well, actually, it turns out that even these specific high-entropy configurations can be transformed to low entropy – as long as we have information about the state. To understand this, you’re finally going to have to meet Maxwell’s demon.

As well as unifying the equations of electromagnetism, James Clerk Maxwell was one of the founders of statistical mechanics. Understanding that entropy was a statistical phenomenon, he came up with a thought experiment to explore just how fundamental the second law of thermodynamics really was.

He imagined a box with two halves, sealed by a wall between them. The wall has a tiny door, large enough for a single molecule to pass through. The air throughout the box is the same temperature, so even if we opened the door, temperature would stay the same. The halves are in thermal equilibrium with each other, and if the box is isolated from its surroundings then this is the state of maximum entropy.

Let’s introduce the demon. The demon has the ability to observe the speed and trajectories of individual particles in the system. It can also open and close the door between the sides. Every time the demon sees a high-speed particle approaching from the right it opens the door to let it pass to the left side. And it lets low-speed particles pass left to right. Soon enough, the left side is full of fast particles and is hot, the right slow particles and is cold. We’re no longer in thermal equilibrium and the entropy is lower than before the demon started.  This temperature differential can be used to drive a heat engine and do work. But all of this appears to have been done without exchanging energy or entropy with the outside universe. This seems to violate the second law of thermodynamics, which demands entropy remain constant or increase unless energy is exchanged with the outside universe.

Believe it or not, Maxwell’s demon was seen as a serious problem for the second law for many years. Physicists tried to figure out the flaw in the argument. It turns out that there are plausible, non-demonic, or even intelligent mechanism to detect approaching particles and open the door. Mechanisms which, in principle, don’t increase entropy.

But it turns out that there’s one step in the process of sorting particles where the increase in entropy is unavoidable. Weirdly, its not in the measurement of particles trajectories or the motion of the door – it’s in the storage of information. It’s in the memory of the demon! See, in order for the demon to do its job in must learn about the particles. The demon, or the particle-sorting system it represents, it must start in a known, predictable state which is altered by interaction with a particle into an unknown state. From our point of view, the randomness of the particles decreases but that randomness is transferred to the memory of the demon.

But even demons have finite memory. At some point, the system for sorting particles needs to be reset to continue its work. To erase the information of past sorting means to transform some arbitrarily jumbled state into a single known state. That’s a physical process that reduces the demon’s internal entropy, and that takes an irreversible transfer of energy. The demon has to radiate heat, which means transferring entropy back out into the box, or to the universe.  Either way, the 2nd law is preserved.

This resolution was proposed by Rolf Landauer in 1960, nearly 100 years after Maxwell proposed the conundrum. Landauer's principle states that "any logically irreversible manipulation of information, such as the erasure of a bit or the merging of two computation paths, must be accompanied by a corresponding entropy increase," and this has come to be seen as a fundamental limit to the efficiency of computation.

Both Maxwell’s conundrum and Landauer’s resolution are fascinating because they highlight the fundamental link between entropy and information. Firstly, they show that information about a system allows us to extract useful work. If you have perfect knowledge of the state of all particles in the box you could open and close the door in exactly the right sequence to sort particles any way you liked. You could turn any perfectly-known equilibrium state into literally any other non-equilibrium state. You could turn information into work. But ultimately, processing that information DOES increase entropy – you generate heat as you compute – in particular as you reset your own memory. The entropy of the universe must increase. And yet, knowing the microstate of a system, no matter how thermodynamically mixed, allows you to decrease its entropy. As long as you can power your computation. Information is power.

Claude Shannon, the father of information theory, was deeply inspired by the close connection between entropy and information. He defined a new type of entropy – Shannon entropy. It can be interpreted as the amount of previously-hidden information that can potentially be revealed by the observation of a random event. In a sense it’s a measure of the uncertainty of the event – a more uncertain or more unusual event is more revealing than a more predictable event. So, for example, the roll of a die has more entropy than the flip of a coin. There were more potential outcomes in the former, so more uncertainty has been eliminated when the die is cast. Shannon used the term entropy for this measure because his formula was almost identical to the formula for thermodynamic entropy – just multiply by the Boltzmann constant to get the latter. If you take all possible results of the die roll to be equally likely, then Shannon’s formula looks just like the Boltzmann equation. This isn’t a coincidence.

In fact Shannon entropy is, in a sense, just a generalization of thermodynamic entropy. The former describes the amount of hidden information in any system. The latter applies specifically to thermodynamic systems. And there’s a third, perhaps even more fundamental type of entropy – that’s quantum entropy, also known as Von Neumann entropy. It describes the hidden information in quantum systems, but more accurately is a measure of the entanglement within quantum systems. In fact, the evolution of quantum entanglement may be the ultimate source of entropy, the second law, the limits of information processing, and even the arrow of time. We’ll get tangled up in all of that in upcoming episodes of Space Time.