The Misunderstood Nature of Entropy (Script) (Patreon)

Entropy and the second law of thermodynamics has been credited with defining the arrow of time, predicting the ultimate heat death of the universe, providing the driving force for the development of structure as well as decay, and excusing the messiness of your room. But what is entropy really, and how fundamental is it to our universe?

Entropy is surely one of the most intriguing and misunderstood concepts in all of physics. The entropy of the universe must always increase – so says the second law of thermodynamics.  It’s a law that seems emergent from deeper laws – it’s statistical in nature – and yet may ultimately be more fundamental and unavoidable than any other law of physics.

Einstein said that the thermodynamics that encapsulates the 2nd law “is the only physical theory of universal content which I am convinced will never be overthrown.” And the great astrophysicist Sir Arthur Eddington warned  “if your theory is found to be against the second law of thermodynamics I can give you no hope; there is nothing for it but to collapse in deepest humiliation.”

We’ve looked at entropy in the past, but it’s time to go much deeper – to unravel the great unraveler. Over some upcoming episodes we’ll explore different aspects and consequences of entropy, including its role in black hole thermodynamics and how it’ll lead to the the end of our universe. But today we’ll see what entropy really is, and why the second law of thermodynamics is considered to be so fundamental and so unavoidable.

Let’s start from the beginning. In 1824 Sadi Carnot published his Reflections on the Motive Power of Fire, in which he revealed the theory for perfect engine efficiency. Heat engines – which in Carnot’s day were the new-fangled steam engines – work by turning the flow of heat energy into mechanical energy. For heat to flow you need two reservoirs of different temperature. A perfectly efficient engine – one undergoing the Carnot cycle – converts all transferred heat energy into useful work. In principle, that work can be converted back into to heat and so the temperature differential can be reestablished. However an INefficient engine will slowly deplete the difference in temperature, reducing the heat flow and the engine winds down.

Around a half century after Carnot, Rudolf Clausius was inspired to quantify this tendency of heat engines to decay over time. Enter entropy. He defined entropy as the internal property that changes as heat energy moves around within a system. Specifically, the change in entropy of each reservoir is the heat energy going into or out of it divided by its temperature. For a Carnot cycle, the change in entropy is zero, but for any less efficient cycle entropy increases. In fact an increase in entropy means the heat reservoirs approach the same temperature, reducing their capacity to do useful work.

Carnot and Clausius’s work revealed entropy as a measure of how evenly spread out a system’s energy is. The more evenly spread, the less useful the energy. And for an isolated system, the best you can hope for is that the separation of energy and the entropy remain constant. In reality it will almost always increase unless energy comes in from the outside to reestablish a temperature differential.

This understanding of entropy is in terms of flowing heat, and it came from the days when many, including Carnot himself, believed that heat was a physical fluid called caloric. It took a revolution to understand the reality of entropy. That revolution was statistical mechanics, founded by the great Ludwig Boltzmann with his kinetic theory of gases, which explained thermodynamic behavior as the summed result of the individual motion of tiny particles under Newton’s laws of motion.

Stat mech is really astounding.  It’s founded on an absurdly simple idea: for a given set of large-scale, observable properties, every possible configuration of particles that could give those properties is equally likely. Let’s add some physics-speak. By configuration I mean the exact arrangement of positions, velocities, etc. for all microscopic particles. We call this the microstate. And we call the specific combination of large-scale, macroscopic properties the macrostate. Macrostates are entirely defined by thermodynamic properties - temperature, pressure, density, volume, and number of particles.

For a given macrostate, all microstates consistent with its thermodynamic properties are equally likely. For some macrostates there are lots of different microstates that lead to roughly the same thermodynamic properties, while other macrostates can be produced by only very few microstates.-

OK, one more fact: if you leave a system to do its own thing it’ll eventually try out all possible microstates that are possible given the laws of physics. All particle arrangements will eventually happen. So if you look at the system at some random point in time, it’ll be in a completely random microstate chosen from all possible microstates. And what macrostate will it be in? Well, probably the one that’s consistent with the most microstates.

We can think about these micro- and macrostates with an analogy. This is a Go board. Let’s say you place 180 black stones at random. Every possible specific arrangement is a microstate, while the overall shape of the distribution is a macrostate. There are nearly 2-to-the-power-of-107 ways to arrange the pieces and almost all of them are pretty evenly mixed – so all roughly the same macrostate. Some microstates are weird though, and give different macrostates – different “average distributions”. For example there’s one where all the stones are on one side. That microstate is a factor of 2x10^107 less likely than one of the many smoothly mixed microstates. And the larger the board the less likely it is to end up in a weird arrangement. And for a room full of 10-to-the-26 molecules of air, the chance of getting all of the air molecules on one side of the room by chance is so small that it never happens.

We’ve been talking a lot about particle position, but really, that Go board is an analogy for all possible combinations of all properties – position, momentum, spin and vibration, really any “degree of freedom” that the system can have. We call this space of properties a phase space. And instead of particles being distributed through position space, a microstate is really defined by how energy is distributed through phase space. The average distribution of individual particles in phase space defines the thermodynamic properties of a system. That’s why these similarly-shaped distributions on the Go board correspond to the same macrostate, while the clustered spread does not.

OK, so you leave a system alone long enough: its particles and its energy will find its way into all the different forms that are possible. The vast majority of the possible distributions of energy leave the system very close to a single macrostate – that’s the state of thermal equilibrium, in which energy is maximally spread out, and temperature, pressure, density, volume, etc. have the values we expect from classical thermodynamics.

Statistical mechanics tells us why large-scale systems have the properties they do. But what does this have to do with entropy? Ludwig Boltzmann figured this one out too. The Boltzmann equation tells us that entropy is the logarithm of the number of microstates consistent with the current macrostate, times the Boltzmann constant. So our smoothly spread-out “equilibrium” Go board has high entropy and our clustered board has low entropy.

By the way, there are certain - arrangements special microstates – that look highly ordered but are still consistent with our high-entropy macrostate. For example, if we try draw pictures or write words in phase space. This is where we get to a point of common confusion. Order is not the same thing as low entropy, and the second-law isn’t always the tendency towards disorder. Thermodynamic entropy, the only “special” arrangements of particles that change entropy are the ones that change the thermodynamic properties; not the ones that spell out cuss words or mess up your room. To get deeper into that we’ll need to talk about information entropy, which we’ll also need for black hole thermodynamics, and will also take another episode.

OK, so the macrostate that defines thermodynamic equilibrium is by definition the one with the most microstates, which also means the maximum entropy. Any system NOT in equilibrium must increase in entropy simply because, at any future time, its microstate will most likely be in one of the more common types of microstate. This is assuming you don’t force the system from the outside. It’s possible to take each Go stone and place it in a particular spot - to construct a special microstate. Or to use a vacuum pump and a glass wall to move all of the air to one side of the room. In both cases you’re reducing the number of accessible microstates, which by definition must reduce entropy.  But to do so you must introduce an external source of energy. Heat must flow between your system and the outside universe is a way that increases the entropy of the universe as a whole.

Statistical mechanics inevitably leads to entropy and the 2nd law. And it does so by something so fundamental and basic that it’s impossible to deny. It comes from counting the ways that energy can be distributed. The inevitability of the rise of entropy is as fundamental as counting. That’s why Einstein and Eddington were so sure of it. But entropy is also statistical and emerges from the behavior of particles under the laws of motion. This is where the 2nd Law appears to add something new to the universe, not seen in the more fundamental laws. It seems to add the arrow of time. 

See, the laws of motion – whether Newtonian or quantum mechanical - don’t care about the direction of time. And yet the 2nd law of thermodynamics clearly distinguishes between the past and the future. We talked a little about this in our episode on the Physics of Life, where we saw how entropy drives both the increase and decay of complexity. It’s almost like the concept of time is emergent and statistical, just like entropy. Again, we’ll delve deeper into this in the future. For now, please be careful to keep your number of accessible microstates low, avoid thermal equilibrium, and keep being that brilliant macrostate that is you until I see you next week on Space Time.