Crackpots #65: Mapping Tones (Patreon)

Hello everyone,

YC here. To quote Howard Beale in Network (1976), “I don't have to tell you things are bad. Everybody knows things are bad.” But while we’re all figuring out what we can do to make things a little less bad, we've still got to keep tinkering; so this dispatch will be, in the parlance of our times, at the intersection of art and technology. A little bit technical, a little bit philosophical. I’ll start with the big idea, then tell you how I got there: I think a lot of people have fundamental misconceptions about photography. Whereas we all understand that paintings or sculptures are artists’ interpretations of the world, we tend to see photographs more like neutral documents of the real. And as a corollary, this way of thinking implies the existence of some platonic version of a photograph onto which we can, post-facto, overlay choices in the form of “editing,” “processing,” or “filters.” My contention is that this is not and has never been the case: all photographs are interpretations. There is no underlying "real" photo, only different choices. Sometimes those choices are hidden from our view, but they're always being made.

I started learning about this stuff a few years ago. Like over 90% of adults in the US, I own a smartphone, and I often used it to take pictures. In the past, whenever a new line of phones hit the market, I’d get excited about the promised advances in its photographic capabilities. And for a while, each new generation seemed to deliver on that promise. The photos were sharper, more vivid, more “professional” looking. But at some point, that sense of straightforward improvement started to change. Newer smartphone photos looked better in some ways, but worse in others, in ways that were hard to define and harder to avoid. People started to catch on to the fact that phones were doing a lot of magic to make photos look the way they do, even if few seemed to understand the specifics. I began to see lots of posts complaining about “overprocessed” smartphone photography, and a whole slew of apps hit the market which promised to make more “natural,” or “unprocessed” photos. That’s where my interest really picked up; I wanted to understand what made the photos look the way they did, and whether I could use my programming skills to do something different. And to do that, I had to start poking around under the hood and understand how photos are made.

So let’s zoom out a bit (pun intended) and talk about what’s happening, why it happens, and what you can do about it. There are some complex technical topics here, but I’ll do my best to give an overview that’s accurate while keeping it simple. The first and most important concept to understand is dynamic range. Dynamic range describes the difference between the largest and smallest values of something. It can be applied to different domains, and comes up often when discussing both audio and photography. In photography, dynamic range is a more technical term for what is commonly called contrast. How big a difference is there between the darkest and brightest parts of an image? That’s the dynamic range.

We can visualize the distribution of brightness (the technical term here is luminance) in a photo with a histogram:

The histogram shows us how much of the image is taken up by each brightness range. Here’s an example of the same image with different amounts of contrast, as well as what each image’s histogram looks like:

In the low-contrast version, all parts of the image are fairly close together in brightness value. In the high contrast version, some parts are very dark and other parts are very bright. Note that low-contrast vs. high-contrast are just descriptive terms; neither one is necessarily good or bad.

In photography, dynamic range is often quantified in stops, which represent amounts of light. Without getting too technical, just note that each stop represents a doubling in the amount of light; so 3 stops are twice as much light as 2 stops, and half as much as 4 stops. The true dynamic range present in a real-world scene can be extremely large; the bright parts of a sunny sky can be about 100,000 times brighter than areas in shadow. Our eyes are very good at adapting to different levels of brightness, and our brains are very good at stitching them together as our eyes move around, so we have the perception of being able to make out details in the bright and dark parts at the same time. The human visual system, with its adaptive ability, is said to possess a dynamic range of more than 20 stops, although in a single moment that number is smaller.

But any given light sensor, whether it’s the eye, film, or digital, has a maximum dynamic range that it can perceive and record. And here’s the first important insight: no camera, analog or digital, sees or records exactly the way our eyes and brain do. Although modern negative film and large digital sensors can get fairly close to the dynamic range of our eyes, they can’t match it yet. And other types of film, or smaller digital sensors like the ones in our phones, have physical constraints which limit them to an even smaller dynamic range. And even if we could capture the full dynamic range of the world with some kind of sensor, the means we have of displaying that information — on a printed page or some kind of electronic screen — are even more limited in range. Note that while I’m focusing here on ranges of brightness, the same concepts apply to reproducing other aspects of reality, like color and fine detail. Somewhere along the way, we inevitably either have to make choices or sacrifice information. 

When we try to capture a scene with a wider range than our medium is capable of recording, some information will be lost. This is often referred to as “clipping.” Clipping shows up as a “wall” on one side of the histogram. There might be detail in the shadows of the left image, or in the highlights of the right image, but we can’t be sure; that detail got unrecoverably flattened out to pure black or pure white because the medium was pushed to its limits.

Clipping is what it looks like when we sacrifice information. The other option is to make choices to fit the range of values in the world onto our recording medium, then fit the range of values on our recording medium into our display medium. Techniques for achieving this have existed since the early days of photography. Ansel Adams, the influential 20th-century photographer, famously remarked that the photo negative is like a composer’s score, and the print is like a musical performance; it’s an interpretation made up of various subjective choices. Those of us who grew up with software like Photoshop and Lightroom are familiar with tools for masking, dodging, and burning; these are named for physical darkroom processes which pre-digital photographers used to meticulously create their prints, because there is more information contained in the negative than can be represented on the page. We can see the markup on, for example, this photo of Audrey Hepburn in 1950 by Dennis Stock. They indicate areas of the print to lighten or darken in order to realize the photographer’s intended vision.

Stock's 1955 photo of James Dean provides another striking example:

So, contrary to the idea espoused by some younger photographers that shooting on film is about achieving some “authentic” unedited ideal, film photography has always involved interpreting, translating, editing. Going from world to film to page necessitates those choices. Modern film photography isn’t really different; it’s just that many of us outsource those choices to the workers at the lab who develop and then digitize our film with a scanner. While modern computer screens can display a much wider dynamic range than paper, they still can’t display the full range captured by a film negative (which itself can’t capture the full range of the world). Scanning and inverting a negative involves making choices like photographers used to make in darkrooms.

Another technique that photographers have long used to work around some of these limitations is exposure bracketing. This technique involves taking multiple shots of the same scene with different exposure values; e.g. one exposure each for capturing the details in the dark parts, the mid-range, and the bright parts of the image. These different exposures are then combined to create the final image, which will contain detail in the very dark or very bright parts of the image which would’ve been clipped in a single exposure.

Now let’s get back to digital cameras, and smartphones in particular. When a digital camera takes a photo, it takes a readout from lots of little light-sensitive cells on its sensor. That sensor data is like a film negative; it contains the “score” for the image, but it has to be interpreted and processed to produce the “performance” — i.e. an image that we can view on a screen. The bare sensor data can be stored in a RAW file, which is sometimes referred to as a “digital negative.” Just like a film negative, the digital data will often contain a wider range of information than we can display on screen, and there are infinite variations in how we can process and interpret that data to produce our final image. (The RAW file will consequently have a much larger file size than the finished image.) If you’re a casual photographer, you may never touch the RAW file, and will allow your camera to do the processing for you. That processing step produces a final image, often in a compressed format like JPEG. JPEGs are like prints — they are lightweight and can be viewed without any further processing. We can produce a variety of different prints from the same negative, or a variety of different JPEGs from the same RAW, but the process is one-way: we can’t make the negative from the print or the RAW from the JPEG.

So if we want the most control of the photos we take with a digital camera, we can shoot in RAW, and then manage the “development” ourselves — I mostly use Adobe Lightroom, but there are lots of options, including free open-source ones like DarkTable. And smartphones can shoot in RAW, too! On an iPhone, this requires the use of a 3rd-party app rather than Apple’s default Camera app. Shooting RAW is less convenient than letting your phone handle the processing for you, but it gives you more control. It also opts out of the default post-processing your phone does for you — for better or worse. What exactly does that processing entail, though? I’m going to speak about iPhones here, as that’s where I have the most knowledge, but I believe non-Apple smartphones use similar approaches.

Getting a smartphone to produce high-quality photos is a major engineering challenge primarily due to the small size of the sensor. A larger sensor, like in a dedicated digital camera, can gather more light and achieve more detail, less noise, and a larger dynamic range. One of the ways that smartphones punch above the weight of their small sensors is by using the bracketing technique I discussed above. This happens automatically behind the scenes whenever you take a standard photo with your iPhone: it actually takes several photos at different levels of exposure and merges them into the final result. This technique is sometimes referred to as “HDR,” even though that’s a bit of a misnomer. A more accurate term for this technique is “tone mapping.” Tone mapping is the process by which a mathematical function takes a range of values larger than a screen can reproduce, and “compresses” them so they all fit (sort of like using a compressor on an audio signal to give it a narrower dynamic range). Modern tone mapping functions take advantage of our knowledge about vision to do this in a smart way. It’s a bit like the dodging, burning, and masking that analog photographers do in the dark room, but it happens via algorithm. Tone mapping isn’t the only type of processing that smartphones do by default, but I think it’s the most impactful.

A few years ago, Apple introduced a new photo process called ProRAW, available only on Pro model iPhones. ProRAW is really neat, but widely misunderstood. ProRAW photos are not actually RAW files — read that link for a more technical breakdown of that point. But they do allow for the editing leeway that a RAW file provides, while also providing the extra dynamic range and reduced noise afforded by the multi-exposure merging. What I think trips people up is that a ProRAW image, by default, looks exactly the same as an equivalent JPEG image — you don’t see the benefits of the extra data (and file size) until you start to tweak the parameters. And the single parameter that makes the biggest difference is the tone mapping strength.

Here’s an example to illustrate what tone mapping does to your images. Here’s a view of a Brooklyn street on a sunny day:

This was captured on an iPhone 15 Pro in ProRAW format. You can clearly make out the details in the sky as well as the street, even though the sky is in reality much, much brighter than the street; tone mapping has compressed the wide dynamic range of the scene, captured in multiple different exposures and merged together, into one that fits on the screen, so I can see detail everywhere. This image would look exactly the same if I captured it as JPEG, but because it’s ProRAW, I can manipulate the strength of the tone mapping. Different editing apps label this slider differently: In Adobe Lightroom, manipulating the strength of the Apple ProRAW profile will change the amount of tone mapping. In RAW Power, there is a “tone map” slider in the “RAW” editing section. In Liit, there’s a slider labeled “HDR.”

This is what it looks like at 50% strength:

And here it is at 0%:

As you can see, we still have detail in the sky (although more of the area around the sun is now clipped to white), but most of the detail in the street is lost. The image became more high-contrast; maybe that’s an effect that we wanted. But by default, the iPhone is optimized for preserving detail. Its tone mapping algorithm is extremely good at preventing clipping. This makes sense when you consider that it’s meant to be used by everyone, including people who are not photographers or tech savvy. It seems fair to assume that the median user would want to be able to make out the details in both the bright and dark parts of the image when they take a photo like this. But sometimes this isn’t what I want; the default values will prevent me from taking a very stark, high-contrast photo, like these by Alan Schaller:

The default iPhone settings would lift the shadows of the image to preserve the detail and prevent clipping to black. Here’s another example of a ProRAW photo I took at default settings:

And here is that same photo with tone mapping set to 0:

I lose a lot of detail in the shadows, but in this case I like the stark, high-contrast effect it creates. Simply adjusting the tone mapping strength is a blunt approach; I could get a lot more specific with my edit by tweaking different parameters if I want to. If I had shot this in JPEG, though, I would quickly start to see artifacts in the image when I pushed it in one direction or another, because the extra information isn't there; it was thrown away as soon as the image was created.

In the end, none of these choices are right or wrong, better or worse, or more real or fake. They’re simply different ways of capturing the light with the sensor and interpreting it for the screen. Modern cameras often make a bunch of choices for us by default, but they also give us the ability to make different choices if we have the know-how. They say the best camera is the one you have on you. That’s why I think it’s much more important to know how your tools work than to acquire the fanciest tools. I hope this was instructive, and helps you get a little bit more out of your photos. Until next time,

YC