CHAPTER 89: Lighting and Shadows (Patreon)
Content
CHAPTER 89: Lighting and Shadows
In Hollow Knight, the light was simple. One source per scene: the Knight's halo. A precalculated lighting map that darkened the edges of the screen and left the center bright. The bioluminescent mushrooms of Greenpath and the lanterns of City of Tears were secondary sources that added to the map with quadratic falloff functions. All in 2D. All flat. All solved with arithmetic a 1998 processor could run without breaking a sweat.
Los Santos needed real light.
The light that exists in three-dimensional space and that interacts with surfaces according to their angle, their distance, their material. The sunlight that falls on a building and lights the facade facing the sun and leaves in shadow the one facing the opposite side. The light of a lamppost that creates a cone of illumination on the sidewalk and that fades with distance. The light of a car's headlights that sweeps the street when the car turns a corner and that casts shadows of the poles and the pedestrians that move with the turning of the vehicle.
George knew that lighting was what separated a 3D game that looked like a game from one that looked like a place. The polygons could be perfect, the textures could be detailed, the animations could be fluid. But if the light wasn't right, everything looked fake. The human brain evaluated the reality of a space by how light behaved in it, and light that didn't behave the way the brain expected produced a dissonance that no amount of polygons could compensate for.
*
George implemented three types of light sources in NGN.
The first was the directional light. An infinitely distant source that emitted parallel rays in a uniform direction. It was the sun. All the rays of a directional light traveled in the same direction, which meant that every surface of the world received light from the same angle. The intensity didn't diminish with distance because the source was infinitely far: a building 10 meters from the player received the same amount of sunlight as one 2 kilometers away.
The second was the point light. A point source that emitted light in all directions from a specific point in space. It was a lamppost, a bulb, a fire. The intensity of a point light diminished with the square of the distance: at 2 meters the light was 4 times weaker than at 1 meter. George used the same attenuation function he had used in Hollow Knight but extended to three dimensions, which meant a point light lit a sphere of space instead of a circle.
The third was the spot light. A point light with direction: a cone of light that narrowed or widened according to a defined angle. They were the headlights of cars, the flashlights, the searchlights. The spot light combined the point light's attenuation by distance with an attenuation by angle: the light was most intense at the center of the cone and faded toward the edges.
The sun, the lampposts, and the headlights. Three types of light that combined could simulate any lighting condition of Los Santos: midday with the directional sun dominating, sunset with the sun low and the lampposts starting to turn on, night with the lampposts and the headlights as the only sources, and dawn where the sun gradually took control again.
*
The question that separated the engines of 2005 from the ones that would come after was where the lighting was calculated: in the vertices or in the pixels.
Per-vertex lighting was the industry standard technique in 2005. The engine calculated the color of the light at each vertex of a model and then interpolated the colors between the vertices to fill the surface. It was fast because a 500-polygon model had about 300 vertices, and calculating the light at 300 points was cheap.
But it was imprecise: large surfaces with few vertices lit uniformly and the transitions between light and shadow were smooth in ways that didn't correspond to reality. A flat floor with 4 vertices lit by a lamppost looked like a smooth gradient that didn't show the sharp edge of the cone of light.
Per-pixel lighting calculated the light at each pixel of the screen. It was precise because each point of each surface received its own lighting calculation, which produced sharp transitions between light and shadow and specular effects that shone exactly where they should shine. But it was expensive: a 640x480 screen had 307,200 pixels, and calculating the lighting of three or four light sources at each one of them 30 times per second was a volume of computation that most of the hardware of 2005 couldn't handle.
George chose a hybrid system that didn't exist in any commercial engine of the era.
On PC, NGN used per-pixel lighting for the objects near the player (within a radius of 50 meters) and per-vertex for the distant objects. The transition between the two systems was gradual: a 10-meter blend where the lighting transitioned from per-pixel to per-vertex with no visible cut. The player never saw the change because the objects 50 meters away were already small enough on screen that the difference was imperceptible.
On PS2, NGN used per-vertex for everything with one exception: the ground immediately below the player and the nearby vehicles used a low-resolution dynamic light map that simulated per-pixel at a fraction of the cost. Daniel implemented the map as a 64x64 texture that was projected onto the ground in a radius of 20 meters around the player and that updated every frame with the positions of the nearby lights.
It wasn't real per-pixel. But the result was that the lampposts created convincing pools of light on the asphalt and that the player's car headlights lit the street with edges that weren't completely smooth.
On Xbox, NGN used per-pixel in a radius of 30 meters with per-vertex beyond. The Xbox GPU, based on the Nvidia architecture, was significantly more powerful than the PS2's for pixel shader operations, which gave it a headroom Daniel exploited whenever he could.
The hybrid system was more complex to implement than either of the two separately. But it produced results that were visually competitive with the best PC games of the era on the PC version, and that were better than any PS2 game on the PS2 version. George was using the hardware of each platform to the maximum of its capacity instead of settling for the lowest common denominator.
*
The shadows were the second component of the light. Where the light defined what was illuminated, the shadows defined what wasn't. And the shadows were harder to render than the light because they required the engine to know which objects blocked which light sources, which was a geometry problem that grew exponentially with the number of objects and sources.
George implemented shadow maps: the engine rendered the scene from the perspective of each light source, generating a depth image that represented the distance from the source to the first object the light touched. Then, when rendering the scene from the player's perspective, the engine compared the depth of each pixel with the depth stored in the shadow map. If the pixel was farther from the source than the shadow map indicated, it meant there was something between the source and the pixel. That pixel was in shadow.
The problem with shadow maps was the resolution. A low-resolution shadow map produced shadows with pixelated, stepped edges that looked like saw teeth. A high-resolution shadow map produced smooth, precise shadows but consumed GPU memory and rendering time.
George assigned resolutions by platform: 1024x1024 on PC, 512x512 on Xbox, 256x256 on PS2. And he applied PCF: Percentage Closer Filtering, a technique that softened the edges of shadows by sampling multiple points of the shadow map around each pixel and averaging the results. With PCF, a 256x256 shadow on PS2 looked as if it had twice the resolution because the edges blurred instead of stepping.
PCF was a technique that in 2005 existed in academic papers but that almost no commercial game implemented because the cost was 4 to 16 samples per pixel in shadow. George implemented it with 4 samples on PS2, 9 on Xbox, and 16 on PC, with an optimization that didn't exist in the papers: it only applied PCF at the edges of the shadows, not in the interior.
The interior of a shadow was complete shadow with no ambiguity, so sampling it multiple times was waste. Only the edge pixels, where the shadow transitioned to light, needed the softening. This optimization reduced the cost of PCF by 60% and was what made it viable to use on PS2.
Daniel looked at the implementation of edge PCF and asked:
"Where did you get the edges-only optimization?"
George looked at him.
"It's logic. Why sample the interior of a shadow if you already know it's shadow?"
"It's logic no one has thought to publish."
"Maybe no one needed it until now."
Daniel nodded. He did it resignedly, because he had already learned that George's answers to the question "where did you get that" were never entirely satisfying and that insisting changed nothing.
*
In mid-March, George integrated the lighting and shadows with the day/night cycle that didn't yet exist as a complete system but that could be simulated by moving the direction of the directional light manually.
George rotated the sun from twelve noon to six in the evening in the test scene: the cube (which was no longer a single cube but a set of twenty cubes of different sizes representing primitive buildings on a primitive street) plus a plane that was the ground plus three point lights that were lampposts.
At twelve noon simulated, the sun was directly overhead. The shadows of the cubes were short, hugging the base of each building. The light was white and uniform. The lampposts were off because their intensity was programmed to activate when the ambient light dropped below a certain threshold.
George moved the sun toward the horizon. The shadows lengthened. The direction changed: from vertical to diagonal, casting toward the east as the sun descended in the west. The buildings started to block the light of the buildings behind them, creating shadow zones that grew with each degree of the sun's rotation. The color of the light changed from white to yellow to orange, because George had implemented a color gradient that modified the tint of the directional light according to its angle with the horizon.
At five thirty simulated, the lampposts turned on. Not all at once: each point light had a random delay of 0 to 3 seconds that George had implemented as a detail. The cones of light of the lampposts appeared on the ground as amber pools that were invisible during the day because the sun overpowered them but that at sunset started to stand out from the surroundings, becoming more visible as the natural light dropped.
At six, the sun touched the horizon. The shadows of the buildings extended to the edge of the visible space, long and thin like fingers pointing east. The sky was orange. The lampposts were the dominant sources. The ground had two kinds of light: the amber pools of the lampposts and the darkness between them.
George stopped. He looked at the test scene: twenty cubes that were buildings, a plane that was a street, three points of light that were lampposts. It wasn't Los Santos. It didn't look like Los Santos. But the light was right. The shadows were right. The sunset of a generic street with generic cubes felt like a real sunset because the light did what light did in reality: change angle, color, intensity, and transform the same space into something different with every minute that passed.
*
Marcus saw the sunset of the cubes one Friday at four in the afternoon.
George had asked him to look at the test scene before going home. Marcus sat in front of George's monitor and looked at the twenty cubes with the sun going down and the lampposts turning on and the shadows lengthening across the ground.
He said nothing for a minute. George knew the silences: the short ones were technical evaluation, the medium ones were artistic processing, and the long ones were something Marcus didn't have words to describe.
This was a long silence.
"The light is right," Marcus said finally. "The shadows lengthen like real shadows. The color transitions like real color. The lampposts turn on with different timing like real lampposts. And it's all cubes with no texture on a plane with no detail."
"And?"
"And if the light is right with cubes, it's going to be spectacular with buildings. The light is what makes a space a space. I can paint the best textures in the world and if the light is flat they look flat. I can put mediocre textures and if the light is good they look alive." Marcus took off the glasses, cleaned them, put them on. "George, the light you just made is better than that of any game I've seen."
"It's the same math everyone uses. Three lighting components, shadow maps, attenuation by distance."
"The same math with better implementation. The edge PCF. The hybrid per-pixel and per-vertex system. The lamppost delay. They're details. But the details are the difference between right and beautiful, and what I just saw is beautiful."
George didn't respond because there was no need. Marcus had seen what George wanted him to see: that NGN was going to produce graphics that rivaled the best in the industry, by having a smarter lighting implementation than anything on the market instead of more polygons or bigger textures. The light of NGN was technical elegance applied to limited hardware by someone who knew exactly how much he could squeeze out of each processor because he had seen the future of each processor.
Marcus got up from the chair.
"I'm going to need to redesign the materials of the buildings," he said. "The textures I was planning assumed flat light. With the light you have, I need materials that react to the shadows: variation of brightness in the facades, different reflectivity for glass and concrete and metal, colors that change tone with the angle of the sun. It's more work. But the result is going to be..." Marcus searched for the word.
"Real?" George offered.
"No. Not real. Better than real. Real is what you see with your eyes. This is going to be what you remember with your memory. Memory makes sunsets more orange than they are. Memory makes shadows longer. Memory makes lampposts glow brighter. That's what the light of NGN is going to do: render Los Santos the way people remember Los Angeles, not the way Los Angeles actually looks."
It was the same philosophy Marcus had articulated for the tiles: stylized realism, the memory of a place instead of the photo. But now the philosophy applied not only to the textures. It applied to the light. And the light touched everything. Every building, every street, every vehicle, every NPC was going to be lit by a system that made the world look the way people remembered it looking, which was more beautiful than the way it actually looked.
Marcus left the office at five with a list of materials to rewrite and a glow in his eyes George recognized: the glow of an artist who has just discovered he has a new tool that changes what he can do.
George stayed in front of the test scene. The cubes. The lampposts. The sunset. The shadows lengthening as the sun went down toward a horizon that still had no palm trees or skyscrapers or the golden smog of Los Santos.
But the light was already there. And the light was the first thing.
End of Chapter 89