Bug postmortem (Patreon)

I released the multiclass archetypes yesterday, and with them a massive bug that I somehow hadn't noticed before: it was giving people the wrong pages!

The problem is fixed now. For those with the appetite, here are some technical details on how I broke it, and how I fixed it.

Getting to the bottom of it

Several people reported the problem at around the same time, but somehow I'd never seen it myself until after it went live.

The bug itself was particularly frustrating because it seemed to make no sense. The resulting character sheet showed every sign of having the right pieces selected - for example, it would say "Barbarian" on the cover page, use the Barbarian icon, and even use the Barbarian's custom font - even while displaying the Rogue's pages. It didn't even seem to do this all the time - just sometimes!

I started at the top, checking whether the website was passing the right information to the server, whether the server was receiving that information intact, whether it was turning it into the right set of units (ancestry, class, options etc), then to the right list of pages, etc. It stayed baffling - all those tests reinforced that the same information was coming through.

The website is split into two parts: the character sheets library and the website service. 

• The library provides an API that takes simple JSON requests and returns full HTML character sheets.
• The website service accepts requests from a browser and calls the library's API. It also uses Puppeteer to create PDFs from the HTML character sheets.

The breakthrough came when I realised that, every time I restarted the server, it was the first request that would stick, and get copied to subsequent requests. That meant it had to be sticky state of some sort: the first person makes a request, and the system hangs onto that information. 

Knowing how to accurately reproduce the problem was immensely valuable. I quickly eliminated the website as the source of the problem, since I could reproduce it just by generating two test files in a row without using the website at all. I narrowed in on the Document section, which is where the request is turned into a tree of elements representing the pages before being rendered.

How character sheets are made

The character sheets are built out of modular pieces I call units. Here's the complete list of units it might use to build one character:

core, base, base/character, theme/adventurer, option/cover, option/permission, option/reference, option/build, option/minis, option/pfs, ancestry/gnome, background/noble, class/rogue, rogue/racket/ruffian, option/inventory/half, feat/diehard

Units are written in YAML, not JavaScript. They can also each have their own Sass stylesheet and their own fonts and images. Units can also pull in other units - for example, the sorcerer and wizard both require the unit for a spellbook page.

The library turns the list of units into a tree of elements, then renders those as HTML. Some of those elements are large, like a whole page or a two-column layout; others are small, like a line of text, a field or a single number. Some of the elements in the tree are special, in that they get transformed into other elements, such as an elements that sorts its contents in alphabetical order. Chief among these are zones, which are elements that allow units to insert elements based on the zone name.

The way the process works is:

• It starts with a base document that contains nothing much, just a zone called '@pages'.
•  Each unit contributes elements to zones in the document, including the zone called '@pages'.
• It walks through the whole tree replacing any transformational element with its results. Since a transformational element may turn into other transformational elements, it keeps going until there are only ordinary elements left.
• Then it walks through the tree again, this time turning each element into HTML.

So somehow it was incorporating some parts of the units - the font, icons, names and other things - but keeping the pages from previous requests.

Then I saw it.

The bug

After all that, here's the bug and its fix.

That's right. One line. I replaced a shallow copy with a deep copy.

How this happened

Something I added for the archetypes was the ability for two half-pages to be merged into a single full page. To do this I added a transformational element above the @pages zone, whose job it was to take a list of pages, and return a slightly altered list of pages. That meant the base document was slightly more complex than before - it didn't just contain a zone, it contained an element that contained a zone.

That meant the first request replaced the zone with the resulting list of pages, setting that as the contents of the transformational zone, inside the base document. It was changing something that should never have been changed. A shallow clone wasn't enough to protect us from the side-effects of previous requests.

The solution was to use a deep copy. Start with a whole fresh document every time, as it should have been all along.