258MB of JSON, and the query parameter that fixed it

The app was timing out on login. Not for everybody. For the tenants who’d been using it longest, which is the detail that gave it away eventually, though not for a day or so.

There’s a bootstrap endpoint that the client calls once on startup to get everything it needs in one round trip. Companies, users, sites, settings, the lot. One request, one response, no waterfall of a dozen calls before the app can render. It’s a decent pattern and I’d still use it.

The response was 258MB.

How

Employee photos. Stored as base64 strings on the user records, and the bootstrap endpoint returned user records, so the bootstrap endpoint returned every photograph of every employee in that tenant. Base64 adds about a third on top of the raw bytes, and nobody had thought about what a few hundred staff photos would weigh once they were inlined into a JSON document.

In development I had four test users, two of which had no photo. The response was small enough that I never looked at it. It worked, so I moved on.

Production had tenants with hundreds of employees who’d all uploaded a photo, because we’d asked them to, because the photo is genuinely useful in the interface. The endpoint was doing exactly what I’d told it to.

The 502s were nginx giving up on a response that took longer to serialise and send than anything upstream was prepared to wait for. The error pointed at the proxy, so that’s where I looked first. Waste of an afternoon.

The fix, and the fix I didn’t do

?lean=1. The bootstrap endpoint takes a flag that strips the heavy fields and returns the shape without the payload. Photos get fetched separately, per record, when something actually needs to display one.

That cut the response by 99.1%.

It took about twenty minutes, which is roughly the ratio I keep running into: days to find, minutes to fix.

The right answer is not to store images as base64 in the database at all. They should be files with URLs pointing at them, served by nginx, cached by the browser, never touching the application at all. I know that. I didn’t do it, because it’s a data migration against live tenants and a change to every write path that touches a photo, and the site was down while I was thinking about it.

So lean=1 is a tourniquet. It’s in the roadmap to do properly, and it has been in the roadmap for a while now, which is honest if not flattering.

What I actually changed about how I work

I now look at the size of responses in development, even when they’re fast. Especially when they’re fast. A local response is quick because the data is tiny and the network is a loopback interface, and neither of those things is true anywhere it matters.

The test data being small is the whole problem. Four users with two photos will never show you this. The only reason the bug had a shape I could see was that it hit the oldest tenants first, and the oldest tenants are the ones with the most rows.