Skip to content
Back to blog
Engineering

Cutting p99 latency in half by deleting a cache

The cache was added to make the endpoint faster. Years later it was the reason the slowest requests were so slow. A profile, a hypothesis and a very satisfying pull request.

Tomás Beltrán

Staff Engineer, Ledger

Published
Reading time
2 min
A network switch with rows of numbered ports and blue and grey cables plugged in
Photograph via Unsplash

Our GET /balances endpoint had a healthy median of 11 milliseconds and an embarrassing p99 of nearly 400. Nothing about the median suggested a problem. The tail was where customers were feeling it, and it was bad enough that a few had started polling twice to hedge.

A cache with a past

The endpoint sat behind a read-through cache with a 30-second TTL. It had been added in 2021, when balances were computed by summing rows and the query was genuinely slow. It worked. It also acquired a family of friends: a warm-up job, a stampede guard and an invalidation hook that fired from six places.

What the profile said

We sampled slow requests and grouped them by where the time went. The breakdown surprised everyone, including the person who wrote the cache.

  1. Cache hit: about 2 ms. Fast, as expected.
  2. Cache miss, no contention: about 18 ms. Fine.
  3. Cache miss with stampede guard: 150–400 ms. Waiting requests queued behind a single refresh.

Since the ledger rewrite, balances are maintained incrementally, so the underlying read takes about 3 ms. The cache was now protecting us from a database that was faster than the cache's own miss path.

balances/get.ts (before)
async function getBalance(accountId: string) {
  const hit = await cache.get(accountId);
  if (hit) return hit;

  // The stampede guard: only one caller refreshes, the rest wait.
  return guard.run(accountId, async () => {
    const fresh = await db.balances.read(accountId); // ~3 ms
    await cache.set(accountId, fresh, { ttl: 30 });
    return fresh;
  });
}

Every cache is a bet that the thing behind it will stay slow. It's worth checking that you're still right.

Deleting it safely

We didn't just remove the cache. We ran the endpoint with the cache bypassed for 5% of traffic, compared latency distributions and correctness, and widened over a week. At 100%, p99 fell from 392 ms to 187 ms. The median moved from 11 ms to 6.

balances/get.ts (after)
async function getBalance(accountId: string) {
  return db.balances.read(accountId);
}

The lesson

We keep a list of every cache in the system with a one-line reason and a review date. This was the first one to fail review, and it won't be the last. Performance work often means adding; the best wins are the ones where you subtract.

Keep reading

All stories