One outage, four root causes, each hiding behind the others

Most outages have one villain: a bad deploy, a maxed-out database, an expired certificate. The hard ones don't. They're four small problems stacked on top of each other, each one wearing a different disguise, all failing at once and pointing you in four different directions at the same time.

That's what we walked into on DIAREV, our camp-based healthcare platform running on Cloud Run. Here's how we found, and fixed, all four.

---

The symptom

Requests to the backend were failing intermittently. Not consistently enough to look like a config error, not rare enough to ignore. Some admin dashboard calls timed out. Some support-plan enrollments threw 500s. The frontend, meanwhile, was occasionally unable to reach its own backend at all.

The instinct in a moment like this is to chase the loudest error first. We didn't, because in our experience, the loudest error is rarely the real one.

Root cause #1: IAM bindings had quietly reset

The first thread we pulled was permissions. A service account binding had been reset, likely during an earlier infrastructure change, which meant certain backend calls were silently losing their authorization mid-flow. This wasn't throwing a clean "access denied"; it was surfacing as vague, inconsistent failures downstream, which is exactly what made it hard to spot first.

Fix: Re-established the IAM bindings and added them to our standard post-deploy checklist so a reset doesn't go unnoticed again.

Root cause #2: CPU throttling under concurrent load

Separately, we found the Cloud Run service was hitting CPU throttling during bursts of concurrent traffic, particularly when the admin dashboard fired off several parallel API calls at once. Throttling doesn't crash a container; it just makes everything slow and unpredictable, which looks a lot like a database problem until you check the metrics.

Fix: Tuned the Cloud Run configuration: gen2 execution environment, 1Gi memory, and a concurrency setting matched to actual traffic patterns (concurrency=80), so the service had headroom instead of ceiling-hitting under normal load.

Root cause #3: Cold starts racing Firestore connections

Related, but distinct: when the service scaled from zero, a burst of parallel admin dashboard calls would race against Firestore's connection establishment. The container was technically "up" before its database connection was actually ready, so early requests failed even though a retry a second later would have succeeded.

Fix: Set min-instances=1 so the service never fully scales to zero, eliminating the race condition for the traffic patterns that mattered most.

Root cause #4: A frontend env var pointing nowhere

The last piece was the one that looked like it should've been the first thing anyone checked, and yet it hid the longest. A frontend environment variable was pointing to a Firebase App Hosting domain that no longer resolved. Every request built from that variable failed at the DNS level, before it ever reached the backend logic we'd spent hours debugging.

Fix: Corrected the environment variable, then added a smoke test that verifies every public-facing env var actually resolves before a deploy is considered complete.

---

What made this hard

None of these four issues was individually exotic. IAM resets, throttling, cold starts, and a stale env var are all things any backend engineer has seen before. What made this incident hard was that they were overlapping: each one masking or mimicking the others. A slow request could be throttling or a cold start. A failed request could be a bad IAM binding or an unresolvable domain. You can't fix what you can't isolate, and you can't isolate four causes by treating them as one bug.

The actual fix was going root-cause by root-cause instead of symptom by symptom: reproducing each failure mode in isolation, confirming it independently, and only then trusting that the system was actually healthy, not just quiet.

The takeaway

Production incidents rarely have the courtesy of a single cause. Real infrastructure work means treating "it's fixed" as a hypothesis to test, not a feeling to trust. That discipline, chasing each thread to its actual end instead of stopping at the first plausible explanation, is what we bring to every platform we operate, at Rise Digital India.

---

Running into intermittent, hard-to-pin-down production issues? Get in touch with Rise Digital India. This is the kind of problem we like.