Global Resilience in 3 Steps: Understanding Flag Delivery Multi-Region Failover

Published August 24, 2026

Flag delivery uses Redis as its primary cache to ensure that flag evaluation never depends on the backend being reachable. Redis is fast, but it’s regionally isolated. Each region only holds what’s been requested there. That design makes requests very fast, but it means a regional traffic shift starts cold. Every payload is a cache miss in the receiving region, because that region’s cache has no history of serving that traffic.

Last October, a us-east-1 disruption made that cold-start gap a real problem. That’s one of the reasons we decided to add a global cache tier that’s warm everywhere, all the time. Now we don’t have to think twice about shifting traffic. We know the caches can support the new requests without putting stress on the backend.

The solution: S3

The fix is adding S3 into the caching layers. After this project, we have an in-memory cache, then regional Redis, then S3, then the backend. When Redis misses, S3 answers, the layered read path backfills Redis and the in-process cache on the way up, and the backend is never contacted.

As an intern, I worked with two senior software engineers on implementing this, and with their guidance we significantly strengthened flag delivery infrastructure. Let’s walk through some of the components of the project that I focused on.

Step 1: Infrastructure

The obvious cheap version is one bucket. S3 buckets are globally reachable, so why not point all three regions at us-east-1 and go home? Because cross-region S3, even over the AWS backbone, still pays the latency tax of the physical distance between regions.

We measured it: a single remote bucket is 9x slower on reads and 16x slower on writes than a local one, and from Australia it pushes writes past a second. This cache has to stay current with every flag change, so that wasn’t going to cut it.

The solution: one bucket per region, bidirectional cross-region replication between them, and a Multi-Region Access Point in front, so a node asks one global endpoint and gets routed to the nearest replica over Global Accelerator’s edge network. That cut reads 51% and writes 64% against the single-bucket approach.

Step 2: Keep the cache warm

A cache tier is only a failover tier if the data is there before you need it. Three paths put it there.

The fast path is a flag change. The backend notifies a node, that node updates memory and Redis as it always did, and the same saver now writes S3 alongside Redis. Lag is seconds.

The slow path is the backstop for a notification that was delayed or dropped entirely. A background poller compares local state against the backend and writes through to both remote caches when it finds itself behind.

Hydration covers the awkward remainder: payloads sitting in Redis that are perfectly current and haven’t changed since we turned S3 on. Nothing in the first two paths will ever touch them. So one in a thousand Redis hits sends its payload to a hydrator, a fleet-wide claim with a 12-hour time-to-live (TTL) means a payload gets hydrated at most once per half-day no matter how many nodes noticed it, and each unit of work gets 10 seconds.

Step 3: Move traffic

All of the above only matters if we can move traffic when circumstances require it. With the S3 tier keeping caches warm across regions, the data is already where it needs to be. Next, we have to address shifting connections safely and predictably under pressure.

Failover is deliberately human-initiated. An operator invokes a CLI that is the thinnest possible layer. It validates input, resolves your AWS identity, picks a coordinator region, and starts a Step Functions execution.

The drain itself is graduated: 1%, 10%, 25%, 50%, 100%. Between every step, a health gate checks the receiving region before advancing. Past 50%, propagation waits get longer to give downstream systems time to absorb the shift. Two traffic levers move underneath: Route 53 weights for one service class, Global Accelerator dials for another, and the state machine coordinates both so they advance in lockstep. If a health gate fails at any step, the execution halts and can automatically roll back to the last known-good distribution.

Final thoughts

Putting it all together, flag delivery now has a four-tier read path: in-memory, regional Redis, S3, and the backend. S3 acts as a globally consistent layer that any region can fall through to at any time. Per-region buckets with cross-region replication and a Multi-Region Access Point keep that tier fast, and three warming paths (write-through on flag changes, a reconciliation poller, and sampled hydration) keep it populated so the data is already where it needs to be, whenever it’s needed. On top of that, a graduated traffic-shifting system with server-side orchestration and health gates at every step gives us the flexibility to redistribute traffic across regions safely and on our own terms.

This was my intern project, and I’m grateful I got to work on something that sits directly in the critical path of flag delivery. Huge thanks to the senior engineers who guided the design, gave me room to own the implementation, and made sure I understood not just what we were building but why each piece mattered.

If you’re interested in more details or want to connect, reach out!

• Email: ssriya3018@gmail.com
• LinkedIn: linkedin.com/in/sriyasth