Shadow Reads: Verifying a Zero-Downtime Datastore Migration Under Live Traffic
Language Mismatch Disclaimer: Please be aware that the language of this article may not match the language settings of your browser or device.
Do you want to read articles in English instead ?
Swapping the datastore under a live product is not really a code problem. Writing the reads against a new store is the easy part. The hard part is trust: how do you know the new store returns the same answer the old one does, for every shape of data your users actually hit, before a single user depends on it?
The temptation is to test in staging, eyeball a few records, and flip the switch on a Friday you will regret. Staging never has the weird data. Staging never has the row someone hand-edited in 2019, the null that should not exist, the encoding that only one importer ever produced. Production has all of it, and production is exactly where you cannot afford to be wrong.
The pattern that gets me out of this bind is shadow reads. Read from the new store in parallel with the old one, on live traffic, but only ever serve the old answer. Compare the two in the background. Let real traffic tell you where the new store diverges, before you promote it. This post is the general method, with Laravel you can lift into your own migration.
The stance: the old store stays the source of truth
The whole method rests on one rule. Until a module is promoted, the old store answers every request. The new store is read, compared, and discarded. Nothing a user sees depends on the new store being correct yet.
That rule is what makes the migration safe to run for weeks. A divergence is a log line, not an incident. A bug in the new read path returns a wrong value into a comparison, never into a response. You are gathering evidence, not taking risk.
Four moves get you from here to a finished cutover:
- Backfill the new store in the background.
- Read in shadow and compare under real traffic.
- Promote one module at a time behind a config flag.
- Keep the old store as a fallback until the evidence is overwhelming.
Step 1: backfill in the background
Before you can compare anything, the new store needs the data. Backfill is a chunked, resumable, idempotent job. Chunked so it never loads the world into memory. Resumable so a crash costs you a batch, not the run. Idempotent so re-running it is always safe.
namespace App\Jobs;
use App\Models\Article;
use App\Contracts\DocumentStore;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
class BackfillArticlesJob implements ShouldQueue
{
use Dispatchable, Queueable;
public function __construct(public int $afterId = 0) {}
public function handle(DocumentStore $store): void
{
Article::query()
->where('id', '>', $this->afterId)
->orderBy('id')
->chunkById(500, function ($articles) use ($store) {
foreach ($articles as $article) {
$store->put("article::{$article->id}", $article->toDocument());
}
});
}
}
Keep the writes upserting by a stable key, so the job is safe to run again the moment you find a bug in toDocument(). You will find a bug in toDocument(). Backfill is where you discover that your new document shape forgot a field that only three records use.
Step 2: read in shadow and compare
This is the heart of it. Every time the application reads through the old store, it also reads through the new one, compares them off the hot path, and returns the old answer untouched.
I wrap this in a single comparator so the read sites stay clean and every module goes through the same instrumentation.
namespace App\Support\Migration;
use Closure;
use Throwable;
use Illuminate\Support\Facades\Log;
class ShadowReadComparator
{
/**
* Serve $source of truth. Read $candidate in the background,
* compare, and record divergence. Never let the candidate
* affect the returned value.
*/
public function read(string $module, string $key, Closure $source, Closure $candidate): mixed
{
$truth = $source();
rescue(function () use ($module, $key, $truth, $candidate) {
$shadow = $candidate();
if (! $this->matches($truth, $shadow)) {
Log::channel('shadow')->warning('shadow_divergence', [
'module' => $module,
'key' => $key,
'diff' => $this->diff($truth, $shadow),
]);
}
}, report: false);
return $truth;
}
private function matches(mixed $truth, mixed $shadow): bool
{
return $this->normalize($truth) === $this->normalize($shadow);
}
private function normalize(mixed $value): string
{
$array = $value instanceof \Illuminate\Contracts\Support\Arrayable
? $value->toArray()
: (array) $value;
ksort($array);
return json_encode($array);
}
private function diff(mixed $truth, mixed $shadow): array
{
return [
'only_in_truth' => array_diff_key((array) $truth, (array) $shadow),
'only_in_shadow' => array_diff_key((array) $shadow, (array) $truth),
];
}
}
Two details carry the safety. First, rescue(..., report: false): if the candidate read throws, the request does not care. The old store already produced the answer. A broken new read path degrades to zero shadow coverage, never to a failed request. Second, normalize() sorts keys and serializes, so you compare values rather than incidental ordering. Naive == will drown you in false divergences from key order and loose type juggling.
The read site stays readable:
$article = app(ShadowReadComparator::class)->read(
module: 'articles',
key: "article::{$id}",
source: fn () => Article::findOrFail($id)->toDocument(),
candidate: fn () => $documentStore->get("article::{$id}"),
);
Now let it run. Watch the shadow log channel. Every divergence is a real question about your migration that production is answering for free: a field you forgot to map, a default the old code applied in PHP that the new store does not, a record that never backfilled. You fix the mapping, re-run the backfill for the affected keys, and watch the divergence rate fall toward zero. When a module goes quiet for long enough across the traffic shapes you care about, it has earned promotion.
Step 3: promote behind a config flag
Promotion flips which store is the source of truth for one module. It is a config change, not a deploy of new logic, so it rolls forward and back in seconds.
// config/migration.php
return [
'primary' => [
'articles' => env('MIGRATION_PRIMARY_ARTICLES', 'legacy'),
'authors' => env('MIGRATION_PRIMARY_AUTHORS', 'legacy'),
'search' => env('MIGRATION_PRIMARY_SEARCH', 'legacy'),
],
];
The read site asks the config which way the shadow points, and swaps source for candidate when the module is promoted. The other store keeps getting read, so the comparison keeps running in reverse and you keep your evidence.
public function readArticle(int $id): array
{
$promoted = config('migration.primary.articles') === 'document';
$legacy = fn () => Article::findOrFail($id)->toDocument();
$document = fn () => $this->documentStore->get("article::{$id}");
return app(ShadowReadComparator::class)->read(
module: 'articles',
key: "article::{$id}",
source: $promoted ? $document : $legacy,
candidate: $promoted ? $legacy : $document,
);
}
Rollback is one environment variable. If a promoted module misbehaves, MIGRATION_PRIMARY_ARTICLES=legacy puts the old store back in charge without a deploy, without a rebuild, without a war room. That reversibility is the entire point of the flag.
Step 4: promote module by module, never all at once
Do not promote the whole datastore in one move. Promote articles. Live with it. Promote authors a week later. Then search. Each module is an independently reversible slice with its own evidence, so a problem in one never forces a rollback of the others, and the blast radius of any single flip is one module wide.
Module by module also keeps the writes honest. Only once a module's reads are promoted and stable do you narrow its writes toward the new store and let the old one go. Reads first, writes last, one slice at a time.
I used exactly this method migrating a Fortune-50 platform's content store to Couchbase, cutting over live regulated traffic with no maintenance window and no lost records: the full case study is here. Shadow reads were what let each module pass change control on its own evidence instead of a promise.
Why this beats the alternatives
A bulk cutover freezes writes, dumps, loads, and flips. Simple to reason about, and it gives you no safe rollback the moment traffic lands on an unverified store. Dual-writing from day one keeps both stores current but couples every write path to a store whose correctness you have not proven yet, so a write failure on the new side threatens the live one.
Shadow reads invert the risk. You verify correctness on real traffic while the old store still owns every answer, and you only shift ownership once the evidence is in. The cost is a comparison read on the hot path and a log channel to watch. The return is that you promote on data, not on nerve. That is the trade I make on every serious migration, and it is the discipline I bring to this kind of work as a solutions architect: move the risky thing in slices, each one measured and reversible, so the scary cutover becomes a series of boring config flips.
