Near real-time
CDC search sync
Overnight full reindexes meant search could be up to a day stale. I replaced them with a change-data-capture pipeline that streams MySQL writes into Elasticsearch in under a minute — continuously, idempotently, and safe to replay.
The problem
Search was only as fresh as last night's batch job.
The product served search over a very large MySQL store, and the index was rebuilt by an overnight full reindex. That meant a price change, a new product or a corrected attribute could take hours — sometimes most of a day — to show up in results. The reindex was also expensive: it hammered the database, took a long window to run, and an intraday correction meant either waiting or triggering another costly rebuild.
What the platform actually needed was continuous sync: when a row changes in MySQL, that change should be reflected in search seconds later, without reprocessing everything else.
The approach
I moved from batch to a streaming change-data-capture pipeline. Instead of polling or dumping tables, the pipeline reads the database's binary log, turns each committed change into an event, and applies it to Elasticsearch.
- Debezium tails the MySQL binlog and emits an ordered stream of row-level change events — inserts, updates and deletes — without adding load to application queries.
- Kafka buffers those events per table, decoupling capture speed from indexing speed and giving the system a durable, replayable log.
- Idempotent consumers transform events and upsert into Elasticsearch keyed by document id, so re-delivering the same event is harmless — the basis for safe retries and exactly-once effect.
- Retries + dead-letter handling keep a single poison message from stalling the stream, and a replayable backfill can rebuild any slice of the index from the log.
Architecture
Committed row changes flow from binlog to search in seconds; the log makes backfills a replay, not a rebuild.
Key decisions
Capture the log, don't poll the tables
Reading the binlog with Debezium gives an ordered, low-overhead stream of exactly what changed, in commit order. Polling would have added query load and still missed deletes and rapid successive updates.
Make writes idempotent
Every consumer upserts by document id, so redelivery — which will happen with at-least-once messaging — converges to the same result. That single property is what makes retries, restarts and backfills safe instead of scary.
Treat the log as the recovery tool
Because Kafka retains the change stream, rebuilding part of the index becomes a replay from an offset rather than another full, database-crushing reindex.