PulseSearch — real-time
CDC search + RAG

The problem

Most "AI search" demos are thin wrappers on a static index. Real products aren't.

In production, the data never stops moving and the index has to keep up: writes land in a database of record, and search has to reflect them within seconds — not overnight — while staying consistent under retries, restarts and out-of-order delivery. On top of that, users increasingly expect semantic relevance and a question-answering assistant that is grounded in the freshest data, not hallucinated.

I wanted to build the whole spine of that system end to end — streaming ingestion, CDC, relevance engineering, real-time fan-out and retrieval-augmented generation — as one coherent, observable pipeline, and prove it with metrics rather than adjectives.

The approach

PulseSearch treats MySQL as the system of record and streams every row change into Elasticsearch, then serves both hybrid search and grounded RAG over that live index. It ingests a real firehose — Wikipedia's live edit stream — so the pipeline is genuinely continuous rather than a seeded demo.

• CDC, not polling — Debezium tails the MySQL binlog and publishes ordered change events to Kafka (Redpanda); a worker embeds each change and upserts it into Elasticsearch. Wikipedia summary enrichment runs async after indexing, so it doesn't inflate commit→searchable latency.
• Hybrid retrieval with RRF — BM25 and dense-vector kNN result lists are fused in application code with Reciprocal Rank Fusion, keeping it free of paid Elastic tiers and making ranking explicit and tunable. A labeled live eval script keeps relevance measurable.
• Grounded RAG — Groq's free-tier LLM (llama-3.1-8b-instant) answers strictly from retrieved context, cites sources with timestamps, reports the freshest source, and refuses when retrieval is empty — with an extractive fallback if the API is unavailable.
• Observable by design — end-to-end "commit → searchable" latency, throughput, and DLQ rate are Prometheus metrics surfaced as p50/p95/p99 in Grafana.
• Ship the ops story too — Compose stays the one-command data-plane demo; the app services (ingest, worker, api, web) also run on Kubernetes with readiness/liveness probes, a ServiceAccount, and Ingress. CI runs Ruff, Bandit, unit tests, an Elasticsearch upsert→BM25 smoke, web lint/build, and kubeconform, then publishes images to GHCR on main.

Architecture

In action

Screenshots from a live local run — the product UI recruiters can relate to, plus the Grafana panels that make the pipeline's SLOs and failure modes observable rather than asserted.

The hard parts

Effectively-once indexing on at-least-once delivery

Kafka offsets are committed only after a batch is durably written to Elasticsearch. Upserts are version-guarded using Debezium's ts_ms as an external monotonic version (version_type=external_gte), so replays and out-of-order delivery can never regress a document to an older state.

Replayable backfill & poison-message isolation

Because the change history lives in Kafka, rebuilding the index is a replay from the log, not a database-crushing reindex. Unparseable or repeatedly failing messages are routed to a dead-letter topic instead of wedging the whole stream.

Ranking that's free and explicit

Native RRF retrievers require a paid Elastic tier, so I fuse BM25 and kNN result lists in code (k = 60). Every hit exposes its BM25 and kNN ranks, making relevance transparent and tunable rather than a black box.

Keeping the CDC path fast

Wikipedia page summaries are useful for search and RAG, but fetching them on the critical path inflated sync latency. Enrichment now runs after the document is searchable, so commit→index stays lean while summaries still land asynchronously.

Grounding the assistant

The RAG endpoint answers only from retrieved context, cites sources with timestamps, reports the freshest source, and returns grounded: false when retrieval is empty. Generation uses Groq (llama-3.1-8b-instant, free tier) behind an LLMClient protocol — with an extractive fallback if the API is unavailable, so the endpoint stays demonstrable.

Engineering

The codebase is deliberately layered and dependency-inverted so every concern is swappable and testable: a shared kernel for config / models / embeddings / the ES repository / metrics; Strategy interfaces behind Protocols for firehose sources, embeddings and the LLM client; a Repository layer that keeps SQL and query DSL out of business logic; and an anti-corruption layer that translates Debezium envelopes into clean domain events. Unit suites cover the shared kernel, the Debezium adapter and the RRF fusion logic with no external services required; CI also runs a live Elasticsearch smoke (upsert → BM25 find) plus Ruff and Bandit on every PR.

CI/CD & Kubernetes

Recruiters ask for delivery, not only demos. PulseSearch wires that in:

• GitHub Actions — Ruff lint/format, Bandit (medium+), pip-audit (advisory), unit tests, ES smoke, Next.js lint/build, kustomize + kubeconform, then Docker Buildx for api / worker / ingest / web.
• CD on main — the same images publish to ghcr.io/shubhamsinha1010/pulsesearch-cdc-rag/{api,worker,ingest,web} tagged :latest and :sha-<short>.
• Kubernetes app tier — Deployments with resource limits, HTTP readiness/liveness (API uses /health/ready → 503 when ES is down; workers probe /metrics), a non-automounting ServiceAccount, and host-based Ingress (pulsesearch.local / api.pulsesearch.local). A GHCR kustomize overlay points at the published images.

Outcome

Measured on a local Docker/Colima laptop demo (~38k Wikipedia change docs) — not cloud prod. BM25-only search is much cheaper (p50 ~27 ms); hybrid cost is mostly query embedding.

Stack