SCRAG

Self-correcting retrieval-augmented generation: hybrid search, an LLM guardrail that filters context, and answers that are graded and corrected by an evaluator.

Connecting…GitHub

The pipeline

  1. Step 1

    Hybrid retrieval

    BM25 keyword ranking and dense embeddings (all-MiniLM-L6-v2 via ONNX) are fused with Reciprocal Rank Fusion, so exact identifiers and paraphrases are both found.

  2. Step 2

    Guardrail

    An LLM scores every retrieved chunk for relevance and drops off-topic context before it can mislead the answer. If it rejects everything, raw retrieval is used.

  3. Step 3

    Generate

    The generator answers only from the kept context.

  4. Step 4

    Evaluate and self-correct

    An evaluator grades how well the answer is supported. Below the threshold, its critique goes back to the generator for another attempt; the best-scoring attempt is returned.

The Agent tab adds a bounded planner: it picks tools (document search, a safe calculator, read-only SQL over a catalog of the knowledge base), runs them in parallel, writes an answer that cites its evidence, and verifies it, within hard limits on steps, LLM calls and time.

Measured retrieval quality

60 labelled questions over the 16 sample documents, at k = 5. Every number was measured (2026-11-10) and can be reproduced with scrag eval.

RetrievalRecall@5Hit@5MRRnDCG@5
Dense only0.9670.9830.9430.939
Hybrid (default)0.9921.0000.9660.965

Hybrid search helps most on questions with exact terms (lexical MRR 0.883 → 0.938) at the same latency (under 3 ms). Two hard questions stay in the set as known weak spots. Methodology and full results

What the comparison showed

  • On 8 answerable questions, a single hybrid generation, the self-correcting pipeline and the agent all scored perfectly. Self-correction used about 2.6 times the tokens without changing an outcome on this clean corpus.
  • Every wrong answer we measured came from a missed chunk, so retrieval depth mattered more: raising the demo's top-k from 3 to 5 let hybrid search answer the two hardest questions.
  • Asked 4 questions the documents can't answer, including ones the model knows from training, the system declined every time instead of hallucinating.

Small samples (indicative), measured 2026-11-10 with one LLM judge plus keyword checks. Full results and caveats

Runs on free tiers

  • API: FastAPI in Docker on Render Free (512 MB, sleeps when idle; the first request wakes it).
  • Database: Neon Free Postgres with pgvector, for sessions, uploads, history and jobs.
  • Web UI: a static export on a CDN. No secrets in the browser.
  • LLM: Groq's free tier, protected by per-visitor rate limits and a daily budget, with a second free model as a fallback if the first is rate limited or retired.

Privacy and limits

  • No accounts and no tracking cookies. A session is an anonymous token kept in your browser.
  • To share the daily AI budget fairly, a keyed hash of your IP address (never the address itself) is kept with today's count and deleted after a day.
  • Sessions, uploads and history are deleted after 24 hours, or at once with “Delete all my data”.
  • Only extracted text is stored, and other sessions can never read it.
  • Questions and document text are sent to the LLM provider: don't use personal data.

Source code, architecture decisions and the roadmap are on GitHub (MIT licensed).