RAG-01
AtlasOps Governed Knowledge Platform
Permission-aware RAG with verified citations: zero permission leaks over its probe set, and cost and latency measured against real models.
Ingest versioned sources, run hybrid retrieval and reranking, produce grounded answers, enforce access controls, evaluate quality, and observe cost/latency.
Inspect it
Problem
An organisation of any size accumulates written knowledge across systems that were never designed to be read together: a wiki, a document store, a ticket tracker, a set of runbooks, a policy library, release notes, and the long tail of exported PDFs nobody owns. Two properties of that corpus create the problem. It is versioned and contradictory — the same policy exists in four revisions, three of them superseded, and nothing in the text says which is current.
Measured evidence
Retrieval recall@10, served configuration: 0.9242424242424242 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Retrieval nDCG@10, served configuration: 0.8663381881612779 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Retrieval mean reciprocal rank, served configuration: 0.859090909090909 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Citation precision: 1 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Citation recall: 1 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Cited spans that resolve to the cited source text: 1 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Correct abstention on questions the corpus cannot answer: 0.75 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Abstention on questions the corpus can answer: 0 ratio (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Permission leaks over the probe set: 0 count (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
Existence disclosures over the probe set: 0 count (synthetic)
Evaluation, development split: corpus-relevance@1.0.0, corpus-grounded-answers@1.0.0, corpus-abstention@1.0.0, corpus-permission-probe@1.0.0; corpus sha256:f05a2049d6de; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker stand-in-reranker; 1 run at commit d1b5c50
End-to-end answer latency p95, all requests (95% retrieval-cache hits): 4755.820900000001 ms (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Time to the model's first token p95, from the start of the request: 726.6957000000002 ms (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Retrieval stage latency p95, both retrievers and fusion: 564.0732000000007 ms (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Permission resolution and predicate compilation latency p95: 0.030600000012782402 ms (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Citation verification latency p95: 0.05490000000281725 ms (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Cost per answered query p50, excluding reranking: 0.0008052000000000001 usd (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Cost per answered query p95, excluding reranking: 0.002122 usd (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Ingestion embedding cost per 1,000 chunks: 0.0010000000000000002 usd-per-1k (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
End-to-end answer latency p95, retrieval-cache misses only (the slowest sample): 8211.4041 ms (synthetic)
Load profile local-openai-c4: 12 x 12th Gen Intel(R) Core(TM) i5-1245U, 16 GB, win32/x64, single process; concurrency 4; 260 requests over workload sha256:e32b75e87ef4; embedder text-embedding-3-small, generator gpt-4.1-mini, reranker none; 95% retrieval-cache hits
Supporting evidence
Ablation across the four retrieval arms
Governance report: permission probes, leak count and the prompt-injection subset
Cost and latency report from a scripted load run, with the stage breakdown
Raw span export of the load run, one trace per request
Boundary enforcement: the check result and the generated module graph
Threat model, each mitigation mapped to the test that exercises it
Limitations
- The corpus is synthetic. Every corpus in this repository — examples/corpus, the ingest and parser fixtures, the evaluation datasets — was written to exercise the code. Metrics computed over it measure the corpus at least as much as the system, and its adversarial subpopulations are the ones we thought to write rather than the ones users produce.
- English only. Chunking, stopword removal and the lexical analyser assume English. The stopword list is a conventional English one, not tuned and not language-detected. A corpus in another language needs its own list, and that is a per-connector decision the way chunking strategy is. Nothing detects a mismatch, so a non-English corpus would ingest and retrieve badly rather than loudly.
- No real user traffic. Every measurement in this repository is synthetic in telemetry's literal sense — the flag is true by default and nothing here has ever set it to false. No query in any dataset came from a person trying to get work done, and the distribution of real questions is the thing none of this has seen.
- Judged metrics carry the judge's bias. Groundedness and contradiction cannot be computed by string matching, so they need a judge, and a judge introduces a bias that can be mistaken for a quality improvement — particularly when the judge and the generator share a family.
- Real models answer, and the served configuration has no reranker. The published evaluation (docs/evidence/run-*.json) and load run (docs/measurements/load-run.json) used text-embedding-3-small and gpt-4.1-mini, priced from the dated table openai-2026-09-24.
- Reranking is bypassed in the served configuration (ADR 0011), because the only reranker is the unselected stand-in, and it made ranking worse. OpenAI publishes no rerank model (ADR 0006). With real embeddings, serving the stand-in gave nDCG@10 0.71 on development and 0.64 on held-out, against 0.87 and 0.77 with it bypassed.
- The judge is a stand-in, so the groundedness rows (supported-claim rate, contradiction rate, judge-human agreement) are not quality evidence, whatever they read. PRD 8.3 does not let a judged metric gate a release alone, and here it cannot inform one either.
- Cost is measured, with one qualification. Cost per answered query is p50 $0.00081 and p95 $0.00212, and ingestion costs $0.001 per thousand chunks — all far inside PRD 9.3. Nothing is excluded now that no reranker runs, but a selected rerank model would add its own price. The evaluation report's own cost row counts generation only; the load run's is the whole-request figure.
- The answer path is over its latency budget, and the breach is accepted, not fixed. End-to-end p95 is 4,756 ms against 3,000 (the cache-miss p95 is 8,211 ms, the slowest of thirteen). One workload question — "when should an incident be escalated and to whom" — gets a complete 563–608-token answer that takes about five seconds to generate, and its twenty repeats set the p95 alone.
- The retrieval-stage budget is breached on thirteen samples, and accepted as such. Retrieval runs only on retrieval-cache misses — 13 of 260 requests — so its p95 is the slowest single one: 564 ms against 400, almost all of it one embeddings call. The other ten misses took 161–298 ms, earlier runs peaked at 329 and 247, and the retrieval code did not change.
- Generation failures degrade, and an evaluation leaves the degraded queries out. Since ADR 0010, a model failure in generation returns the ranked passages with no prose (PRD 9.4) instead of aborting the request. It took two aborted real evaluations to get there: one transient failure in P18b, and one answer that hit the 800-token output limit in the first held-out run.
- Time to first token measures the model, not the user. Since ADR 0012 the generator streams, and the time from the start of a request to the model's first token is 727 ms at p95 over 260 requests, within PRD 9.3's 1,200. No token reaches the caller then: the answer is returned whole, after verification. Only the OpenAI set streams; the stand-ins report no first token.
- No persistent storage adapter exists. The only store profile is memory, in-process. Four components in four processes therefore cannot share a corpus, which is why the API and the evaluation runner crawl one themselves rather than reading what the worker wrote.