A RAG system over the oncology literature
Search by concept and get back the paper and the passage where the concept appears, with character offsets into the source. Built for the question an R&D team actually asks, “what is known about X?”, which is answered by a set of papers, not one.
- Documents indexed
- 3,166 live
- Retrievable passages
- 180,850 live
SELECT count(*) FROM documentsSELECT count(*) FROM chunks WHERE kind='passage'The literature this searches
3,166 peer-reviewed oncology papers from PubMed and PMC, every one held as verbatim full text rather than an abstract, split into 180,850 individually retrievable passages. Abstract-only records were removed: an abstract is already a summary, so there is no passage inside it to point at.
The set was grown along its own citation graph. Starting from MeSH-seeded oncology searches, the papers those papers cite were ingested too, screened against NLM's MeSH tree so the corpus stays oncology rather than drifting into general molecular biology. Screening the top 9,000 candidates kept 5,389 of them, so two in five fail: they are cited by oncology work without being oncology work, and would have drifted the corpus off its subject while every retrieval metric continued to look healthy.
Below, each point is one paper, positioned by the same embedding retrieval uses, so two papers sit together for the reason a query would return both. Regions are named by the MeSH major topics most distinctive to each, measured by log-odds against the whole corpus, which is why they read “Cytokine Release Syndrome” rather than “Humans”. Drag to rotate, click a region to list its papers.
The axes are not labelled because they have no individual meaning. They are the three directions in which the 14 cluster centres are furthest apart, each one a weighted blend of all 192 embedding dimensions. No axis is “immunotherapy” or “year”. Only relative distance carries information, which is why the regions are named and the axes are not.
3,166 papers positioned by their retrieval embeddings, 14 clusters labelled by the MeSH major topics most distinctive to each. These three axes carry 18% of the variance in a 192-dimensional space and 60% of the separation between clusters, and the clustering itself accounts for 23% of total variance. Both views are linear projections, so distances stay comparable. the “separation” view rotates to the plane where the clusters are furthest apart, which is a camera angle chosen after seeing the labels, not invented structure. t-SNE would separate the clusters far more cleanly and the separation would not mean anything.
Research areas in the corpus
Select a region to see its papers. Clusters are found in the embedding space, then named by NLM's human indexing.
How it works
A query takes six steps from typed text to a cited passage. The one design rule everything else defers to is that a result must be able to point back at its source, because a claim you cannot check is a claim a generator can invent.
- 01
Ingest
PubMed + PMC Open Access
Papers arrive as verbatim full text, never abstracts. Bibliographies are stripped before indexing: they are a median 19% of an article's characters and they match queries lexically while containing no findings.
- 02
Chunk
~900 characters, offsets preserved
Each passage keeps (doc_id, section, start_char, end_char) back to its source. That provenance is the product: a retrieval change that improves ranking but loses it is a regression, not a trade.
- 03
Index
BM25 + dense vectors in Postgres
A lexical index over the passage text and a pgvector column of embeddings, in the same database as the documents, so grants and authors can be joined against retrieval results.
- 04
Retrieve
two arms, fused by rank
The query runs against BM25 and the dense index independently. Reciprocal Rank Fusion combines the two ranked lists rather than their scores, because BM25 scores are unbounded while cosine similarity lives in [-1, 1] and a naive sum is dominated by whichever has the larger numeric range.
- 05
Aggregate
passages to documents
A document scores as its single best passage. That is a real choice: rewarding one decisive passage rather than a paper that is vaguely relevant throughout.
- 06
Serve
the passage, with its offsets
Results carry the matched clause and the characters it sits at, so a claim can be checked against the paper instead of trusted.
- R
Read
the paper, with its figures
Opening a result shows the whole article in reading order, plus its figures: 9,549 images served from NCBI's own Open Access store, with the publisher's verbatim caption. Figures are NOT part of retrieval — captions are already indexed as body text, so their effect on ranking is unmeasured and they are held out of both arms until it is.
The dense arm, and why the model choice was not obvious
The lexical arm matches words. The dense arm is supposed to match meaning, so that a search for “EGFR TKI resistance” finds a paper that only ever says “osimertinib”. Which embedding model does that best for oncology is a measurable question, and the answer was not the one we expected.
MedCPT is NCBI's biomedical retriever, trained contrastively on 255 million real (query, clicked-article) pairs harvested from PubMed itself. It is a model of what biomedical researchers actually search for and which paper they then opened. It uses two separate encoders, one for queries and one for articles, because a two-word query and a 900-character passage are not the same kind of object. Against it we ran text-embedding-3-small, a general model, at the same 768 dimensions specifically so that a MedCPT win could not be confused with simply having four times the vector capacity.
MedCPT turned out to be a trade, not an upgrade. It is a better topical matcher and a worse pinpointer: it won literature-review coverage decisively and significantly regressed find-the-source. Click data encodes “this article is about what you asked”, not “this is the exact sentence you want”, and the smoothing that lifts topical recall costs exact attribution.
What worked was refusing to choose. Running both dense models alongside BM25 and fusing three ranked lists beat either model alone on both query types, and cancelled the regression entirely. Arms that are individually competent and wrong about different queries are exactly when rank fusion pays.
| dense arm | review coverage | find the source | deployable |
|---|---|---|---|
| MedCPT alone | +0.0261 | −0.0166 | needs a hosted GPU endpoint |
| general model, 768-dim | +0.0016 | +0.0128 | one API call |
| all three, fused | +0.0305 | +0.0321 | both of the above |
Measured on the dev split, both gains significant at p < 0.0001. The fused configuration is not shipped: two controls that would attribute its gain have not run, two of the four query types were not evaluated, and the locked test split is unspent.
What it runs on, and why
Each row names the choice, what it was chosen instead of, and the constraint or measurement that decided it. Several of these exist only because the obvious option was tried first and measured worse.
Store
Neon Postgres + pgvector
Passages, embeddings, documents, grants and MeSH live in ONE database, so a retrieval result can be joined against funding and indexing without a second system. HNSW over cosine, because the vectors are L2-normalised and IVFFlat needs a training step this corpus does not justify.
instead of a dedicated vector DB — which would put the metadata a query needs on the other side of a network hop
Retrieval
BM25 + dense, fused by Reciprocal Rank Fusion
RRF combines RANKS, not scores. BM25 is unbounded and cosine lives in [-1,1], so a naive weighted sum is dominated by whichever has the larger numeric range. Both arms run in SQL in one round trip.
instead of score normalisation, which needs a calibration set that does not exist here
Embeddings
text-embedding-3-small at 192 dimensions
Matryoshka truncation: the same model serves 192 or 768 dims, so vector width is a measurable knob rather than a model migration. 192 keeps the whole index inside a serverless function's reach.
instead of MedCPT as the dense arm — measured WORSE on find-the-source, and it needs torch (~2 GB), which does not fit a Vercel function
Reranking
MedCPT cross-encoder over the fused top 50
A bi-encoder embeds query and passage separately and cannot represent an interaction between them: it knows a passage is ABOUT a topic, not whether it REPORTS the finding. Largest measured gain in the project.
instead of an LLM reranker at ~$0.0017/query, which made depth a budget decision and a full evaluation sweep cost real money
Figures & tables
JATS markup; images served from NCBI's own S3
PMC publishes structured XML, so figure captions, image URLs and table GRIDS are already stated by the publisher. Nothing is OCR'd and no image is re-hosted. Tables keep their row and column headers, so a hazard ratio does not become a decontextualised number.
instead of page-layout detection — which would re-derive from pixels what the markup already states, and PubLayNet's own training labels came from this same XML
Serving
Next.js on Vercel, Python functions for the API
The retrieval path is Python because the harness is; the same code answers a query in production and in the evaluation loop, so the thing measured is the thing served.
instead of a bundled snapshot artifact — which served a corpus the site no longer had
Evaluation
Citation-mined labels, four strata, a locked test split
Labels are FOUND, not written: a sentence citing a paper is a domain expert describing it after reading it. Nothing was annotated for this project, so the benchmark cannot be tuned toward the answers.
instead of hand-judged relevance — which does not scale and grades our own homework
Figures and tables
Opening any result shows the whole paper with its 9,549 figures and 3,682 tables. Images are served from NCBI's own Open Access store rather than copied, and captions are the publisher's verbatim text. Tables keep their row and column headers, so a hazard ratio arrives with the label that gives it meaning instead of as a loose number in a stream of cells.
What is deliberately missing is the reading of the plots. A Kaplan-Meier curve, a dose-response curve and a forest plot all encode their result positionally — the answer is where the lines sit relative to each other, and nothing in the file says which arm did better. So the honest question is whether a vision model can recover that, and it was measured rather than assumed.
| figure type | n | sees the image | caption only | gain from looking |
|---|---|---|---|---|
| Kaplan-Meier | 58 | 9% | 0% | +9% |
| ROC curve | 15 | 0% | 0% | 0% |
| Forest plot | 11 | 0% | 0% | 0% |
| Dose-response | 6 | 0% | 0% | 0% |
A vision model was asked to recover the exact value the paper's own authors quoted from each figure, scored twice: once seeing the image, once seeing only the caption. 6% against 0%. The gap is what looking buys, and it is small enough that indexing those numbers would be indexing a guess. Forest plots scored zero despite printing their effect sizes as text, which points at resolution rather than reasoning. So figures are shown, and their values are not asserted.
Design decisions
The four questions a reader is most likely to think are oversights.
Why are figures shown but not searched?
Captions are ALREADY indexed — NCBI inlines them into the plain text, at a measured median of 100% token coverage. So the text a figure carries is already retrievable, and adding figure rows to the index would change the corpus every candidate has been measured against. They are stored, displayed, and excluded from both retrieval arms by an explicit SQL predicate until that effect is measured.
Why isn't a vision model reading the plots?
Because the failure is silent. A model that writes "median OS 18.9 months" where the truth is 22.4 means a query for 18.9 retrieves the WRONG figure confidently and a query for 22.4 misses the right one — and the reader sees the real caption and image, so the error is invisible. It lives in the ranking layer, where nothing is displayed and nothing is checked. The rule is: prefer transcription over inference for anything that enters the index.
Why a keyword list for figure types, when the notes argue against regexes?
Because the publisher already named the type: a caption saying "Kaplan-Meier" is the paper stating a fact, and matching that word is recognition, not inference. A semantic classifier was built for the half it misses and MEASURED AS UNUSABLE — 80% precision only at 3.3% coverage. It is not applied. What did work was noticing 36.5% of figures match SEVERAL types, so the fix was to stop discarding what the keyword list already computed.
Why does nothing here ship on a good-looking number?
One pre-registered gate metric per query type, Holm-corrected across the candidates tried in a round, with every other metric acting as an uncorrected veto. A change ships only if it improves at least one query type and worsens none. Five rounds have produced far more refutations than promotions, which is the point.
Using it
Search a concept, get the passage
Type what you would say to a colleague: a mechanism, a drug, a variant. Results are papers, each carrying the exact passage that matched and the character offsets it sits at, so you can check the claim rather than trust it.
EGFR C797S resistance to osimertinib →Compare papers on the same dimensions
Switch to Compare and the same query returns a table: papers down the side, technical dimensions across the top: cohort, assay, endpoint, effect size. Each filled cell cites the passage it came from.
CAR-T persistence in solid tumors →Open any cell to read the passage it came from
A comparison table you cannot audit is one you should not trust. Every filled cell opens the verbatim passage at the character offsets it was drawn from, so the claim in the grid can be checked against the paper rather than believed.
How it is measured
Four query types, because they are different jobs and a single average hides a regression in the one that matters least often and costs most when wrong.
| query type | shape | judged by | scored on |
|---|---|---|---|
| synthesis | “what is known about X” | review authors | recall@20 · coverage of the set |
| e.g.Antigen-positive relapse. in Mechanisms of resistance to CAR T cell therapytry it | |||
| concept | 2-word MeSH term | NLM indexers | success@5 · was it on screen |
| e.g.Cytokine Release Syndrometry it | |||
| identifier | bare gene, drug, cell line or trial | citing author | success@1 · exact lookup |
| e.g.EGFR T790Mtry it | |||
| claim | 28-word sentence | citing author | MRR · find the source |
| e.g.Blocking antibodies against PD-1 or PD-L1 have demonstrated substantial clinical activity in patients with metastatic melanoma, renal cell carcinoma, non-small cell lung cancer, and other tumors.try it | |||
Labels are found, not written. A citing sentence is a domain expert's description of the work it cites; a review section heading is an R&D question whose cited papers are the answer set; MeSH major topics are NLM's human indexing. Nothing was annotated for this project, and no model chose its own training signal.
What each number means for a RAG answer
Retrieval quality is the ceiling on generation quality. A model can only be as accurate as the passages handed to it, and it cannot know what it was never shown, so a retrieval metric that flatters itself produces a system that is confidently wrong. Each metric below is described by the failure it predicts.
recall@20gate: synthesisOf the papers that belong in the answer, how many are in the top 20?
A literature-review answer is bounded by the set the generator sees. A paper missing from the context is a claim the model cannot make and will not know it failed to make. This is the only metric here that measures an absence.
success@5gate: conceptDid at least one right paper reach the first screen?
Short topical queries are typed dozens of times a day while scoping. If the right paper is at rank 12 it is not in the context window and, for the reader, does not exist.
success@1gate: identifierWas the very first result the right one?
For an exact lookup the top hit is often the only one read. Returning a paper about a DIFFERENT variant is worse than returning nothing, because the mistake is invisible in a fluent answer.
mrrgate: claimHow far down the list is the first correct answer?
Proxy for how much irrelevant context the generator wades through before reaching signal. Rank 1 and rank 8 both count as a hit at k=10, and they are not the same prompt.
ndcg@10secondaryGraded, position-weighted quality of the whole top 10.
The top-k IS the context window, so scoring it as a unit rather than as a hit or miss is the closest single number to what the generator receives.
success@10 / @20secondaryDoes the answer survive at deeper cutoffs?
Catches a change that sharpens the head while pushing answers out of the tail. A reranker can look excellent at k=1 and lose documents at k=20.
unjudged@10diagnostic, never a targetWhat fraction of returned documents has nobody judged?
The honesty valve. If a change raises this, its apparent gain may only be that it returns documents the pool never assessed. Optimising it directly would mean preferring documents that happen to have been judged.
bprefreported when definedRank quality counting only JUDGED non-relevant documents.
Robust to incomplete judgments, which is the normal condition here. It returns nothing below 10 judged negatives rather than averaging noise into a verdict.
What the benchmark can and cannot see
The minimum detectable effect is the smallest change each stratum could distinguish from noise, at 80% power. Anything below it is invisible by construction, so a result there says nothing about the idea being tested. Publishing it next to the sample size is what separates a negative result from a blind one.
| stratum | queries | judgments | weight | gate metric | smallest visible effect |
|---|---|---|---|---|---|
| synthesis | 2,179 | 10,368 | 0.35 | recall@20 | 0.0100sees 0.02 |
| concept | 533 | 8,758 | 0.30 | success@5 | 0.0544blind below |
| identifier | 1,853 | 6,171 | 0.20 | success@1 | 0.0218blind below |
| claim | 7,056 | 7,408 | 0.15 | mrr | 0.0087sees 0.02 |
What stops these numbers from flattering themselves
Each of these exists because the corresponding failure actually happened here, which is why they are rules rather than intentions.
Per-stratum gating
Four query types scored separately; no pooled mean.
An aggregate rises while exact-identifier lookup collapses. Round 1 found the optimal lexical weight points in OPPOSITE directions for short and long queries; a pooled mean would have averaged that into 'no change' and discarded it.
Pareto dominance, not a weighted score
Ships only if better on at least one stratum and worse on none.
Measured, not hypothetical: MedCPT wins the weighted composite by 3x and is refused, because its coverage gain is paid for with a significant regression on find-the-source. Those are different jobs for the same user, and the weights trading them off were chosen by us rather than measured.
Pre-registered predictions
Each candidate states its stratum, metric, direction and minimum effect before it runs.
Run enough candidates against enough metrics and something looks significant. Writing the prediction down first is what makes a result capable of being wrong, and an unpredicted win is recorded as a hypothesis for next round, not as success.
Holm across candidates
One gate metric per stratum, corrected over the candidates tried.
The earlier gate corrected across 5 to 8 metrics that are deterministic functions of a single rank. That controls nothing and inflated the detectable effect by about 25%, on a harness already short of power.
Regression veto, uncorrected
Any significant drop on any secondary metric refuses the change.
A false positive here means refusing a change, which is the safe direction. Making a safety check harder to trip is backwards.
NO_EFFECT detection
Byte-identical rankings are reported as a wiring bug, not a negative result.
Three candidates in this project were structurally incapable of moving the metric they were judged on. Each time the loop was one step from recording a confident negative about an idea that never ran.
Minimum detectable effect
Published per stratum, next to the result.
'No significant change' and 'this experiment cannot see a change that size' look identical in a table and mean opposite things. Blindness that reads as rigour is the most dangerous failure a harness has.
Locked test split
Candidates only ever touch dev; test is spent once.
The real defence against fitting the benchmark across many rounds. Nothing on this page has spent it.
The citing paper is excluded from its own results
assert_source_excluded() raises rather than warns.
The source contains the query verbatim, so leaving it in would measure string equality and score near-perfectly. A convention you can forget is not a guard.
A raw term-frequency floor
A ~20-line scorer with no IDF and no length normalisation.
Random and popularity baselines are flattering. The honest question is how much the system beats the simplest thing that could work.
A change ships only if it is a Pareto improvement: better on at least one query type and worse on none. A weighted average would let a gain on common queries pay for a regression on exact lookup, where returning the wrong variant is worse than returning nothing because the error is invisible in the results.
| system | ndcg@10 | recall@10 | mrr | map | unjudged@10 |
|---|---|---|---|---|---|
| bm25 | 0.4513 | 0.6058 | 0.4063 | 0.4007 | 0.9372 |
| openai | 0.4813 | 0.6335 | 0.4370 | 0.4315 | 0.9341 |
| hybrid-openai | 0.5262 | 0.6806 | 0.4809 | 0.4754 | 0.9294 |
These are lower bounds, not absolute quality. 93%-94% of the documents each system returned have never been judged by anyone, because the labels come from citations rather than from an exhaustive pool. The COMPARISON between systems holds - the unjudged rate is near-identical across all three, so none is being flattered - but a single number here is not this system's retrieval quality.