Many people in clinical research care about a practical question: general-purpose large language models can answer many medical questions, but how reliable are they when the task is to design a clinical trial or systematically review clinical evidence?
These are not simply medical question-answering tasks. Designing a trial requires a model to reason about treatment arms, eligibility criteria, endpoints, statistical assumptions, and operational risks. A systematic review requires it to search broadly, apply selection criteria consistently, and synthesize evidence without losing important studies along the way. A model can produce plausible clinical prose while still failing at any one of these steps.
So our goal was not just to build another clinical language model. We wanted to connect four pieces that are often developed separately: a global clinical trial database, a set of tasks aligned with real research workflows, a training pipeline, and standardized evaluation. The result is TrialPanorama, recently published in npj Digital Medicine [1].
The main lesson from this work is straightforward: in a specialized setting such as clinical research, scaling the data foundation may matter as much as, and sometimes more than, scaling the number of model parameters.
From scattered records to a clinical research data foundation
There is already a large amount of public clinical trial information. The problem is that it is distributed across journal articles, international registries, and national platforms, each with its own schema, terminology, and reporting quality. A drug may be represented by a brand name in one source and a generic name in another. Conditions, endpoints, trial phases, biomarkers, and adverse events are described at different levels of detail. These records are public, but they are not automatically ready for model training.
So we started with the data layer.
TrialPanorama aggregates more than 1.6 million clinical trial records from fifteen global sources. About 48.7% of the records come from PubMed, 31.6% from ClinicalTrials.gov, and 19.6% from other global registries. We also collected around 30,000 systematic reviews and linked them to the clinical studies that the reviews included or excluded [1].
We then used a combination of rules, machine learning, and large language model extraction to map the sources into a common representation. Core entities such as drugs, conditions, adverse events, biomarkers, and outcome measurements were normalized against established biomedical resources including DrugBank, MedDRA, RxNorm, MeSH, and COMET.
The final resource is not just a collection of documents. It includes a relational database with tables for studies, conditions, drugs, endpoints, biomarkers, outcomes, adverse events, and results, together with a knowledge graph connecting trials, publications, systematic reviews, and standardized biomedical entities.
This distinction matters. A million raw records do not naturally produce a better clinical model. Their value comes from making the records queryable, linking related evidence, and turning the same underlying data into reproducible training and evaluation examples.
What should “clinical research capability” mean for an LLM?
Once we had the database, the next question was how to evaluate models on it. We did not want to choose a single prediction task simply because it was easy to score. Instead, we worked backward from two clinical research workflows: trial design and systematic review.
The benchmark contains five trial design and optimization tasks:
- Arm design
- Eligibility criteria design
- Endpoint selection
- Sample size estimation
- Trial completion assessment and rationalization
It also contains three tasks from the systematic review workflow:
- Study search
- Study screening
- Evidence summarization
These tasks cover several different types of capability. Some are multiple-choice decisions grounded in a trial synopsis. Study search asks the model to generate a PubMed query that retrieves the trials included in a real review. Sample size estimation requires the model to interpret group definitions and statistical assumptions before carrying out power analysis. Trial completion assessment combines prediction with a judgment about whether enrollment, safety, efficacy, or operational factors are likely to drive termination.
Using the TrialPanorama pipeline, we constructed approximately 152,000 training and testing samples across the eight tasks. We also used temporal splits, holding out more recent studies for evaluation, to reduce the risk that models were tested on examples already represented in their training data [1].
Particularly important here is that the benchmark does not define clinical research ability as “knowing medical facts.” It tests whether a model can perform specific operations inside a workflow. That makes failures easier to locate: a model may summarize provided evidence well but fail to retrieve the relevant evidence in the first place.
Where does the ground truth come from?
A reasonable question about benchmarks built from real-world data is whether the ground truth is actually defensible. In TrialPanorama, the labels are not answers written from scratch by annotators. They are recovered from source artifacts produced during clinical research: registered protocols, reported trial status, and the decisions documented in systematic reviews [1].
For arm design, eligibility criteria, and endpoint selection, the correct answer comes from the corresponding fields in the registered trial. The incorrect options are often drawn from other trials cited in the same Cochrane review. This makes them clinically related enough to be plausible rather than obviously random distractors. For trial completion, the labels come from observed registry status and reported termination reasons; those outcome fields are excluded from the model input to prevent direct leakage. For systematic review tasks, included and excluded studies are derived from review authors' documented decisions, while the evidence answer is grounded in the review conclusion and the abstracts of its included studies [1].
Sample size estimation makes the provenance especially concrete. We linked full-text protocol sections to ClinicalTrials.gov records, extracted the statistical assumptions described by the study authors, and used the trial's official target enrollment as the answer. This produced 2,143 matched examples in which the protocol text explicitly mentions the enrollment number [1]. The task therefore asks whether a model can reconstruct the sample-size decision made by the original trial team from the stated assumptions.
This design gives us traceable labels at a scale that would be difficult to reproduce with manual annotation. It also has a clear boundary: a registered endpoint or target enrollment is an observed design decision, not proof that it was the only or universally optimal decision. Review inclusion labels can also reflect judgment calls, and registry records may contain incomplete reporting. We therefore treat these labels as reproducible, workflow-grounded targets for training and comparison, rather than as unquestionable clinical truth.
Learning reasoning without hand-written reasoning traces
We then asked: if the model size stays fixed, what happens when the model is trained on data and objectives that closely match clinical research?
Starting from Llama 3 8B and Qwen3 8B, we first used supervised fine-tuning across the benchmark tasks. One limitation, however, is that the source data usually record the final decision, not the intermediate reasoning that produced it. A trial registry may contain the target enrollment and a protocol may state the statistical assumptions, but neither provides a carefully annotated, step-by-step derivation. Writing such reasoning traces manually would be expensive at this scale, and for many clinical tasks there may be more than one reasonable path to the same answer.
For tasks with automatically checkable outcomes, we can instead let the model explore candidate solution paths and score what they produce. We used reinforcement learning with verifiable rewards (RLVR) for sample size estimation and study search. For each prompt, the model generated multiple rollouts. In sample size estimation, a rollout received a performance reward when its final estimate fell within ±20% of the registered target. In study search, the reward was based on Recall@100: how many studies included in the reference systematic review were retrieved by the generated PubMed query. The training algorithm then increased the probability of rollouts that achieved better verified outcomes [1].
This setup does not require a human-written “correct” chain of thought. The model is free to explore different reasoning strategies, while the final answer or downstream retrieval result supplies the learning signal. In that sense, the ground truth supervises reasoning indirectly: it tells us which trajectories led to a useful result without prescribing every intermediate step.
There is an important limitation. A correct final answer does not prove that every intermediate statement was valid, and outcome-based reward can sometimes reinforce shortcuts. RLVR therefore provides scalable supervision for learning task-effective reasoning, not a complete audit of the reasoning process. For high-stakes use, intermediate assumptions and calculations still need external checks or expert review.
This training pipeline gave us the domain-adapted Llama3-8B-TP and Qwen3-8B-TP models. The outcome was consistent across the benchmark: the TrialPanorama-adapted 8B models outperformed generic 70B counterparts on all eight tasks, despite using roughly one-tenth as many parameters. They also outperformed GPT-4o across the evaluated tasks [1].
The interesting part is not the headline that a small model beat a large one. The comparison shows that parameter count is only one scaling dimension. The density of domain supervision, its alignment with the target workflow, and the availability of reliable feedback signals also determine what a model learns.
This does not mean model scale is unimportant, or that an 8B model is generally more capable than a 70B model. It means that general capability does not automatically transfer to every specialized research process. For clinical research, carefully constructed post-training data can unlock capabilities that are difficult to obtain from scale alone.
What did the models learn about clinical trial design?
Clinical trial design is a useful test because a plausible answer is not necessarily a good answer. The treatment arms must match the intervention and objective. Eligibility criteria need to define an appropriate population without becoming unnecessarily restrictive. Endpoints need to reflect the clinical question. Sample size depends on the design and statistical assumptions rather than on surface-level medical language.
Across arm design, eligibility criteria design, and endpoint selection, the domain-adapted models achieved the strongest overall accuracy. For example, Llama3-8B-TP reached 90.0% accuracy on arm design, while Qwen3-8B-TP reached 87.1% on eligibility criteria design. On endpoint selection, the two adapted models reached 74.3% and 78.2%, compared with less than 62.0% for the general open-source baselines [1].
Sample size estimation was more difficult. The model had to read the trial setup, identify the control and experimental groups, interpret the statistical assumptions, and estimate the target accrual. The domain-adapted models produced the lowest mean absolute errors among the evaluated systems: 356.3 for Llama3-8B-TP and 393.6 for Qwen3-8B-TP, compared with 452.1 for GPT-4o and substantially larger errors for the open-source base models [1].
For trial completion, domain adaptation also improved balanced accuracy, from the mid-50% range for generic open-source models to 74.3% for Llama3-8B-TP. Explaining why a trial might terminate remained much harder: the best rationalization accuracy was only 31.1% [1]. This gap is useful. It shows that a model may improve at predicting an outcome before it can reliably identify the underlying operational or clinical reason.
These results should not be read as evidence that a model can independently design a protocol. The benchmark necessarily simplifies real clinical development, and many decisions depend on regulatory strategy, operational constraints, and expert judgment that are not fully represented in registry data. What the results show is narrower and more useful: task-aligned training can improve several concrete components of trial design, and the remaining gaps can be measured rather than hidden behind fluent output.
Systematic review: improving the workflow before the final summary
Systematic review is another setting where evaluating only the final answer can be misleading. A model may write a convincing synthesis from the studies it sees, but the synthesis will still be incomplete if the search missed relevant trials or screening excluded them incorrectly.
We therefore evaluated three consecutive steps. In study search, the model receives a review question represented through Population, Intervention, Comparison, and Outcome (PICO) and generates a PubMed query. Qwen3-8B-TP achieved about 57% Recall@500, compared with about 17% for the base Qwen3-8B and 47% for GPT-4o. In study screening, Qwen3-8B-TP reached approximately 86% recall when applying a review's inclusion and exclusion criteria. In evidence summarization, it reached about 82% accuracy and 81% macro F1, again the strongest result among the evaluated models [1].
For me, the broader point is that domain training did more than make the model a better writer. It improved search-query generation and eligibility decisions upstream of synthesis. At the same time, 57% Recall@500 is far from complete retrieval, and missed evidence can materially change a medical conclusion. The practical direction today is therefore human–AI collaboration: use models to reduce repetitive work while retaining expert review at the decisions where errors carry the highest cost.
Scaling clinical research AI beyond parameter count
Looking back, the most important part of TrialPanorama is not a single benchmark number. It is the connection between data integration, task construction, model training, and standardized evaluation.
The first lesson is that many specialized fields do not lack raw data. They lack a process for converting raw data into forms that models can learn from and researchers can evaluate. For TrialPanorama, that process required harmonizing sources, normalizing entities, linking reviews to studies, and defining tasks from real research workflows.
The second lesson is that scaling has several dimensions. Parameter count is one. The quality and concentration of domain data, alignment between training objectives and intended use, and access to verifiable feedback are others. Our 8B results suggest that these dimensions can sometimes produce more direct gains than another increase in model size.
The third lesson is that a benchmark should be infrastructure, not a finish line. Clinical research changes over time: new trials are registered, standards of care evolve, and new designs such as adaptive and decentralized trials become more common. A useful data foundation therefore needs continual updates, new task formulations, and evaluation that remains sensitive to temporal drift and underrepresented domains.
That is how we hope TrialPanorama will develop: not as a frozen leaderboard, but as a growing foundation for asking a more demanding question of clinical AI systems: have they actually learned a new research capability, and can we verify it?
Reference
- Wang, Z. et al. Developing Large Language Models for Clinical Research Using One Million Clinical Trials. npj Digital Medicine (2026). View paper
Resources
- Paper: Developing Large Language Models for Clinical Research Using One Million Clinical Trials (npj Digital Medicine, 2026)
- Project: TrialPanorama
- Database: TrialPanorama database
- Benchmark: TrialPanorama benchmark
- Code: GitHub