TrialCurveAccuracy & methodology
We test the pipeline the API runs in two ways: on real published figures, against the statistics their authors printed, and on simulated trials, where the true patient data are known exactly. None of the test figures were used in development.
The benchmark results could not be loaded. Please refresh the page.
The production pipeline, measured
The API reads each figure with Claude Opus 5.5: it finds every survival panel and reads the curves, the numbers-at-risk table and every printed statistic, zooming into the figure wherever the full view is too small to be sure. The Guyot (2012) algorithm then reconstructs patient-level data. Where our pixel measurement of a panel agrees with the model's reading, the measured curve replaces it. Every reconstruction is then checked against the statistics printed on the figure.
Real published figures
We ran the pipeline on 87 Kaplan–Meier figures from 40 randomised trials published open access (CC BY) and harvested automatically from PubMed Central. Each whole figure went in unedited, as rendered from the article PDF (75 figures; 12 from PMC's web images where no PDF match was found). The model found 161 survival panels and the pipeline reconstructed patient-level data for 154. Each reconstruction was compared with every statistic the authors printed on the figure or in its inset table.
| Printed statistic | Checks | Within tolerance* | Median difference | 90th percentile |
|---|---|---|---|---|
| Hazard ratio | 141 | 95% | 2.1% | 6.6% |
| Median survival | 128 | 96% | 0.6% | 4.2% |
| Number of events† | 86 | 98% | 0.0% | 2.4% |
| Number of patients† | 86 | 100% | 0.0% | 0.0% |
*10% for hazard ratios, medians and events; exact for N. Hazard ratios: 82% within 5%, 95% within 10%. †Consistency checks, not independent validation: when a figure prints N and event counts, the reconstruction uses them as inputs (as the Guyot method recommends), so agreement confirms the reading is internally consistent. Hazard ratios and medians are outputs of the reconstruction and are the real test. 123 of 135 panels passed every check; the rest are flagged in the API response so an analyst knows exactly which panels to review. Printed values are rounded, and printed hazard ratios are often stratified or adjusted, while the reconstruction gives an unstratified Cox estimate, so a few percent of difference is expected even from a perfect reading. Every validated panel is published, with its checks, in the trial library.
What the remaining differences are. The figures with a flagged check were read again, independently, by a larger model (Claude Fable 5.1). It reproduced 14 of the 14 flagged differences to within 5%. When two models agree on the reconstruction, the difference comes from the figure itself: printed hazard ratios that are adjusted or stratified, medians of arms with only 3 to 13 patients, where one event moves the median, and event counts that do not match the plotted curve. The API labels a flag that a second reading reproduced, so the analyst knows where to look.
Simulated figures with known truth
The same pipeline on 80 held-out simulated figures (seeds 11000–11079), where the true patient data are known. To isolate each part, the same model readings were also scored without pixel refinement ("model only"), and the measurement engine was run alone on the same figures.
| Metric | Production | Model only | Measurement only |
|---|---|---|---|
| Hazard ratio within 5% of truth | 91.2% | 93.8% | 85.5% |
| Hazard ratio within 10% of truth | 96.2% | 96.2% | 90.8% |
| Median |log HR| error | 0.009 | 0.007 | 0.012 |
| Curve error (median of per-arm mean |ΔS|) | 0.0027 | 0.0027 | 0.0028 |
| Error in median survival (median, months) | 0.09 | 0.08 | 0.10 |
| Events per arm (median relative error) | 0.9% | 0.9% | 1.0% |
| At-risk table cells read exactly | 100.0% | 100.0% | 95.6% |
| Arm names read exactly | 95.9% | 95.9% | 88.1% |
| True HR inside reconstructed 95% CI | 100% | 100% | 100% |
How these readings were produced. The model readings for this page were made by Claude Opus 5.5 running in Claude Code, with the production instructions and output schema, the same image the API would send, and the same zoom tool (identical crops, enlargement and ruler). Readers worked blind: they saw only the figure and its caption, never our results or the printed-value checks. The pipeline downstream of the reading is exactly the code the API runs. Through the API, the reading runs at a fixed effort level with the same tool and a budget of 14 zooms.
Measurement engine: 400 simulated figures
Pixel measurement and on-device OCR, without the vision model. It refines the model's curves in production and takes over when the model is unavailable. Results on the full held-out set of simulated figures:
All figures in the held-out set, including degraded, black-and-white and table-less figures. Hazard-ratio metrics use every figure where both of the first two arms were reconstructed (n = 357).
Distributions
| Metric | n | Median | Mean | 90th pct | Max |
|---|---|---|---|---|---|
| |ΔlogHR| (per figure) | 357 | 0.0113 | 0.0839 | 0.0919 | 4.2756 |
| Median survival error (months, per arm) | 638 | 0.083 | 0.614 | 0.659 | 41.056 |
| Median survival error (relative, per arm) | 638 | 0.4% | 2.9% | 3.4% | 383.0% |
| Events error (relative, per arm) | 786 | 1.1% | 9.3% | 26.1% | 328.2% |
| Curve mean |ΔS| (per arm) | 824 | 0.0026 | 0.0145 | 0.0054 | 0.8268 |
| Curve max |ΔS| (per arm) | 824 | 0.027 | 0.056 | 0.082 | 1.000 |
| RMST error / τ (per arm) | 786 | 0.11% | 0.87% | 0.41% | 70.28% |
| Processing time (s per figure) | 398 | 11.519 | 11.484 | 15.891 | 22.708 |
n counts figures for HR metrics and arms for the per-arm metrics. Means are pulled up by a small number of failures. That is why the page leads with medians and with the share of results inside fixed tolerances.
Breakdowns
The same metrics, split by what makes a figure hard. "Processed" is the share of figures that returned a result. HR columns only count figures where a hazard ratio could be reconstructed.
Numbers-at-risk table present or absent
Without a table, IPD is reconstructed from the true N per arm (N-only Guyot).
| Subgroup | Figures | Processed | HR n | Median |ΔlogHR| | Mean |ΔlogHR| | HR within 10% | Median events error | At-risk cells exact |
|---|---|---|---|---|---|---|---|---|
| no table (N supplied) | 53 | 98% | 48 | 0.0658 | 0.0987 | 63% | 24.3% | — |
| with at-risk table | 347 | 100% | 309 | 0.0095 | 0.0816 | 94% | 0.8% | 92% |
Colour vs black & white
Black & white includes curves separated only by grey level or line style.
| Subgroup | Figures | Processed | HR n | Median |ΔlogHR| | Mean |ΔlogHR| | HR within 10% | Median events error | At-risk cells exact |
|---|---|---|---|---|---|---|---|---|
| black&white | 31 | 97% | 23 | 0.0480 | 0.6929 | 52% | 5.3% | 73% |
| colour | 369 | 100% | 334 | 0.0110 | 0.0419 | 93% | 1.0% | 94% |
Clean vs degraded images
Degraded = downscaled to 55–100% and/or JPEG quality 55–92.
| Subgroup | Figures | Processed | HR n | Median |ΔlogHR| | Mean |ΔlogHR| | HR within 10% | Median events error | At-risk cells exact |
|---|---|---|---|---|---|---|---|---|
| clean | 266 | 100% | 244 | 0.0093 | 0.0236 | 94% | 0.9% | 96% |
| degraded (downscaled/JPEG) | 134 | 99% | 113 | 0.0167 | 0.2141 | 80% | 1.7% | 85% |
Shaded confidence bands
Figures with shaded 95% CI bands around each curve.
| Subgroup | Figures | Processed | HR n | Median |ΔlogHR| | Mean |ΔlogHR| | HR within 10% | Median events error | At-risk cells exact |
|---|---|---|---|---|---|---|---|---|
| CI bands | 59 | 100% | 48 | 0.0176 | 0.2556 | 69% | 1.8% | 81% |
| no CI bands | 341 | 99% | 309 | 0.0104 | 0.0572 | 93% | 1.0% | 95% |
Number of arms
| Subgroup | Figures | Processed | HR n | Median |ΔlogHR| | Mean |ΔlogHR| | HR within 10% | Median events error | At-risk cells exact |
|---|---|---|---|---|---|---|---|---|
| 2 arms | 354 | 99% | 312 | 0.0115 | 0.0725 | 90% | 1.1% | 93% |
| 3 arms | 46 | 100% | 45 | 0.0096 | 0.1626 | 84% | 1.2% | 92% |
Methodology
Simulated trials
- Each figure shows 2 arms (about 88% of figures) or 3 arms. Arms have 40–450 patients each, with Weibull survival (shape 0.75–1.6), control-arm median 5–36 months and hazard ratios 0.4–1.05.
- Censoring combines staggered accrual (8–36 months), administrative censoring at the end of follow-up (6–40 months after accrual closes) and random dropout.
- Ground truth for each figure: the exact individual patient data, the exact Kaplan–Meier curves, and the Cox hazard ratio (Efron ties) of arm 2 vs arm 1 fitted to the true data.
Rendering
- Six palettes modelled on common journal and software styles: tab10, ggplot/survminer, NEJM-, Lancet- and JCO-like colours, and black & white with solid, dashed or dotted lines.
- Five fonts (DejaVu Sans, Arial, Helvetica, Times New Roman, Verdana) at 8–12 pt. Line widths 0.9–2.4 pt.
- Censor marks (none,
+or|), shaded 95% confidence bands, legends in four positions, HR/p-value annotations inside the plot, light grids, titles. - Survival axis as a proportion or a percentage, several tick spacings. The time axis is sometimes truncated before the end of follow-up, as journals often do.
- Numbers-at-risk tables under ~85% of figures, sometimes with coloured rows, thinned to every second or third time point when columns would collide.
- A "hard" third of figures is downscaled to 55–100% of its size and/or JPEG-compressed at quality 55–92, simulating screenshots of PDFs.
Held-out protocol
The engine was developed on separate seed ranges. The benchmark reported here uses seeds 7000–7399, generated after development and never used for tuning or threshold selection. Each figure goes through the same code path as a standard API request, with no hints. For figures without an at-risk table, reconstruction uses the true N per arm, as a user would supply it from the paper.
Metric definitions
- HR within 5% / 10%
- |log(HRreconstructed / HRtrue)| below log(1.05) or log(1.10), comparing arm 2 vs arm 1 on the reconstructed IPD with the same comparison on the true IPD.
- |ΔlogHR|
- Absolute difference in log hazard ratio. For small values it is roughly the relative HR error (0.01 ≈ 1%).
- True HR in 95% CI
- Whether the true HR lies inside the Wald 95% confidence interval computed from the reconstructed IPD.
- Curve mean |ΔS|
- Per arm: mean absolute difference between the reconstructed and true survival curves over 600 time points from 0 to the end of the plotted curve (an integrated absolute error normalised by time).
- Median survival error
- Per arm: absolute difference in months between the KM median of the reconstructed and true data, where both are reached within the plotted range.
- Events error
- Per arm: |eventsreconstructed − eventstrue| / eventstrue, counting true events within the plotted time range.
- RMST error
- Per arm: |RMSTreconstructed − RMSTtrue| / τ, where τ is the end of the plotted follow-up.
- At-risk cells exact / misread
- Share of numbers-at-risk cells, within the plotted range, returned with exactly the true value or with a wrong value. The remainder were not returned.
- Arm names
- Share of arms whose returned name matches the true name, ignoring case, spaces and punctuation.
- Processed
- Share of figures for which the engine returned a result instead of an error.
Known limits
Where results are least reliable, and what the API tells you when it happens.
The production pipeline
- Adjusted or stratified hazard ratios. Many trials print a stratified or covariate-adjusted HR. The reconstruction gives an unstratified Cox estimate, so differences of 5–15% are expected even from a perfect reading. This is the most common reason a hazard-ratio check fails on real figures.
- Very small arms. With fewer than about 15 patients, one event moves the median by months, and a printed median may not be reproduced within 10%.
- Figures that disagree with themselves. Printed event counts sometimes come from a different data cut than the curve. The reconstruction follows the curve and the at-risk table, and the check says so.
- Image quality. The model can only read what is visible. PubMed Central's web images are often about 700 px wide; the article PDF usually carries the figure at 2–5× that resolution, so export from the PDF or zoom to 200% before taking a screenshot.
- No numbers-at-risk table. Reconstruction from the initial N alone is much less precise (63% of HRs within 10% versus 94% with a table, in the simulated benchmark). Pass the total events when the paper reports them.
- Cumulative-incidence plots with competing risks. These are reconstructed on the 1 − cumulative incidence scale, which is approximate when competing events are frequent. The result carries a warning.
- Vector PDFs. Figures must be supplied as images (PNG or JPEG) for now.
When a check against a printed value fails, the figure is read again by a larger model. If the second reading reproduces the difference, the check is labelled accordingly, which points to the figure rather than to a reading error.
The measurement engine (fallback)
Without the vision model, results come from pixel measurement and on-device OCR. It handles single panels only, and is weakest on:
- Small or compressed screenshots: 80% of HRs within 10% versus 94% on clean figures.
- Black-and-white figures where same-coloured curves cross: 52% within 10% versus 93% for colour.
- Shaded confidence bands: 69% within 10% with bands versus 93% without.
Which model reads curves best?
We re-test whenever a new model ships. Each model below read the whole figure on its own, curves included, with no pixel measurement, and the same Guyot reconstruction ran on its reading. Scores are against the exact truth on 24 held-out simulated figures (seeds 11000–11023; the xhigh row uses the first 12). The first four rows are single API calls without the zoom tool.
| Model (reading alone) | Curve error | Median-survival error (months) | HR within 5% | Median |log HR| error | Cost / figure |
|---|---|---|---|---|---|
| Claude Opus 5 | 0.0155 | 0.72 | 86% | 0.033 | ~$0.06 |
| Claude Fable 5.1 | 0.0075 | 0.42 | 95% | 0.014 | ~$0.14 |
| Claude Opus 5.5 (high effort) | 0.0057 | 0.29 | 95% | 0.011 | ~$0.05 |
| Claude Opus 5.5 (xhigh effort) | 0.0060 | 0.21 | 100% | 0.008 | ~$0.10 |
| Claude Opus 5.5 with the zoom tool (production reader) | 0.0026 | 0.07 | 92% | 0.006 | ~$0.10–0.30 |
Curve error is the median of each arm's mean absolute difference in survival probability. With the zoom tool the model crops and enlarges the full-resolution figure wherever it needs to (the at-risk table, steps, crossings, the 50% line). That roughly halved the curve error and cut the median-survival error by three quarters, which matters more than effort level or a larger model. The zoom row was produced through Claude Code with the production instructions; API cost depends on how many zooms a figure needs.
How the pipeline got here
Earlier in 2026 the balance was the other way round. On the same 80 simulated figures, Claude Opus 5 reading curves on its own put 72.5% of hazard ratios within 5% of the truth, against 85.5% for pixel measurement, and its curve error was five times larger. So the API used a hybrid: the model read the text, and the curves were measured in pixels. Opus 5.5 with a zoom tool now reads curves as precisely as the measurement engine, and reads text far better, so the model's reading comes first. Pixel measurement checks each curve, replaces it where the two agree closely, and takes over when the model is unavailable.
How this compares
Published evaluations use different figure sets, mixes of difficulty and metric definitions, so this comparison is indicative only.
| Source | Setting | Reported result | TrialCurve |
|---|---|---|---|
| Saluja et al. 2019 | Manual digitisation + Guyot, 118 curves from 55 RCTs | Mean HR error 0.0094 | Median |log HR| error 0.009 (simulated); median difference from printed HRs 2.1% (real figures) |
| Kim et al. 2025 | Reconstruction of 58 published HRs | 84% of HRs within 5% | 82% of 141 printed HRs within 5% (real figures); 91% of true HRs (simulated) |
| KM-GPT (2025) | Automated pipeline, 540 synthetic plots | Median absolute S(t) error 0.005 | Median per-arm mean |ΔS| 0.0027 (a related but not identical statistic) |
| ISPOR 2025 poster | GPT-4o alone vs a dedicated computer-vision pipeline | Median survival error ±1.9 vs ±0.6 months | Median error in median survival 0.09 months (simulated); printed medians reproduced to a median 0.6% (real figures) |
How to cite
TrialCurve (2026). Automated reconstruction of individual patient data from published Kaplan–Meier figures, engine v0.1.0. https://trialcurve.com/accuracy. Benchmark data: benchmark.json (CC BY 4.0).
Reproducing the benchmark
The figure generator and scoring harness are part of the TrialCurve codebase. Enterprise customers can get them on request for validation audits, together with the exact seed list, so that every number on this page can be regenerated.