Kurate
Research quality assurance at portfolio scalePortfolio review already measures impact and productivity. Now it can measure whether the science holds up.
NIH's portfolio analytics capture productivity, translation and impact well. Methodological rigor — whether a funded study used a valid comparator, held to its prespecified endpoints, and drew conclusions its design could support — is a distinct axis, and one no portfolio review has been able to measure at scale, because appraising it took expert-hours per study. When Kurate graded a corpus of trials drawn from higher-quality domains, the spread across that axis was wide.
Even in a sample drawn from higher-quality domains, grades span the full range — the same wide spread found across the published literature generally, not a property of any one funder's choices. What's new isn't the spread; it's that it's now measurable. Once it is, a funder can see which work clears the bar for confident interpretation (A–C) and direct more of the portfolio toward it.
Kurate's premise doesn't rest on our numbers alone. A large, independent metascience literature has documented that questionable research practices are common — outcomes switched between registration and publication, selective reporting, underpowered designs, p-hacking and HARKing — leaving a public record in which a study can look authoritative and still not support its own conclusions. Public and philanthropic funding flows into that record. Kurate turns decades of this evidence into a check that runs at portfolio scale.
Kurate is built by researchers who publish in statistics, research methodology and metascience — including in Nature, NeurIPS and ICLR. Members of the team have separately built the peer-reviewed precursors to Kurate's core method — automated comparison of trial registrations against published results. That background shapes how the tool handles fragile claims, uncertain comparisons and noisy literature.
Research in causal inference, deep generative modelling and multimodal ML; 50+ peer-reviewed publications including ICLR and NeurIPS.
Two doctorates — Engineering (Vision, Speech & Signal Processing), University of Surrey; and Applied Mathematics for the Human & Social Sciences, University of Lausanne. Affiliations: University of Lausanne, University of Surrey, and the Sense Center for Innovation and Research.
Contributes to Kurate's evidence-evaluation methodology and clinical rubric development; 50+ peer-reviewed publications, including three metascience papers in Nature.
Expertise in research-integrity assessment and LLM-workflow evaluation; author of published registration-versus-report comparison tools.
How Kurate works
Kurate reconstructs each study's intended design from its own paper trail, then measures the published report against it — the work a methodologist and a statistician would do together, applied uniformly at scale. Two documents in; one auditable verdict out.
One rubric, applied identically to every study — reproducible, auditable, and fast enough to run across a whole portfolio.
Where Kurate fits
NIH already scores rigor at review and measures influence after publication. The one question neither answers: did the funded work deliver the rigor it proposed? Kurate is built for exactly that gap — complementing the tools already in place, not replacing them.
Is the proposed science rigorous?
The 2025 Simplified Review Framework scores Factor 2 — Rigor & Feasibility as one of its two numerical criteria, and the Rigor & Reproducibility policy requires applicants to address experimental design, prior-research rigor and key-resource authentication. Assessed on the proposal, before any data exists.
Is the output influential?
The Office of Portfolio Analysis measures it with iCite and the Relative Citation Ratio — field- and time-normalized citation impact, benchmarked to the median NIH paper. A measure of influence, not of method.
Did the funded work deliver the rigor it proposed?
No portfolio-scale measure closes this loop today. Kurate compares each published study against its own pre-registration, across the whole portfolio — the delivered-rigor counterpart to the rigor NIH already scores at review, and reputation-blind by construction.
The closest existing check — ClinicalTrials.gov results reporting under the FDAAA Final Rule, with NIH's complementary policy extending it to every NIH-funded trial — already requires that results, protocols and analysis plans be filed, and screens them for internal errors. What it does not do is verify that the reported endpoints and analyses match what was prespecified. That comparison is exactly what Kurate performs — automatically, across the whole portfolio.
What it lets NIH do
Across roughly 50,000 grants a year, Kurate grades every study on one rubric in a single pass — turning rigor from something you could only sample by hand into a portfolio-wide layer alongside the analytics your team already runs. What that puts within reach:
See which programs and calls actually deliver rigor
Rank programs, funding mechanisms and calls by the methodological quality of the work they produce — so NIH can direct more funding toward the ones that consistently deliver, and give the ones that lag the evidence to sharpen their guidance or review criteria. The same lens resolves to the investigator level where that's useful — surfacing where rigor support would help most, not a blacklist. Grounded in method, not reputation, and defensible because every grade is auditable.
Track portfolio quality over time
Watch methodological quality across an institute, a funding stream, or a decade — and test whether a given call raises or lowers the rigor of the work it produces.
Strengthen the mechanisms that need it
See which funding mechanisms and review formats produce the most rigorous work — and where adjusted guidance would raise quality the most.
Report defensible rigor metrics
Give leadership, Congress, and the public a reproducible measure of research quality that complements impact metrics instead of competing with them.
See a field's quality evolve
Track whether rigor in a domain is improving or eroding over years — the kind of question that shapes where a new call or initiative should be aimed.
How to read the rigor
- Registration-anchored. Quality is judged against the study's own pre-data-collection commitments, not a reviewer's taste.
- GRADE-like, without pool contamination. Comparable in spirit to a Cochrane/GRADE appraisal, but dealbreaker studies are excluded rather than downweighted — so one flawed study can't quietly dilute a synthesis.
- Cross-domain in one pass. Clinical, methodological and statistical checks applied uniformly — precisely where human review is scarce, slow, and inconsistent.
- Reproducible & auditable. The same rubric, applied the same way, every time — a property no distributed panel of reviewers can guarantee, and the basis for defensible portfolio metrics.
- Scope-honest. Strongest in medication, treatment and RCT-relevant domains, where the registration-versus-report comparison is most decisive.
Start with your own portfolio
Don't take an extrapolation — take a reading. We'll run a representative slice of NIH-funded work through Kurate and return a graded, program-level view: where your most rigorous science already sits, and where directing more funding would raise the portfolio's quality the most.
About the figures. The grade distribution is Kurate's own result on a corpus drawn from higher-quality research domains; it is presented as a floor, not a random portfolio sample, and does not yet establish how NIH-funded output is distributed — a pilot on a representative slice would. The NIH budget ($47.2B, FY2026) and grant-volume figures are from public appropriations and NIH reporting. NIH mechanisms referenced — the 2025 Simplified Review Framework (Factor 2: Rigor & Feasibility), the Rigor & Reproducibility policy, iCite / the Relative Citation Ratio, and ClinicalTrials.gov results reporting (FDAAA Final Rule, 42 CFR Part 11) — are per NIH OER, the Office of Portfolio Analysis, and NLM public documentation. The average R01-equivalent award (~$664K/year, FY2025) is from NIH extramural reporting; the savings floor scales that figure by an illustratively small share of the portfolio and is a lower-bound sketch, not a projection — a pilot would establish the real figure. Kurate is decision-support for research quality assessment and does not replace clinical, regulatory, or institutional review.
Selected related publications
Methodological and meta-scientific work associated with the Kurate team.
- Vowels, M. J. (2023). Misspecification and unreliable interpretations in psychology and social science. Psychological Methods, 28(3), 507–526. DOI
- Vowels, M. J., Vowels, L. M., & Wood, N. D. (2023). Spectral and cross-spectral analysis: A tutorial for psychologists and social scientists. Psychological Methods, 28(3), 631–650. DOI
- Vowels, M. J. (2023). Prespecification of structure for the optimization of data collection and analysis. Collabra: Psychology, 9(1), Article 71300. DOI
- Vowels, M. J. (2024). Trying to outrun causality with machine learning: Limitations of model explainability techniques for exploratory research. Psychological Methods. DOI
- Vowels, M. J. (2025). A causal research pipeline and tutorial for psychologists and social scientists. Psychological Methods. DOI
- Vowels, M. J. (2024). Typical yet unlikely and normally abnormal: The intuition behind high-dimensional statistics. Statistics, Politics and Policy, 15(1), 87–113. DOI
- Aczel, B., Szaszi, B., Clelland, H. T., Kovacs, M., Holzmeister, F., et al. (2026). Investigating the analytical robustness of the social and behavioural sciences. Nature, 652(8108), 135–142. DOI
- Higgins, W. C., Clarke, B., Elson, M., & Cummins, J. (2026). Recommendations for incorporating LLMs into psychological research: A commentary on Austin and colleagues (2026). PsyArXiv. DOI
- Ahnström, L., Bruckner, T., Aspromonti, D. A., Caquelin, L., Cummins, J., et al. (2026). TrialScout links published results to trial registrations using a large language model. medRxiv. DOI
- Elson, M., Hussey, I., Clarke, B., Norwood, S. F., Grinschgl, S., Arslan, R. C., et al. (2026). Against anonymising meta-scientific data. PsyArXiv. DOI
- Cummins, J., Clarke, B., Hussey, I., & Elson, M. (2026). RegCheck: A tool for automating comparisons between study registrations and papers. arXiv. DOI
- Miske, O., Abatayo, A. L., Daley, M., Dirzo, M., Fox, N., Haber, N., Hahn, K. M., et al. (2026). Investigating the reproducibility of the social and behavioural sciences. Nature, 652(8108), 126–134. DOI
- Cummins, J. (2025, September 1). Psychology needs… an AI revolution. The Psychologist. Article
- Röseler, L., Kaiser, L., Doetsch, C., Klett, N., Seida, C., Schütz, A., Aczel, B., et al. (2024). The Replication Database: Documenting the replicability of psychological science. Journal of Open Psychology Data, 12(1), Article 8. DOI
- Tierney, W., Hardy, J. H., III, Ebersole, C. R., Leavitt, K., Viganola, D., Clemente, E. G., Gordon, M., Dreber, A., Johannesson, M., Pfeiffer, T., Hiring Decisions Forecasting Collaboration, & Uhlmann, E. L. (2020). Creative destruction in science. Organizational Behavior and Human Decision Processes, 161, 291–309. DOI
- Van Dessel, P., Cummins, J., Hughes, S. J., Kasran, S., Cathelyn, F., & Moran Yorovich, T. (2020). Reflecting on twenty-five years of research using implicit measures: Recommendations for their future use. Social Cognition, 38(Supplement), S223–S242. DOI