
An AI-assisted audit of 100 papers from the top five economics journals
We reviewed 100 randomly drawn 2026 papers from the top five economics journals. One in ten has a serious error bearing on an important result.
Summary
- Refine's AI review pipeline examined 100 papers drawn at random from those published in the top five economics journals in the first nine months of 2026. It raised 5,421 issues; an independent AI audit judged 77% of them fully correct.
- After second-pass AI checks and expert human review, 10 of the 100 papers (10%) contain at least one serious issue bearing on an important result: a misstated headline number, a missing or inconsistent supporting result, or a proof that fails as written.
- These are careful papers that passed rigorous peer review.
1. What was done in our audit
We drew 100 papers uniformly at random from all articles published in the top five economics journals during the first nine months of 2026. Every paper had passed peer review.
Refine's review pipeline processed each paper using a harness built on frontier AI models to identify conceptual, logical and factual errors. It reads deeply, checks derivations, numbers and internal consistency, tests whether claims follow from the evidence, and ties each issue to a specific passage.
Two verification stages followed: an independent, high-compute AI audit of every issue, and an expert reading of those most likely to be serious.
2. What the review found
The pipeline raised 5,421 issues, about 54 per paper. An independent AI model then read each paper and evaluated every issue blind to the pipeline's rankings, recording its type, correctness, centrality to the paper and how much it weakens the claim.
Accuracy. The audit judged 77% of the 5,421 issues fully correct, 22% partly correct (the underlying point is real but overstated) and 1.1% incorrect after the high-compute review. A second, different AI model re-checked the 196 issues that appeared potentially serious, judging 180 fully correct, 15 partly correct and 1 incorrect.
Types of issue. Among the 4,158 issues assessed as fully correct:
- Nearly half (46%) concern interpretation: an abstract that overstates the evidence, an unstated condition required for a result, or an important but unstated alternative explanation.
- Just under a third (29%) are objective, verifiable errors: a faulty derivation step, a number in the text that contradicts a table, or an internal inconsistency.
- The remainder concern presentation (15%) or suggest further work, such as additional robustness checks (10%).

Figure 1. What the 4,158 confirmed issues are about.
Most objective errors are local; only 8.9% bear on a major result. Objective errors account for half of all fully correct issues in theory papers (50%) and nearly half in econometric-methods papers (48%), where formal proofs provide concrete steps to verify. In empirical papers, interpretation dominates (55%).
3. How serious
The audit sorted papers by the most serious issue each contains.
| Step | Papers |
|---|---|
| Papers reviewed | 100 |
| At least one correct objective error bearing on a main result and weakening it at least somewhat | 59 |
| At least one fully correct objective error that substantially weakens a major point (strict bar, second AI model) | 12 |
| At least one issue judged serious by a human expert rater | 10 |

Figure 2. Papers at each level of seriousness.
The audit models applied the highest severity category, "substantially weakens", sparingly: the first model assigned it to 61 of the 5,421 issues. The drop from 59 papers to 12 reflects how demanding that threshold is, not that the excluded issues are wrong.
An expert rater then evaluated 49 issues across 24 papers, each with a write-up checked against the source paper, classifying every issue as serious, not serious or unsure. The 49 issues came from two sets: the 15 issues in the 12 strict-bar papers, plus 34 issues in 12 further papers that the pipeline had labelled "critical". The rater judged 12 of the 49 issues serious, spanning 10 papers.
A "serious" rating does not mean a paper's main conclusion is false; in several of the ten cases, the stated result likely holds, but the published proof or headline number does not.
4. The ten serious cases
The cases are described in general terms, without identifying papers.
Headline numbers misstated (4 papers)
A1. A quantitative paper decomposes the welfare cost of a firm-level distortion into two components, its dispersion across firms and its average level, and reports the dispersion share as a headline result. The counterfactual intended to eliminate dispersion alone sets every firm to a common value. Under the paper's own model and calibration, every active firm lies above that value, so the counterfactual lowers the average level as well. The reported share, and other outputs from the same counterfactual, therefore conflate both effects. The main estimates are unaffected.
A2. A headline effect reported in the introduction is roughly fifty times larger than the paper's own estimates imply. In the model, the estimated coefficient behind that figure enters multiplied by a small, separately estimated elasticity. Counterfactual results relying on the same parameter are affected as well; the paper's other main findings are not.
A3. A structural paper's welfare conclusions depend on a parameter defined by the model as the ratio of two coefficients on the same scale. Before estimation, the authors standardise the two underlying variables separately and take the ratio of the standardised coefficients. That ratio equals the model's parameter only if both variables share the same standard deviation, which the paper never establishes. The welfare comparisons rely on this value and must be recomputed; correcting it could reverse one welfare conclusion.
A4. An empirical paper reports its headline effect in percentage points, whereas the estimated coefficients measure the share of a baseline gap closed. Converted using the paper's own average gap, the effect is 8 to 10 times smaller than stated. The underlying estimates are sound; the stated magnitude is not.
Supporting result missing or inconsistent (2 papers)
B1. A table panel meant to present evidence for a second main claim duplicates the preceding panel verbatim: every coefficient, standard error and sample size. The estimates supporting that claim are never reported. The paper's primary results are unaffected.
B2. A model aggregates output across units whose productivity follows a distribution governed by a dispersion parameter. Aggregate output is finite only when that parameter exceeds one. The paper's own estimates place it below one, rendering aggregate output infinite. Both the aggregate formula and the policy simulations built on it require revision.
Proofs or assumptions that fail as written (4 papers)
C1. A theory paper extends its main theorems (confined to a binary setting) with two further results whose proofs apply a bound established only for the simpler special case. A counterexample satisfying the paper's conditions violates the claimed bound. The two extensions are unproved as written, though not disproved; the extension beyond the binary setting is important for the paper's significance.
C2. A paper combining experimental evidence with a learning model uses the model to explain why decision-makers try an option that starts at a disadvantage. As written, one assumption makes that disadvantage so severe that the option yields a lower expected payoff than an equally good alternative under any admissible belief, even after a successful trial. A decision-maker facing that assumption would never try or retain the option, so the model rules out the behavior it seeks to explain. A weaker assumption restores the argument.
C3. A theory paper's central theorem asserts that equilibrium is unique. One step of the proof claims that a belief remains below a threshold up to a certain point, but the paper's own equilibrium conditions, evaluated at its parameters, imply the opposite. Subsequent steps depend on this claim, leaving uniqueness unproved as written. Uniqueness is an important part of what makes the paper's main results interesting.
C4. A theory paper's main existence result relies on constructive proofs in which two steps fail against explicit counterexamples, one constructed from the paper's own worked example. As written, the main result is unproved, though it may hold by another route.
5. What this means for institutions that rely on research
Central banks, government agencies, consultancies and journals treat published economics as an input to decisions. Peer review is the primary safeguard, yet referees rarely re-derive every proof or reconcile every figure in the text against the tables. In this sample, one paper in ten carried a serious error of that kind past review. Because our audit did not have access to full replication packages, the true rate of serious errors is likely underestimated.
Systematic checks of this sort are now practical at scale, though they do not replace expert judgement. Most issues the pipeline raises are correct, but only 14% bear on a major point; separating serious errors from minor ones still required a strict filter followed by expert reading. The practical role for such tools is a routine first pass, before publication or before an institution acts on a paper, with experts reviewing what passes the filter.
Contact
This audit was run by Refine. For questions about the method, email feedback@refine.ink.
More posts

Refine 5 beats GPT-6 Astra and Claude Fable 5.1 at paper review
On 108 papers in ten fields, from macroeconomics to genomics and statistical physics, Refine 5 won 197 of 215 matches.

A Retrospective Math Benchmark
Mathematician Daniel Litt ran 20 of his papers through Refine. Of the 278 comments it generated, 97.8% pointed to a real issue that coauthors, referees, and editors had missed.

Refine partners with the American Economic Association and the Econometric Society
Two leading publishers in economics now use Refine's AI-assisted technical verification as part of their publication processes.
