Paper Preflight

Check every reference of a paper against real scholarly records before you submit. No LLM guessing, no false accusations.

English · 简体中文 · Quick start · User guide · Releases · Feedback

Language models invent references, and copy-pasted BibTeX carries wrong years, wrong authors and dead DOIs. paper-preflight reads your paper (LaTeX, Word, Markdown, Typst, a PDF or a plain reference list) and asks Crossref, dblp, arXiv, DataCite, PubMed and OpenAlex about every cited work, with Semantic Scholar too if you have a free key, Open Library for books without a DOI, and GitHub, PyPI, CRAN, Hugging Face and OpenML for software, models and datasets:

• Does it exist?
• Does it match what you wrote?
• Has it been retracted?
• Has the preprint you cite been published since?

Every finding is backed by a record, or by every source that could know the work answering "no". When it cannot tell, it says "cannot determine" and why, instead of guessing. No language model takes part in a verdict.

Status: 1.0. On three weeks of real arXiv papers held out for the release: 0.79 false alarms per 100 references, against 409 real problems found (accuracy). Rule IDs, the JSON report, settings, commands and the Python API are kept through 1.x (what is stable). False positives are the bugs we most want to hear about: please open an issue.

Quick start

With uv nothing needs installing:

uvx paper-preflight check path/to/paper

path/to/paper is the project's folder, its main .tex file or a .bib file. To install it for good (Python 3.11 or later):

pip install paper-preflight

The repository's demo paper cites eleven works, several of them wrong on purpose. The animation above is a real run of paper-preflight check examples/demo-paper against the live sources. In it, the correct NeurIPS paper is verified through dblp although Crossref only holds fake copies of it, the book without an identifier is confirmed by Open Library, and the Chinese book, whose script is not supported yet, is "cannot determine", not "not found".

$ paper-preflight check examples/demo-paper
paper-preflight 1.0.0 · main.tex · 12 entries, 12 cited keys

error   CIT001 main.tex:31
    Citation key 'nonexistent2023' is not defined in any bibliography file (1 use(s)).
error   REF001 refs.bib:43
    The doi of 'devlin2019bert' (10.1109/cvpr.2016.90) resolves to a different work in Crossref: "Deep Residual Learning for Image Recognition" (He et al., 2016).
error   REF003 refs.bib:66
    'lindqvist2024quantum' was not found in Crossref, dblp and Semantic Scholar, and every source responded. Check that the work exists and that its title is correct.
error   REF004 refs.bib:73
    'wakefield1998ileal' has been retracted (reported by Crossref, OpenAlex). Cite it only if the text discusses the retraction.
error   CIT002 refs.bib:128
    Entry key 'kingma2015adam' is already defined at line 47; BibTeX ignores this one.
warning REF015 refs.bib:31
    'he2015residual' cites a preprint that has been published in CVPR (2016), DOI 10.1109/cvpr.2016.90. Cite the published version and keep the eprint field.
warning CIT004 refs.bib:37
    Entries 'devlin2019bert' and 'he2016deep' look like the same work (same DOI).
warning REF013 refs.bib:51
    'kingma2015adam' gives the year 2016, but dblp records 2014, 2015.
warning REF017 refs.bib:112
    The doi of 'tacl2019example' contains LaTeX escapes: '10.1162/tacl\_a\_00276'. Write it as: 10.1162/tacl_a_00276
info    REF005 refs.bib:73
    'wakefield1998ileal' has a published correction (reported by Crossref).
info    REF090 refs.bib:95
    'zhou2016ml' could not be verified: non-Latin titles are not supported yet; grey literature without an identifier (book, report, software, web page).
info    CIT003 refs.bib:116
    Entry 'lecun1998gradient' is never cited.

References: 7 verified · 1 metadata mismatch · 1 identifier conflict · 1 not found · 1 cannot determine
5 error(s) · 4 warning(s) · 3 info

What it reads

What each input can and cannot tell, in detail: docs/inputs.md.

What it checks

Every reference gets one verdict:

The findings say what is wrong:

Every rule has a page: what it checks, when it can be wrong, and what to do. paper-preflight explain REF003 prints it in the terminal.

How accurate is it?

Four measurements, all against the live sources:

Precision comes first: a reference is called fabricated only on positive evidence, and an unanswered or ambiguous lookup is reported as "cannot determine", never as "not found". The evaluation harness and every run's summary are in evals/.

Real papers

Each week is 20 arXiv papers (cs, stat, q-bio, quant-ph and astro-ph) chosen mechanically. For 1.0, three weeks never used before were committed together, before the changes they measure, and each was run once on the same code. The gate was at most 1.5 false alarms per 100 references in each week and at most 1.0 on average. Above them are the more recent weeks that stayed under 1.0, each run once with the version of its time; every week measured, those over 1.0 too, is in evals/README.md.

• The 409 real problems: 220 cited preprints since published, 149 identifiers written so that links break, 30 entries with wrong authors, titles or years (NeurIPS 2025 papers dated 2026, invented co-authors, author fields BibTeX misreads), and 10 works that are not there as cited: 5 found nowhere, 3 identifiers that do not exist, 2 that belong to other works. A few papers' reference lists are largely invented (medical imaging, LLMs).
• The 21 false alarms: registries' own errors (a cut or misspelt title, a garbled name, a dropped symbol) 5; groups credited as their makers ask and Zotero's "N other authors" placeholder 6; works no index holds or that a later version retitled 3; a conference paper compared in place of its journal version 2; APA's [software] and [dataset] labels after a title 2; three others. When a finding is wrong says what to do.
• The gate took four tries. The first three averaged 1.08, 1.24 and 1.28 per 100 references, each with one week over 1.5; each try's false alarms were fixed before the next, on new weeks. The first release, 0.1.0, measured 4.5. Every week measured since is in evals/README.md.

Hallucinations that passed peer review

GPTZero published 151 hallucinated references it found in NeurIPS 2025 papers and ICLR 2026 submissions, each confirmed by its staff. Pasted as plain text, as the papers printed them:

• None of them is verified. The 9 left undecided are web pages and blog posts, a workshop no source indexes, real titles given with invented authors where several works share the title, and two references too garbled to search. Each is listed with its reason in evals/results/gptzero.md.
• GPTZero's own tool found these, so they are the hallucinations a search can find; recall on every kind of hallucination is lower (see HALLMARK below).

HALLMARK

HALLMARK (v1.2.3) is a public benchmark of real and hallucinated BibTeX entries. Fabrication counts a wrong identifier, a work not found and no author in common; any issue also counts wrong authors, title, year or venue.

test_public was run with 1.0.0, dev_public with 0.8.0; no rule was tuned on test_public.

• Most of the false positives are disputed labels. Every flag on a dev_public entry labelled VALID was checked by hand: the 11 that remain are not correct citations (DOIs that belong to other papers, author lists naming people who did not write the paper, a shifted year, a truncated title). Without them, both modes reach 100% precision. The list, each with a reason one lookup confirms: evals/hallmark_disputed.toml.
• What is still missed: author lists cut short at the end, or checked only against a preprint whose later versions add people (named in an info), and invented entries that borrow a title several real works share. Every hallucination type is broken down in evals/results/.

Next to other tools

Badalova & Mayr (2026) checked 104 references by hand and published what five tools flagged. On the same references, with their labels:

The sample is small, so the intervals are wide. The study labels a reference correct when the work exists, so five of paper-preflight's "false" flags point at real errors (a wrong author, a broken DOI). Two causes of false flags found in this data were fixed before the run above (the first run measured 62.8%); see evals/results/badalova-mayr.md.

Limits

• Titles in scripts other than Latin (Chinese, Japanese, Cyrillic, …) are "cannot determine" for now, and so is a Chinese-language work cited by a translated English title, which the open indexes rarely hold.
• Grey literature without an identifier (reports, theses, manuals, standards) is often "cannot determine": the open registries hold little of it. Books are asked of Open Library, software of GitHub, PyPI and CRAN, models and datasets of Hugging Face and OpenML.
• Web pages are never judged true or false. A linked page that is gone, with no archived copy, is reported (REF021).
• A work no open index holds is "cannot determine" when its venue is unknown to the catalogues too, but at a venue they know it can be reported as not found: read REF003's page before acting on that finding.
• Not caught: an invented reference that borrows a real title several works share, an author list cut short at the end (an info), and a citation of a real paper for a claim it does not make (only the experimental support reads the claims).
• It needs the network on a first run: about 0.4 s per reference on a cold cache, so a 50-reference paper takes about 20 seconds. Answers are cached, and --offline works from the cache. A source that is down, with nothing to answer in its place, makes the run incomplete (exit code 2), never "clean".

Fetch verified BibTeX

Instead of writing an entry from memory, ask for it by DOI, arXiv ID or title. Every field comes from the registry record, which a comment above the entry names:

paper-preflight bib fetch 1810.04805

% Verified with paper-preflight against dblp (conf/naacl/DevlinCLT19), 2026-10-03
@inproceedings{devlin2019bert,
  title         = {{BERT:} Pre-training of Deep Bidirectional Transformers for Language Understanding},
  author        = {Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina},
  booktitle     = {NAACL-HLT (1)},
  year          = {2019},
  doi           = {10.18653/v1/n19-1423},
  eprint        = {1810.04805},
  archivePrefix = {arXiv},
}

• Preprints: an arXiv preprint that has been published comes back as the published version, with its eprint kept (--prefer preprint for the preprint itself).
• Titles: --title (with --author/--year if needed) lists the candidates instead of choosing when several works match.
• Retractions: a retracted work comes with a warning.
• Scripts and agents: --format json.

Fix the bibliography

bib fix turns findings into edits of your .bib files, taken from the verified records. It prints a diff and changes nothing until you add --apply:

paper-preflight bib fix path/to/paper --level unsafe

--- a/refs.bib
+++ b/refs.bib
@@ -48,7 +47,7 @@
   title     = {Adam: A Method for Stochastic Optimization},
   author    = {Kingma, Diederik P. and Ba, Jimmy},
   booktitle = {International Conference on Learning Representations (ICLR)},
-  year      = {2016},
+  year      = {2015},
 }

• --level safe (the default) only fixes what cannot change which work is cited: identifiers written so that links break, and DOIs the registry has but the entry lacks.
• --level unsafe also rewrites authors, title, year and venue from the record, removes identifiers that point to another work, and cites a preprint as its published version (the entry type, venue, year, volume, pages and DOI; the eprint is kept). Review the diff first.
• Only the affected fields change; comments, formatting, line endings and encoding are kept. A reference nobody could find is never "fixed": only you can say what was meant.

Options and exit codes

The JSON report follows a published JSON Schema (schema_version 1.0): fields may be added, but none is removed or changes meaning within 1.x. Every option, the cache and its lifetimes: docs/configuration.md.

Project settings

For a whole project, put the settings in paper-preflight.toml next to the paper (or in a folder above it, up to the repository's root), or in [tool.paper-preflight] of pyproject.toml:

ignore-rules = ["REF016"]                  # never report these rules
ignore-keys = ["internal2024*"]            # nor anything about these entries
severity = { REF015 = "info" }             # report published preprints as infos
disable-sources = ["s2"]                   # optional sources only
fail-on = "warning"                        # as --fail-on; the command line wins

check --config <path> names the file instead. The optional sources are s2, openlibrary, github, pypi, cran, huggingface, openml, web (links to web pages) and wayback; the registries that judge a reference cannot be turned off. A mistake in the file (an unknown rule or setting) is an error, never silently ignored. The JSON report names the file in run.notes, and the MCP server reads it too.

Silence a finding you have checked

A comment directly above an entry silences rules for that entry, with an optional reason:

% preflight: ignore[REF003] reason="internal technical report, not indexed anywhere"
@techreport{lab2024internal,
  ...
}

The verdict stays in the JSON report; only the finding is dropped. A suppression that silenced nothing is reported as CFG001 (info), so stale comments do not pile up (for reference rules, only after a complete online run, since offline answers and outages may leave them unrun).

Better results with free credentials

paper-preflight works without any account. These optional environment variables make it faster and more complete; their values are never printed or logged.

paper-preflight doctor shows which are set and whether each source answers right now.

Use it in your workflow

Claude Code: install the plugin. It bundles an MCP server and a skill that makes Claude check the references before calling a paper finished, fix only what is proven wrong, and never invent a reference.

claude plugin marketplace add amos689/paper-preflight

claude plugin install paper-preflight@paper-preflight

Codex, Gemini CLI, Copilot, Cursor and other agents: install the same skill (gh skill install amos689/paper-preflight paper-preflight works too). It runs the CLI when no MCP server is configured:

npx skills add amos689/paper-preflight

MCP clients (Codex, Cursor, VS Code, …): run paper-preflight mcp (the mcp extra). The tools are read-only and confined to your workspace; see docs/mcp.md.

GitHub Actions: uses: amos689/paper-preflight@main checks the paper on every push, with the report in the job summary and optional code-scanning alerts; see docs/github-action.md.

pre-commit: check citation keys and cached verdicts on every commit in seconds; see docs/pre-commit.md.

Python:

import paper_preflight

report = paper_preflight.check_paper("paper/")
for ref in report.references:
    print(ref.key, ref.verdict, [f.rule for f in ref.findings])

The results are plain, frozen dataclasses; see docs/python-api.md.

Experimental: find the passage behind each citation

support looks in each cited work for a passage that says what the citing sentence claims. It reads the work's text (the arXiv source, an open-access full text or PDF, or else the abstract) and a small local model (HHEM-2.1-open, 0.4 GB) scores the passages ranked best for the claim. It needs the support extra.

paper-preflight support path/to/paper --download-model --all

--download-model fetches the model's weights once; --all also lists the confirmed citations with their quotes. arxiv:<id> works as a target, as it does for check.

• What it says: "confirmed", with the passage quoted word for word, or "could not confirm", with the reason (no text, only the abstract, or no passage close enough).
• It never calls a citation wrong. On a gold set of 298 citations, 93% of its confirmations were right [95% CI 82%, 98%], but it confirms only about one real citation in six, and a low score pointed at a mis-citation less than half the time. The gold set's labels were made by AI models, not experts; see evals/results/support.md.
• Or let your agent judge. The MCP tool preflight_cited_passages returns each claim with the cited work's best passages, for the agent to judge by the same rules, without the model. On 100 gold-set citations, a Claude agent judging from it confirmed 39% of the real ones, against HHEM's 9%, every confirmation at least partly supported (the labels come from the same model family).
• What leaves your machine: only the cited works' identifiers, to fetch their text; the claims are scored locally.

How it works

1. Source-first. It reads the LaTeX project as LaTeX sees it: comments, \iffalse blocks and \includeonly are respected, .aux files are used when they are fresh, and the first definition of a duplicated key wins, as in BibTeX.
2. Identifier-first routing. DOIs go to their registration agency (doi.org tells which: Crossref, DataCite, …). arXiv IDs go to arXiv, with DataCite as a fallback, and PMIDs and PMCIDs to PubMed, which also marks retracted articles. Entries without identifiers are searched by title in dblp and Crossref; dblp through its SPARQL endpoint, since its search API is behind a bot challenge. Which source is asked what, and how politely: docs/sources.md.
3. Field-by-field matching with guards. Titles (including earlier arXiv versions'), authors (tolerating transcriptions such as Reiß/Reis), year and venue are compared. A search result is used only when enough of these agree and no other work fits as well; known fake DOI copies are skipped.
4. One verdict per reference, as in What it checks. "Not found" needs every required source to answer "no".
5. Cached locally. Answers are kept in a SQLite file, so re-runs are fast and --offline works.

Design principles

• Positive confirmation or abstain. Rate limits, outages and unindexed works lead to "cannot determine", never to "not found".
• No LLM in the verdicts. Every verdict comes from records and fixed rules, so the same records give the same report.
• Neutral wording. Findings state observations ("not found in Crossref, dblp and Semantic Scholar, and every source responded"), never accusations.
• Local-first, no telemetry. Only the metadata of the cited works (DOIs, titles, authors) is sent to the public scholarly APIs above. Your manuscript never leaves your machine.

What it will never do

Help evade plagiarism or AI-text detection, scrape paywalled or bot-protected sites, recommend or "complete" references from memory, or name and shame authors.

Roadmap

• Next: references in Chinese script, measured on held-out papers before anything ships.
• Fewer references left undecided among grey literature (reports, theses, manuals) without an identifier.

What each release brought is in the changelog; day-to-day progress in docs/PROGRESS.md (in Chinese).

Contributing

Bug reports with a reproducible .bib entry are the most valuable contribution, especially false positives. See CONTRIBUTING.md; security issues: SECURITY.md.

License

MIT. See THIRD_PARTY_NOTICES.md for adapted code.