How we check
AI headlines travel fast and the studies behind them travel slowly. Here is how we close that gap.
01The method
- Pin the claim. We write down exactly what the headline says, in its own words, before reading anything else.
- Read the original. We go to the study, report or survey the headline is about, not a summary of it, and note what kind of source it is.
- Gather context. We look for what else is known, including evidence that supports the headline, not only evidence against it.
- Get a second opinion. A separate review pass reads the draft against the sources and tries to find where we went wrong.
- Give a verdict. We pick one rating from the scale below and say in one sentence why.
02How we check sources
Every check has a "Check our sources" section with a card for each source: who published it, the one claim on our page it backs, and a short quote from the source itself, so you can open it and see for yourself.
- Every source is checked twice. Two different AI models, Claude Haiku and ChatGPT, each read the source page on their own and judge whether it contains the quote and backs the claim. Each card shows both results and the date. A source either model says doesn't back the claim can't be published until we fix the claim or replace the source.
- We re-check every month. Once a check is published, an automated check opens each source again every month and asks a third model whether the page still says what we quote. Sources that pass show "Last re-checked" with the date.
- A person reviews every problem. When a link goes down or a page changes, it goes to us, not straight onto the page. We confirm it, mark the card ("Link down", "No longer says this" or "Needs a new source"), and find a replacement.
03The verdict scale
- Holds up
- The headline is accurate as written. The evidence says what it says.
- Mostly holds up
- The core claim is right; a detail, number or framing is off in a way that does not change the takeaway.
- Mixed
- Part of the headline is supported and part is not, or the evidence points both ways.
- Overblown
- There is a real finding underneath, but the headline inflates its size, certainty or reach.
- Doesn't hold up
- The evidence does not support the headline as written.
- Too early to tell
- The claim rests on evidence that is too thin or too new to judge yet. We say what would settle it.
- Out of date
- It was fair when published, but newer evidence has overtaken it.
Each check also says what kind of source the headline rests on: peer-reviewed study, preprint, company or industry report, survey, government data, news report, nonprofit report, anecdote.
04Ground rules
- We rate the headline as written, not the study. A careful study can sit under a careless headline. We judge the words people actually share.
- We say when a headline holds up. Plenty of worries about AI are well founded. When the evidence backs a headline, that is our verdict, and every check has a section on what the headline gets right.
- We fix mistakes in the open. Every check has a "What changed" log. If you find an error, tell us and we will correct it there.
- We use AI, and a person signs off. AI tools help us spot headlines and do first-pass research. A person reviews every check before it is published.
- We link our sources. Every number links to where it came from, the original source comes first, and each source is checked as described above.
05Who runs this
Progressives for AI is an independent newsletter and site for progressives who want to understand AI, shape how it is built, and use it for progressive change. It is run by Jordan Krueger, who works with progressive nonprofit advocacy groups on technology and operations.