Version: 0.4.0 (draft) Author: Lovro Lucic License: CC-BY-4.0
Frame Check is a small deterministic tool. It reads one English analytical document and reports what perspectives the document takes, how confidently it speaks, and whether its numbers hold up. No language model judges the text; the same input always produces the same output. This page explains how each part works and where it stops being reliable. It describes a tool, not a research program. Where a claim here has been tested and failed, that is said plainly.
Frame Check has two parts.
Framing analysis (deterministic, no model calls). Pattern matchers compute, for the document:
Verification floor (data sources, no model calls). Numeric claims are checked against authoritative data where coverage exists (see section 3).
Identical inputs return identical measurements. Nothing on this path calls a language model.
Frame Check ships a catalog of about twenty named framing patterns, each with a stable ID (FVS-NNN), detection cues, counter-examples, a worked example, and a question worth asking. The catalog and its rules are bundled with the tool and served as MCP resources.
A match means a rule-based signal fired. It is a hypothesis for you to check, not a verdict, so every match is delivered with a question rather than a conclusion. The rules are identification-only: they flag where a pattern may be present and never rewrite or grade the text. They are pinned and guarded by tests so a change to the engine cannot silently shift what a match means.
Frame Check also reports divergence: the catalog entries the document did not trigger, sorted by signal strength. These absences are often more useful than the matches, because they point at perspectives the writer did not take.
Numeric claims are matched against data providers that have genuine coverage for the claim type: SEC EDGAR (US public-company filings), FRED (US macroeconomics), World Bank (country statistics), Alpha Vantage (equities), Wolfram Alpha (computational and physical facts), Wikipedia, and CoinGecko. Web search can surface a page that mentions a number. That is a lead to follow and never counts as verified or contradicted.
A number counts as verified only when the source reports the same thing: the same entity, metric, unit, and period. Matching digits are not enough, so a filing figure cannot confirm a number attached to a different event. A claim with no year can only match the latest period and is never contradicted, since it may be about a year it did not name. A sentence naming two years, or a claim stated as a limit ("exceeded $34 trillion"), cannot be contradicted either.
This layer is deliberately bounded. Coverage is strong for US public-company financials and macroeconomic indicators and close to zero for medical, legal, and niche-industry claims. In one calibration corpus, about 86 percent of claims were not verifiable against any available source. Frame Check reports per-provider precision, recall, and F1 rather than asserting accuracy, and it marks a claim unverifiable instead of guessing.
To test this, reviewers who had not seen the code wrote labeled test sets, half or more of them true, many deliberately misleading, and each set was scored once in November 2026. The result that best describes real use comes from whole documents, graded number by number. The latest set was 14 documents in the forms real text takes: chat answers with lists and tables, press releases, analyst notes, newsletters, FAQs, memos, meeting notes and plain articles. It confirmed 28 of 59 true numbers, caught 4 of 36 false ones, wrongly confirmed 1, and contradicted none. The fault behind that false confirmation was then fixed and has not been re-measured on a new set. Everything else came back unresolved. On isolated sentences, where the subject and year are stated outright, the last two sets confirmed 23 and 18 of 30 true claims, wrongly confirmed 2 and 1 of 30 false ones, and contradicted 0 and 2 true claims.
So expect a quarter to a half of the true numbers in covered areas to be confirmed, and most false numbers to come back unresolved rather than flagged. An unresolved number has not been checked, which is not the same as being correct. The samples are small and the false claims were chosen to be hard, so read these as how it fails, not a general accuracy rate. The test sets and scorers are in calibration/binding_eval/, scripts/score_verifier.py, and scripts/score_documents.py.
Frame Check does not judge reasoning quality, logical validity, insight, or whether a document is useful for your purpose. Those need a human reader.
It does not detect semantic manipulation. A document can pass every structural check and still mislead through word choice, selective emphasis, or a false premise. Structural analysis is a floor to stand on, not a verdict.
The named-pattern layer is the weakest part, and the honest record says so. When the rule-based detector was compared against independent human labeling on a mixed-genre corpus, agreement was near chance (macro-F1 of 0.157; a later rule revision reached about 0.27, still low). So a frame match should move you to look closer, never to conclude. The full study lives in fvs_eval/validation_study/, and VALIDATION_PROGRAM.md and ANTICIPATED_CRITIQUES.md keep the running account of what is and is not established.
One thing does hold up under a controlled check: the same content rewritten from a different frame produces a measurably different structural profile. The structural signals track framing even though the named-pattern labels do not yet match human readers well.
Code is Apache-2.0. The frame vocabulary, this document, and the worked examples are CC-BY-4.0, so they can be cited and reused.
Lucic, L. (2026). Frame Check: a tool for seeing the frame in documents and checking their claims. https://frame.clarethium.com
See CITATION.cff for the canonical record.
The frame vocabulary v0.2.0 catalogs the structural framing patterns this methodology detects. 16 entries are published and curated; 4 were withdrawn during review and are documented with disposition notes at the library index.