L4-FactCheckText analysisLLMLayer 4

Fact Check

Uses a large language model to verify factual claims in the text, catching errors and fabrications that go beyond what a dictionary-based approach can detect.

Technical description

Sends extracted factual claims to an LLM with instructions to evaluate their accuracy based on its training knowledge. The LLM assesses: correctness of attributed scientific findings, accuracy of historical claims, plausibility of quantitative assertions, and internal consistency of the document's factual framework. Returns structured assessments with confidence levels and specific corrections where applicable.

How it works

Layer 4 (LLM-powered): Sends the text to a language model with a rubric of three dimensions, judged independently: internal contradiction (do statements conflict, grounded in the text), plausibility (are numbers reasonable and dates non-anachronistic, with the current date supplied), and consensus (does a claim conflict with settled science, flagged only when the model is confident). The model returns a sub-score and flagged claims per dimension; it is instructed to abstain rather than guess, and low-confidence flags are dropped. Sub-scores combine into one fact score, weighted toward contradiction and plausibility, with the breakdown kept alongside. Runs only when a model is configured.

Why this matters

Confident factual invention is the signature failure of language models, and the costliest case in a manuscript is the claim that is plausible, well-phrased, and wrong. A surface check cannot tell whether a claim is false or whether two statements quietly contradict each other; that needs meaning. A model supplies that read without any lookup: its parametric knowledge carries real signal for internal contradictions and plausibility, while settled-science conflicts are treated as a confidence-gated signal because models are poorly calibrated and tend to overconfidence.

Score thresholds

0-1: All checked facts appear accurate
2-3: Some claims could not be confidently verified
4-5: Multiple factual claims appear to be incorrect

Limitations

Requires a configured LLM provider. The evaluating LLM may itself hallucinate during fact-checking. Very domain-specific facts may be outside the LLM's training knowledge. Recent discoveries after the LLM's training cutoff cannot be verified. Confidence calibration varies across domains.