I built H2AI Chat, an AGPL platform where several different models — from different vendors — debate a topic in turns while a human moderates. Disclosure up front: this is my project.
Over the past two days we hand-verified 41 of those debates, claim by claim: 141 statements marked, 44 of them flatly false.
We don’t delete or correct them. The sentence stays, struck through, and you can still read it by selecting it — with the reason and the source underneath. Editing what a model said would break the only promise the site makes.
Three patterns we didn’t expect:
- Fabricated authority shows up exactly where an argument is challenged. One debate answers a budget objection with three invented citations in a single turn.
- Fabrications spread between models. One invents a figure, a second treats it as established, a third does arithmetic on it.
- One claim contradicts itself inside its own sentence: “62% voted Remain on a 67% turnout, meaning roughly 22% of the electorate” — which is 41.5%.
Debates: https://h2aichat.com/ Code and the fact-check register: https://github.com/Tonterias/h2aichat
You know you can just do math right?
Turns out I answered this three weeks ago — and posted it next to your comment instead of under it, so you never got the notification. A project about machines making confident mistakes, undone by a threading bug. Noted.
Short version: yes, and that’s exactly why that one’s in the screenshot. Of the 141 claims we marked, it’s the only one you can check without leaving the sentence — no source, no taking our word for it. The other 140 aren’t arithmetic. One was a National Holidays Act 1946 that doesn’t exist. Another was a Council of Europe report nobody wrote, which a second model then cited back as established fact. You can’t do the maths on those. You have to go and look.