The problem NEFF solves

The problem it solves

Five models agreeing is often one opinion repeated five times. They share training data and failure modes, so they are wrong together — and averaging is most confident exactly when the panel is most likely to be wrong.

NEFF asks each model three questions, in separate requests with no shared context:

Say yes to both a claim and its opposite → echo, vote zeroed. Name the same evidence as another model → one witness, not two. What survives is N_eff, computed with Kish's design effect (1965):

N_eff = k / (1 + (k−1)·ρ) ρ measured per claim, from the evidence
Truth Score = 50 + 50 × balance × n/(n+1) n = N_eff

Proof, from the live app: three models returned SUPPORTED at 95%, 95%, 85%. A vote-counter calls that unanimous. All three also affirmed the opposite claim.

NEFF scores it NO SIGNAL — 0.0 effective witnesses, and prints the six answers that prove it.

Every one of the 11 inferences shows its Gonka Request ID, serving node, latency and exact request body. No verdict reaches 0 or 100: a panel of language models is evidence, never proof.

Challenges we ran into

Challenges

Measuring ρ at query time, with no corpus to measure against.
Kish's formula needs a correlation between observations, and published inter-model agreement figures are benchmark averages — useless for this claim. So ρ comes from the evidence the models themselves name, scored by overlap. That leaves one free parameter, and we set it to err against our own thesis: in doubt, models count as independent, which raises the score and weakens our headline. npm run sweep:threshold reproduces the argument from runs on disk in two seconds, zero inferences.

Every constant had to survive "why 50, why not 40?"
Each number on screen is labelled definition, standard (published, source named), or chosen (our call, with the reasoning and which way it errs). One argument for the shrinkage prior turned out circular — METHOD.md says so instead of dressing it up.

Our own failure paths were manufacturing evidence.
An unparseable probe fell through to a recovery that accepted a draft missing the key we asked for — a failed probe reported as a witness that "named no source", counting toward the verdict. Failure laundered into data. Caught in review, fixed, replayed over 62 captured responses to prove nothing real was lost.

"No evidence" is not "not measurable".
Treating an unmeasurable witness as a disproven one makes the demo sharper and is a lie. Where overlap can't be measured, the prior is used and labelled as assumed — on screen, and in what the adjudicating model is told.

The visualisation couldn't draw what it claimed to show.
Dark seats on the sphere mean this model didn't answer — but the ring filled by count (i < landed), so it could never render a gap. Structurally incapable of telling the truth its caption claimed. Rebuilt per-probe; a real arrival order now shows gaps in 7 of 9 states.

Neutrality had to live in the protocol, not the prompt.
"Be objective" is not a mechanism. No probe sees the others; the mirror probe is a separate request with no shared context, so a model can't stay consistent with an answer it never saw.

Building against a network we didn't control.
Cached completions replayed one truncated answer forever; timeouts were non-retryable, costing a witness per slow node; the concurrency ceiling moved with everyone else's load. Fixes were the boring correct ones — cache-busting probes, retry budgets clipped to the deadline, and an AIMD gate (halve on refusal, widen after three clean calls) instead of a constant that would be wrong an hour later. 477 probes since: zero lost to parsing.