← README
The Four Nulls
falsify-eval · v0.2.0 · plain English first, technical reference below
Four different ways to ask:
"Could a dumb system have gotten this score?"
Before falsify-eval, an AI system could report a score like 0.77 nDCG and call it a win. But that number only means something if a completely dumb system couldn't have gotten it too. The four-null gate runs four different types of dumb systems against the same benchmark and checks whether your real system beats all of them by a meaningful margin. If it does, the score is real. If it doesn't, the score is noise.
Each null is designed to catch a different type of cheating. That is why you need all four.
The Four Nulls
What it does
Imagine a teacher with 100 students and 100 correct answer sheets. Null A picks those answer sheets up and randomly deals them back out, so each student now has
someone else's answer sheet. Every correct answer still exists — they're just attached to the wrong question. Then it runs the scoring against that shuffled version.
Real example
Your benchmark has the query "What is the Pythagorean theorem?" and the correct answer is "a² + b² = c²". Null A might give that answer to the query "Who wrote Hamlet?" instead. The answer still exists in the benchmark. It just no longer belongs to that question.
🎯
Catches: systems that look good because they learned the shape of the answer distribution, not because they matched questions to their actual answers. If your score barely drops when the labels are shuffled, you weren't using the question at all.
What it does
For each question in the benchmark, Null B ignores the question entirely and just picks a random answer from the answer pool — equal probability for every option. It's like answering a multiple choice test by rolling a die for every single question.
Real example
The benchmark has 500 possible answers. For every query, Null B picks one of those 500 at random with no bias. It doesn't care what the question is. It doesn't favour common answers. Pure lottery.
🎯
Catches: score inflation from the raw structure of the benchmark. Some benchmarks are set up in a way where random guessing does surprisingly well. If your system can't beat a random guesser, it hasn't proven anything.
What it does
Your retrieval system searches through thousands of documents and returns the top results. Null C ignores all of that work and instead returns a random selection of documents from the same corpus. Same corpus, same number of results — just completely random ones with no connection to the query.
Real example
You search for "symptoms of diabetes". Your system returns five relevant medical documents. Null C returns five random documents — one about Roman history, one about cooking, one about jazz, two about engineering. They just happen to be in the same corpus.
🎯
Catches: the most basic failure of all — a system that scores well just because the benchmark has a high density of relevant documents, meaning random retrieval has a non-trivial chance of hitting something relevant anyway.
What it does
Null D looks at the entire benchmark, figures out which answers appear most often, then guesses those more frequently. If "Rigveda" is the correct answer for 30% of all queries, Null D answers "Rigveda" about 30% of the time. Not because it understood anything. Just because it counted.
Real example
A medical benchmark has "hypertension" as the answer for 40% of queries. A system that always says "hypertension" scores 40% recall without reading a single question. Null B wouldn't catch this — it picks randomly, so it'd only say hypertension ~1% of the time. Null D simulates the actual cheat: guessing the common answers with realistic frequency.
Why this is the key contribution
Nulls A, B, and C existed in scattered form before this work. Null D did not. It closes a specific gap: a system that learned "what answers are popular" rather than "what answers are correct" would pass A, B, and C. Null D catches it. If your system cannot beat a frequency-weighted random guesser, it has not learned anything beyond the base rate.
🎯
Catches: systems that game the corpus by learning which documents or answers appear most frequently, then returning those regardless of the actual query. The hardest cheat to detect. The most common one in practice.
How the gate works
Your system runs. Each null runs. The gaps are measured. All four gaps must clear the threshold.
Your system
score: 0.77
vs
Null A
0.45
Null B
0.42
Null C
0.08
Null D★
0.51
→
All gaps
≥ 0.05?
→
PASS ✓
Gap = your score − null score · All four gaps must be ≥ 0.05 (τ) · One failure = gate fails
Bar = real score · Red line = null score · The gap between them must clear 0.05
C · Retriever
gap +0.69 ✓
D · Frequency ★
gap +0.26 ✓
D failing
example
gap +0.03 ✗
Why you need all four
Each null catches a different blind spot. Removing any one leaves a hole a cheating system can slip through.
Without A
A system whose score is driven by the structure of the relevance distribution — not the actual query-answer mapping — passes undetected.
Without B
Score inflation from a benchmark where random guessing already performs reasonably well goes undetected.
Without C
A system that barely outperforms random document retrieval — and therefore doesn't do real search — passes undetected.
Without D
A system that memorised "answer X is the most common" rather than learning anything about relevance passes undetected. This is the most common real-world failure mode.
Technical Reference
Formal definitions
The delta metric
Δk = metric(system) − E[metric(nullk)]
Gate passes on null k if Δk ≥ τ (default τ = 0.05)
The expectation is estimated over n_boot bootstrap resamples (default 1000). A 95% bootstrap confidence interval is reported alongside each delta. The Bonferroni correction is applied across the four comparisons.
Definition 1 (from preprint §3) — the gate
Definition 1
Let S be a retrieval system, B a benchmark with gold relevance judgments G, and σ a scoring metric. The four-null Δ-gate passes if and only if:
Δ
A ≥ τ ∧ Δ
B ≥ τ ∧ Δ
C ≥ τ ∧ Δ
D ≥ τ
where each Δ
k is estimated by bootstrap resampling over queries.
Proposition 1 (from preprint §3) — soundness
Proposition 1
A gate PASS constitutes a joint rejection of four null classes. Each class corresponds to a distinct invariance that, if present, renders the observed score uninterpretable as evidence of retrieval quality. The conjunction is necessary: the four failure modes are orthogonal, and any single null in isolation leaves the other three unprobed.
How each null is constructed in code
Null A — permuted gold labels
A seed-driven random permutation π is applied to the query indices. Each query qi is evaluated against the gold labels originally belonging to qπ(i). The permutation is a bijection: every gold label set appears exactly once.
mapping[query_i] = gold_labels[pi(i)] # bijection, seed-controlled
score_A = metric(system_output, mapping)
Sensitive to: metric_fn, query-label independence assumptions. The bijection preserves the global relevance distribution while destroying query-document correspondence.
Null B — uniform random gold labels
For each query, a gold label set is sampled uniformly at random from the full pool of gold sets (with replacement, independently per query). This differs from Null A in that Null B is iid across queries; there is no bijection constraint.
for q in queries:
null_b_labels[q] = random.choice(all_gold_sets) # iid, uniform
score_B = metric(system_output, null_b_labels)
Null C — random retrieval
The system's ranked list for each query is replaced by K documents drawn uniformly at random from the corpus, without regard to query. The gold labels remain unchanged; only the retrieved documents change.
for q in queries:
null_c_output[q] = random.sample(corpus, k=top_k) # no query signal
score_C = metric(null_c_output, original_gold_labels)
Null D — marginal-matched random labels NEW
The empirical marginal distribution over gold labels is computed across all queries. For each query, a gold label set is sampled from this marginal distribution — so frequently-occurring labels are sampled proportionally more often. This directly simulates a system that learned the corpus frequency distribution.
marginal = compute_label_frequency(all_gold_labels) # P(label)
for q in queries:
null_d_labels[q] = sample_from(marginal) # frequency-weighted
score_D = metric(system_output, null_d_labels)
Why this matters: Null B samples uniformly — so a label appearing in 30% of queries is sampled only ~0.2% of the time in any given trial. Null D samples it ~30% of the time, directly replicating the frequency-gaming exploit. This is the failure mode that prompted the fourth null.
Parameters
τ (tau) — threshold
Default
0.05. Minimum gap required between real score and each null score. Increase for stricter evaluation. Must be set before benchmarking and held fixed.
n_boot — bootstrap samples
Default
1000. Number of resamples used to estimate null distribution means and confidence intervals. Higher values give tighter CIs at the cost of runtime.
top_k — retrieval depth
Number of documents retrieved per query. Affects Null C directly. Passed through to
metric_fn for Recall@K, nDCG@K variants.
seed — reproducibility
All four nulls use a fixed seed. The same seed on the same benchmark produces identical null scores across runs, enabling fair comparison across system versions.
Known limitations
Multi-label metrics and Null A/B: On benchmarks with many relevant documents per query (e.g. NFCorpus under nDCG@10), the gap between legitimate retrievers and Nulls A and B can narrow. This is a property of the metric on that benchmark, not a weakness of the gate. The gate correctly flags it. Recommendation: pair a graded metric with a strict single-gold metric (recall@k_top1) and treat disagreement as a flag for the graded metric.
Scope: The gate applies to retrieval and ranking systems. It does not apply to free-text generative outputs (summarisation, open-ended QA). Extension to that domain is a separate research direction.
Small benchmarks: At N < 40 queries, bootstrap CIs are wide and the gate will often return UNDER-NS (insufficient statistical power) rather than PASS or FAIL. The calibration curve in §5.7 of the preprint shows CI tightening as N increases.
Further reading