Hypothesis Testing for Machine Learning: Separate Signal from Sampling Noise in Model Comparisons

Learn one complete paired-model comparison, from a 2 by 2 prediction table and exact p-value to effect size, confidence interval, test choice, and leakage checks.

KnowledgeGate Team

Exam prep & CS education

Updated 17 Nov 20266 min read
Hypothesis Testing for Machine Learning: Separate Signal from Sampling Noise in Model Comparisons

Model B scores 84.2% while Model A scores 82.0%, but that 2.2-point gap alone cannot tell you whether B is genuinely better or simply lucky on this sample. Your real problem is deciding whether a leaderboard win is a repeatable improvement or sampling noise, and whether it is big enough to act on. Preserve the paired outcomes, test the disagreement with an exact McNemar p-value, and interpret it alongside the effect size and confidence interval.

Hypothesis testing for ML starts with an estimand, not a p-value

First define Delta = P(Model B correct) - P(Model A correct) for future examples from the same deployment population. Test H0: Delta = 0 against two-sided H1: Delta != 0. Use Delta > 0 only if pre-chosen and a worse B changes the decision.

delta_hat tells you the difference observed in your sample. The p-value tells you how incompatible the data are with H0, while the effect size and interval describe plausible values of Delta. None of these gives P(H0), the probability that the null is true.

The sampling unit is one independent labelled example. Repeated users, patients, devices or time blocks require grouping, cluster-aware resampling or blocking. Build the statistical and programming foundations through Skill Development Courses.

Paired model predictions reduce to a 2 by 2 disagreement table

Suppose both models predict 1,000 previously untouched, independent examples.

Model B correct

Model B wrong

Row total

Model A correct

790

30

820

Model A wrong

52

128

180

Column total

842

158

1,000

Margins give A (790 + 30)/1000 = 820/1000 = 82.0% and B (790 + 52)/1000 = 842/1000 = 84.2%. Thus delta_hat = 0.842 - 0.820 = 0.022, or 2.2 points.

An independent proportions test loses pairing. The models agree on 790 + 128 = 918 cases. The 82 disagreements are A-only b = 30 and B-only c = 52. The net is c - b = 22, not 52.

A paired 2x2 table comparing two models on 1,000 examples, highlighting the 30-vs-52 discordant cases that decide whether the 2.2-point accuracy gap is real.

McNemar's exact test turns the 30 versus 52 split into a p-value

Under H0, each of 82 disagreements equally favours either model, so B-only wins follow X ~ Binomial(n = 82, p = 0.5). Expect 82 x 0.5 = 41; 52 is 11 higher.

The exact two-sided result is p = 2 x P(X >= 52 | X ~ Binomial(82, 0.5)) = 0.01983, rounded to 0.020. Reject H0 at pre-chosen alpha = 0.05. This supports unequal paired error behaviour, not a 98% probability that B is better.

The p-value uses the 30 versus 52 imbalance, not deployment value, data quality, calibration, fairness, latency or error cost. Review Probability for GATE CS for the binomial foundation.

Effect size and confidence interval answer the practical question

A makes 1000 - 820 = 180 errors and B makes 1000 - 842 = 158, 22 fewer per 1,000. The absolute effect is 2.2 points; relative error reduction is (180 - 158)/180 = 22/180 = 0.1222, about 12.2%. They answer different questions.

Define D: +1 for 52 B-only wins, -1 for 30 A-only wins, 0 otherwise. Its mean is (52 - 30)/1000 = 0.022. Because D is +1, -1 or 0, each of the 82 disagreements contributes 1 to the sum of squared values, so that sum is 82; dividing by n minus 1, which is 999, gives the sample variance [82 - 1000(0.022)^2]/999 = 81.516/999 = 0.08160. The standard error is sqrt(0.08160/1000) = 0.00903, and the approximate 95% interval is 0.022 +/- 1.96(0.00903) = [0.0043, 0.0397], or 0.43 to 3.97 points.

Set pre-chosen delta_min = 1 point. The estimate clears it, but the interval includes smaller gains. This supports a positive average gain, not 95% confidence that it clears the threshold or an automatic deployment verdict.

Test selection depends on pairing, outcome type and sampling design

Evaluation design

What is paired

Suitable method

Main caution

Same examples, binary correct/wrong

Example predictions

McNemar or exact binomial

Independence or clustering

Same examples, decomposable continuous loss

Example losses

Paired permutation or paired t-test on differences when reasonable

Skew and dependence

Independently assigned user groups, binary conversions

None

Two-proportion or Fisher's exact for small counts

Assignment and exposure definitions

Same cases, ROC AUC

Predictions, AUC is global

DeLong or paired bootstrap

Not an accuracy test

More than two models on the same independent datasets

Dataset results

Friedman or repeated measures, then corrected comparisons

No uncorrected pairs

K-fold scores correlate because training sets overlap. 5 folds x 10 repeats = 50 is not 50 independent observations for a naive t-test. On one dataset, use a pre-specified nested design with paired resampling or correction. Across separate datasets, pair by dataset.

Ask about pairing, outcome type and unit independence. Test names come last.

A decision flowchart for choosing the right significance test to compare two models, branching on paired vs independent data and metric type, and warning that k-fold scores aren't independent.

Sampling, test-set reuse and multiple comparisons can manufacture significance

Audit first: was the test set untouched during model choice? Do rows repeat users or sessions? Did preprocessing learn from it? Were thresholds selected on its labels? Does its population match deployment? A correct p-value cannot repair leakage or bias.

For 20 independent null comparisons at alpha = 0.05, at least one false positive has probability 1 - (1 - 0.05)^20 = 1 - 0.95^20 = 0.6415, about 64.2%. Tests need not be independent; repeated searches still weaken an isolated p.

Pre-register one primary metric, reserve an untouched final test set, and report planned results, including negatives. Use Holm for family-wise error or Benjamini-Hochberg for false-discovery rate. A post-hoc threshold is exploratory, not confirmatory.

ML assessments test interpretation more often than arithmetic

Typical questions ask for H0 and H1, McNemar for paired examples, a valid reading of p = 0.020, and the difference between 2.2 points and 12.2% error reduction. Never claim P(H0) = 0.020.

Swap to A-only 52 and B-only 30: p stays 0.020, but B's effect becomes -2.2 points. Set both to 41: delta_hat = 0, p is 1. Raise the useful threshold from 1 to 3 points: 2.2 fails it, while p is unchanged.

Mistake

Why it fails

Repair

Accuracy margins only

Pairing is lost

Tabulate disagreements

Read p as P(H0)

Wrong probability

State incompatibility under H0

p > 0.05 proves equality

Non-rejection is not equivalence

Use equivalence testing

p < 0.05 proves value

Size and cost are absent

Add effect, interval, threshold

Choose one-sided after B wins

Direction came from data

Pre-specify it

Count CV folds as independent

Training overlaps

Use design-aware methods

Test many metrics, report smallest p

Selection inflates false positives

Correct the declared family

Omit sampling population

Estimand is unclear

Define population and unit

Hypothesis testing for ML: report the whole claim, then take the next step

On 1,000 previously untouched paired examples, Model B achieved 84.2% accuracy versus 82.0% for Model A. The absolute gain was 2.2 percentage points, with an approximate paired 95% confidence interval of 0.43 to 3.97 points. Among 82 disagreements, B was uniquely correct on 52 and A on 30; an exact two-sided McNemar test gave p = 0.020.

It assumes independence, representative sampling, test-set integrity and a pre-specified metric. Calculation cannot create them.

Use this six-step revision card:

  1. Define population and unit.

  2. State H0, H1 and minimum useful effect.

  3. Preserve pairing.

  4. Match method to design and outcome.

  5. Report effect, interval and p.

  6. Audit leakage and multiplicity.

Reproduce 82, 41, 0.020, 2.2 points, 12.2%, and 0.43 to 3.97 without looking back. To take these evaluation habits further through a full applied AI course, continue with AI & ML for Placements. Then rerun it on a real paired prediction file: state its unit, lock a practical threshold, and write the result before comparing another model.