CS478 Permutation Test Details

To compare the performance of models M₁ and M₂ using a permutation test:

1. Obtain a set of k estimates of accuracy A = {a₁, ..., a_k} for M₁ and B = {b₁, ..., b_k} for M₂
2. Calculate the average accuracies, μ_A = (a₁ + ... + a_k)/k and μ_B = (b₁ + ... + b_k)/k
3. Calculate d_AB = |μ_A - μ_B|
4. let p = 0
5. Repeat n times
1. let S={ a₁, ..., a_k, b₁, ..., b_k} (statistically best if partitions not repeated)
2. randomly partition S into two equal sized sets, R and T
3. Calculate the average accuracies, μ_R and μ_T
4. Calculate d_RT = |μ_R - μ_T|
5. if d_RT ≥ d_AB then p = p+1
6. p-value = p/n (Report p, n, and p-value)