What is the chance check?

It shuffles the outcomes — breaking any real link to the conditions — and re-runs the whole sweep. Whatever still passes is what pure luck can do on your data.

This is the answer to the obvious objection to any edge scan: if you try three thousand things, of course some of them clear a threshold.

Check against chance appears next to Scan for edges once a scan exists. It takes the outcome columns, shuffles them so that no outcome is attached to the day it actually happened on, and runs the identical sweep against the scrambled data five times over. Anything that "passes" now passed on noise, by construction.

The result lands on the honesty strip as chance alone passes ≈ N, with the range across the shuffles, and as this sweep filed N× that. It also lands on individual studies as a small tag beside the grade — chance ≈ 17% — meaning: of all the shuffles, that share produced a best-result-anywhere-in-the-sweep at least as strong as this one. Lower is stronger.

That percentage has a floor set by how many shuffles have been run, because the check cannot see finer than its own resolution. Five shuffles floor it at roughly 17%, ten at 9%, twenty at 5%. Add shuffles raises the count and lowers the floor; past twenty the app stops offering, because another doubling does not change what anyone would do.

One rule the app enforces for you: re-running the scan retires the chance check until it is run again, so the two figures are never quoted against different data.