On Fisher’s exact test: What test would have been appropriate if the lady hadn’t known the number of milk-first cups?

In the famous lady tasting tea experiment by RA Fisher, the lady is informed of how many milk-first/tea-first cups there are (4 for each out of 8 cups). This respects the fixed marginal total assumption of Fisher’s exact test.

I was imagining doing this test with my friend, but the thought struck me. If the lady can really tell the difference between milk-first and tea-first cups, she should be able to figure out marginal totals of the milk-first/tea-first cups as well as which ones are which.

So here is the question: What test could have been used if RA Fisher hadn’t informed the lady of the total number of milk-first and tea-first cups?


Some would argue that even if the second margin is not fixed by design, it carries little information about the lady’s ability to discriminate (i.e. it’s approximately ancillary) & should be conditioned on. The exact unconditional test (first proposed by Barnard) is more complicated because you have to calculate the maximal p-value over all possible values of a nuisance parameter, viz the common Bernoulli probability under the null hypothesis. More recently, maximizing the p-value over a confidence interval for the nuisance parameter has been proposed: see Berger (1996), “More Powerful Tests from Confidence Interval p Values”, The American Statistician, 50, 4; exact tests having the correct size can be constructed using this idea.

Fisher’s Exact Test also arises as a randomization test, in Edgington’s sense: a random assignment of the experimental treatments allows the distribution of the test statistic over permutations of these assignments to be used to test the null hypothesis. In this approach the lady’s determinations are considered as fixed (& the marginal totals of milk-first and tea-first cups are of course preserved by permutation).

Source : Link , Question Author : Alby , Answer Author : Scortchi – Reinstate Monica

Leave a Comment