# Is bootstrap problematic in small samples?

In “3 Things That Bother Me” (1988), Ed Leamer writes:

Bootstrap estimates of standard errors are based on the assumption that the observed sample is the same as the true distribution, which is OK asymptotically. But a sample of size $$nn$$ implies a distribution with $$nn$$ mass points, which is quite unlike the true distribution if $$nn$$ is small. For what sample sizes and what parent populations are the bootstrap estimates OK?

I had an impression that one of the main uses of bootstrap in statistics and econometrics is precisely in small samples. There, a bootstrap distribution is used when no analytical distribution is available and the sample is too small for the asymptotic distribution to be a good approximation of it. This makes Ed Leamer’s criticism quite relevant and interesting. But perhaps my impression is wrong and I am misunderstanding things.

Q: Is this a valid piece of criticism? If so, has the problem been studied in any detail? Have any solutions been proposed?

My short answer would be: Yes, if samples are very small, this can definitely be a problem since the sample may not contain enough information to get a good estimate of the desired population parameter. This problem affects all statistical methods, not just the bootstrap.

The good news, however, is that ‘small’ may be smaller than most people (with knowledge about asymptotic behavior and the Central Limit Theorem) would intuitively assume. Here, of cause, I’m referring to the normal (naive) bootstrap without dependent data or other peculiarities. According to Michael Chernick, the author of ‘Bootstrap Methods: A guide for Practitioners and Researchers’, small may be as small as N=4.

But this number of distinct bootstrap samples gets large very quickly. So this is not an issue even for sample sizes as small as 8.

For reference, see Chernick’s great answer to a very similar question: Determining sample size necessary for bootstrap method / Proposed Method

Of cause the suggested sample sizes are subject to uncertainty and no universal threshold for a minimum sample size can be specified. Chernick therefore suggests to increase the sample size and study the convergence behavior. I believe is a very reasonable approach.

Here’s another quote from the same answer, which somehow addresses the premise you quoted initially:

Whether or not the bootstrap principle holds does not depend on any individual sample “looking representative of the population”. What it does depend on is what you are estimating and some properties of the population distribution (e.g., this works for sampling means with population distributions that have finite variances, but not when they have infinite variances). It will not work for estimating extremes regardless of the population distribution.