Can I trust a significant result of a t-test if the sample size is small?

If my one sided t-test result is significant but the sample size is small (e.g. below 20 or so), can I still trust this result? If not, how should I deal and/or interpret this result?

Answer

In theory if all the assumptions of the t-test are true then there’s no problem with a small sample size.

In practice there are some not-quite-true assumptions which we can get away with for large sample sizes but they can cause problems for small sample sizes. Do you know if the underlying distribution is normally distributed? Are all the samples independent and identically distributed?

If you doubt the validity of the test then an alternative you can make use of is bootstrapping. Bootstrapping involves resampling from your sample in order to see how often the null hypothesis is true or false. Perhaps your null hypothesis is μ<0 and your p-value is 0.05 but the bootstrapping shows that the sample mean is less than zero 10% of the time. This would indicate that it was a fluke which caused a p-value of 0.05 and you should be less confident that the null hypothesis is false.

Attribution
Source : Link , Question Author : Eric , Answer Author : amoeba

Leave a Comment