I have been told that I need both significance and power for my AB results to be valid. I researched a lot for this and the above statement is not making sense. I get that we need high enough power to not reject the null hypothesis and assuming that the new feature has bought no actual effect, but why do we need power to reject the null hypothesis when my confidence interval is already so high?
My confusion is as below:
Power is (1-Beta). So higher the power, lower the probability of type 2 error (not rejecting the null hypothesis when it is false). The thing is, I am rejecting the null hypothesis as my results are very significant and alpha is already low.
Lower the alpha, more the sample size required at the same power: This further adds to my belief that you don’t need statistical power to reject the null hypothesis. I mean, are we really saying the more my confidence interval, the more data size i will need to validate the effect?
I am not sure if I am missing some key concept. Please help me out as I am pretty sure that the new feature has positive conversion and I have already reached 99.99% CI.
Power is generally something you calculate before you perform a study. For example, let’s say you are trying to test whether medication A is more effective than medication B. Because of some cost, each new participant is really expensive. So you calculate the minimum effect size you want to be able to detect (e.g. it lowers blood pressure by 10 points) and then determine from that information what sample size you would need to detect a 10 point difference in treatment. Let’s say the power analysis says you need 40 participants.
Now let’s say that the actual difference between treatment A and B is much larger than you minimum— say 30 points. You would be able to detect this difference with a much smaller sample size. The point of your power analysis is to set a minimum effect size you qualitatively feel you need to detect.
So, power analysis isn’t something you really ever do after a study, especially if your results are significant. If your results are significant, they’re significant. No strings attached (well, at least related to power).