Enter a z-score or t-statistic to calculate the p-value and determine statistical significance.
A p-value is the probability of observing a result at least as extreme as the one you got, assuming the null hypothesis (typically "there is no real effect or difference") is true. A small p-value means your observed result would be unlikely if there were truly no effect, giving evidence against the null hypothesis. A p-value of 0.03 means: if there really were no effect, you'd see a result this extreme (or more extreme) only 3% of the time by pure chance.
Use a z-score when your sample size is large (generally above 30) and you know or can reliably estimate the population standard deviation, the normal distribution is a good approximation here. Use a t-score when your sample size is small, since the t-distribution has heavier tails than the normal distribution, correctly reflecting the extra uncertainty that comes with estimating a population from limited data. The t-distribution requires degrees of freedom (typically sample size minus 1), and it converges to the normal distribution as degrees of freedom increase, which is why the difference matters most for small samples.
A two-tailed test checks whether your result differs from the expected value in either direction, "is the new version different from the old one, better or worse?" This is the more conservative and generally recommended default unless you have a specific, pre-registered directional hypothesis. A one-tailed test only checks one direction, "is the new version specifically better?" One-tailed tests produce smaller p-values for the same data, which is exactly why choosing the tail type after seeing your data (rather than deciding beforehand) is considered poor statistical practice, it inflates the apparent significance.
A p-value is not the probability that the null hypothesis is true, and it's not the probability that your result happened by chance. It's specifically the probability of your observed data (or more extreme) given that the null hypothesis is true, a subtle but important distinction. A statistically significant result (small p-value) also doesn't automatically mean a practically meaningful one, with a large enough sample size, even a tiny, unimportant effect can produce a very small p-value. Always consider effect size alongside statistical significance, not p-value alone.