On This Page
- What is a z-test?
- How is a z-test calculated?
- Worked example
- Two-sided vs one-sided z-tests and the t-test
- Z-test example: 60 wins in 100 trades
- P-values, power and sample size
- Testing many trading rules: the multiple-testing problem
- Statistical significance vs profitability
- Limitations of the z-test
- Bottom line
- FAQ
Key Takeaways
- A z-test measures how many standard errors separate an observed statistic from the value assumed under the null hypothesis.
- The test uses a statistic whose null distribution is standard normal or has a justified normal approximation. A known standard deviation alone does not make an arbitrary small-sample mean normal.
- Common two-tailed critical values are 1.96 at the 5 percent level and 2.576 at the 1 percent level, with 1.645 for a one-tailed 5 percent test.
- Proportion tests need a large enough sample, and one published criterion is that the smaller of n times p0 and n times (1 minus p0) is at least 5.
- Statistical significance describes how surprising a result is under the null hypothesis, not how large or profitable the effect is.
What is a z-test?
A z-test is a hypothesis test that expresses the difference between an observed statistic, such as a sample mean or a win rate, and its null value in standard errors. Its p-value comes from a standard normal null distribution or a specified normal approximation. Traders use it to check whether a result such as 60 wins in 100 trades is unusual under a stated null hypothesis.
How is a z-test calculated?
For a mean with known population standard deviation, Z = (sample mean − null mean) / (sigma / √n). For a proportion, Z = (observed proportion − p0) / √(p0 × (1 − p0) / n).
Formula
Mean test: Z = (x̄ − μ0) / (σ / √n). Proportion test: Z = (p̂ − p0) / √( p0 (1 − p0) / n )- x̄ is the sample mean and p̂ is the sample proportion of successes.
- μ0 and p0 are the values assumed under the null hypothesis.
- σ is the population standard deviation, treated as known.
- n is the number of independent observations, and σ / √n is the standard error of the mean.
Worked Example
Test whether a strategy’s win probability differs from 50%. In 100 independent Bernoulli trials, 60 are wins. Set H0: p = 0.50, H1: p ≠ 0.50 and alpha = 0.05 before applying the uncorrected two-sided proportion z-test.
- Sample proportion: p̂ = 60 / 100 = 0.60
- Standard error: √(0.50 × 0.50 / 100) = √0.0025 = 0.05
- Test statistic: Z = (0.60 − 0.50) / 0.05 = 2.00
- Uncorrected normal-approximation p-value: 0.0455; |Z| = 2 exceeds the 1.96 cutoff.
- Sample-size check: min(100 × 0.50, 100 × 0.50) = 50, comfortably above 5
The uncorrected z-test rejects H0 at 5%. The exact two-sided binomial calculation gives p = 0.0569 and does not reject at 5%. Same 60 wins, two verdicts, so choose the test before you read the result. Evaluate the payoff and trading costs separately from win probability.
Two-sided vs one-sided z-tests and the t-test
A two-sided alternative asks whether a value differs from the null in either direction. A one-sided alternative specifies one direction. At a 5% level, two-sided normal cutoffs are ±1.96; the upper one-sided cutoff is +1.645.
For independent normal observations, the known-sigma mean test is exact. With sample standard deviation, the corresponding t-test has n − 1 degrees of freedom. Large-sample inference uses the appropriate standard error for the observation structure.
Z-test example: 60 wins in 100 trades
The example tests H0: p = 0.50 against H1: p ≠ 0.50. The observed proportion is 0.60. Null standard error is √(0.50 × 0.50 / 100) = 0.05, giving Z = 2.
The uncorrected normal approximation gives a two-sided p-value of 0.0455, below 0.05. The exact two-sided binomial calculation sums outcomes at least as extreme as 60 wins under the symmetric 50% null and gives 0.0569, above 0.05.
A continuity-corrected normal calculation gives 0.0574. Name the test beside the result, because the verdict at 5% changes with it.
P-values, power and sample size
A p-value is the null probability of a result at least as extreme under the selected test. It is not the probability that the null hypothesis is true.
Power is the probability of rejecting at a specified alternative. With p0 = 0.50, p1 = 0.60, two-sided alpha = 0.05 and target power 80%, the standard normal sample-size approximation gives 193.85, rounded up to 194 trials. Exact discrete designs use their exact rejection rule.
For a fixed variance and the same independent sampling structure, quadrupling sample size halves the standard error. Dependence enters the variance through covariance terms and can raise or lower it.
Testing many trading rules: the multiple-testing problem
For 20 true nulls, each tested at a 5% level, the expected number of false rejections is one. If those tests are independent, the probability of at least one rejection is 1 − 0.95²⁰ = 64.15%. A positive expected count does not guarantee a rejection in every experiment.
Define the hypothesis family and apply its selected adjustment. Reserve data for evaluation after selecting the rule.
Statistical significance vs profitability
Win probability and profit are different inputs. Combine average wins, average losses and trading costs to calculate expectancy and the break-even win rate.
Limitations of the z-test
- Run the exact binomial test on win counts, above all in small samples. The z-test relies on a normal or approximately normal sampling distribution, and with small samples or discrete outcomes the approximation can mislead: for 60 wins in 100 trades it rejects at 5% while the exact binomial test does not.
- Use a standard error that matches the dependence between trades. The test assumes independent observations, and related trades change the standard error, so the same Z can overstate or understate the evidence.
- Switch to the t-test when you estimate the standard deviation. The exact mean test needs a known population standard deviation.
- Judge profit with expectancy. A significant result measures surprise under the null, not effect size or profit, and a p-value is not the probability that the null is true.
- Adjust for every rule you test. Testing many rules inflates false positives: 20 true nulls at 5% give a 64.15% chance of at least one rejection when the tests are independent.
Bottom line
A z-test measures how surprising a mean or win rate is under a stated null hypothesis, in units of standard error. Fix the hypothesis, sidedness and sample size before looking at results, check independence and the normal approximation, and judge profitability with expectancy rather than the p-value.
Clear answers
Frequently Asked Questions
Sources and Further Reading
- NIST/SEMATECH e-Handbook: Are the Data Consistent with the Assumed Process Mean?
- NIST/SEMATECH e-Handbook: Does the Proportion of Defectives Meet Requirements?
- NIST/SEMATECH e-Handbook: What Are Statistical Tests?
- NIST/SEMATECH e-Handbook: Cumulative Distribution Function of the Standard Normal
How entries are sourced, calculated and updated: editorial methodology.