On This Page
Key Takeaways
- Reserve the evaluation observations before selecting the procedure. Freeze its trading, fitting and update rules before the final test.
- In-sample data is used to build and optimize rules, while the out-of-sample holdout is reserved to estimate how those rules generalize.
- Compare the development and evaluation results, then investigate selection, market exposure, data and execution differences.
- Walk-forward analysis fits on available past data and evaluates the next segment. Each result is out of sample relative to the fit used for that segment.
- Data used to choose a revised procedure becomes validation data for that procedure. Retain a separate final evaluation for the selected version.
What is out-of-sample testing?
Out-of-sample testing is the evaluation of a trading strategy on data reserved from its design. The test observations are set aside first, the procedure is selected and frozen using only the development data, and the frozen version is then evaluated on the reserved data. The result estimates how the rules perform on observations that played no part in choosing them.
The procedure can contain fixed parameters or a predefined schedule for refitting. Its information rules, update schedule, execution and accounting are fixed before the final evaluation.
Formula
For the positive Sharpe values in this example: Sharpe retention ratio = 0.55 / 1.80 = 0.306, or 30.6%.- Compare the same return frequency, risk-free convention and annualization. The ratio works when both Sharpe values are positive; at a zero or negative development Sharpe, compare the two values directly.
- Exclude the final evaluation data from choices about features, parameters, rules and the evaluation stopping rule.
- A walk-forward procedure can refit on past observations according to its predefined schedule. An earlier test observation may legitimately enter a later training window after it becomes available.
Out-of-sample testing example: a chronological 70/
A researcher has 10 years of daily data, about 2,520 trading days. They reserve the last 30% as an untouched out-of-sample set, tune the strategy only on the first 70%, then run the frozen rules once on the holdout.
- 2,520 × 0.70 = 1,764 in-sample days for design and tuning
- 2,520 × 0.30 = 756 out-of-sample days held out
- In-sample Sharpe 1.80 versus out-of-sample Sharpe 0.55
- Sharpe retention: 0.55 / 1.80 = 0.306, or 30.6%.
The split uses 1,764 development days and 756 reserved evaluation days. The Sharpe values are 1.80 and 0.55, giving 30.6% retention. Review the two records and the selection process to explain the difference.
The development Sharpe is 1.80 and test Sharpe is 0.55. Their ratio is 0.306, or 30.6%. Use the same return frequency, risk-free convention and annualization for that comparison.
A 30.6% retention means the strategy kept less than a third of its development Sharpe on new data; the review below explains why. When the development Sharpe is zero or negative, or the periods differ in length, compare the values on a common horizon instead of as a ratio.
How to keep future information out of the fit
Fit learned transformations and selected parameters within their permitted training data. Check when outcome labels finish so future outcomes do not enter an earlier fit.
A predefined moving average or adaptive rule can update using observations as they arrive during evaluation. The rule is what you freeze; its moving values keep updating as the test runs.
Apply the same position accounting and cost definitions to the development and evaluation records, using the period-specific inputs called for by those definitions.
Validation data vs final evaluation data
If a result guides a model revision, that dataset has become validation data for the revision. Preserve a separate final assessment for the selected version.
In walk-forward testing, each segment is evaluated by a fit made before that segment. Once those observations become historical, they can enter a later training window under the declared schedule.
Why in-sample and out-of-sample results differ
Compare net returns, drawdowns, exposures, trades and implementation costs. Review the candidate search and market conditions alongside the numerical gap.
Use forward evaluation to check signal availability, orders and fills under the same procedure. The holdout tests the rules on unseen history; the forward record tests whether you can trade them.
Limitations of out-of-sample testing
- A holdout is one historical period. Its 756 days carry their own market conditions and sampling variability, so start live trading at reduced size and let the forward record confirm the 0.55.
- Run the final test on reserved data once. After its result guides a revision, it becomes validation data for that revision, and repeated peeking turns the holdout into part of the search.
- Clean the data before the split. Look-ahead in features or labels, or a survivor-only universe, contaminates the development and test periods alike.
- Read the 30.6% retention as the size of the drop. Find its cause in the selection review, and compare raw Sharpe values when the development figure is zero or negative.
- Forward evaluation tests live signal timing, orders and fills; run it before committing full capital.
Bottom line
Out-of-sample testing checks whether a strategy’s edge holds on data that played no part in building it. Reserve the test data before research starts, freeze the full procedure before the final evaluation, and treat any data that guided a revision as validation data rather than a final test.
Clear answers
Frequently Asked Questions
Sources and Further Reading
- CFA Institute Research Foundation: Investment Model Validation
- Federal Reserve (SF): The Anatomy of Out-of-Sample Forecasting Accuracy
- CFA Institute: Backtesting and Simulation
How entries are sourced, calculated and updated: editorial methodology.