Backtest Evidence and Its Procurement Limits

Identify the claim before checking the number

Ask whether the supplied series represents a fixed model replay, a changing research process, an illustrative portfolio or actual account results. Record the exact model version and the period covered. A current implementation applied to older prices is not necessarily a record of decisions that could have been made then. The SEC's performance-claims bulletin explicitly distinguishes backtesting from actual performance, a distinction that should remain visible in the buyer's evidence register and conclusions.

Request definitions for the universe, benchmark, return calculation and excluded instruments. A percentage can refer to individual trades, a fully invested portfolio or a portfolio with cash and overlapping positions. These are different objects. Check whether the displayed result incorporates dividends, financing and currency translation, and whether its stated denominator matches the underlying records. Avoid comparing suppliers until the quantities being compared have sufficiently similar meanings for the intended procurement decision.

Reconstruct information availability

For a sample decision, establish when each input was observed, released, received and revised. A historical label on a row does not prove the value was available at that time. The St. Louis Fed's real-time-period documentation explains how historical information sets differ from today's revised data. Use this as a concrete reason to ask for vintage handling, especially when economic releases or corrected fundamentals contribute to the model's historical signals.

Also examine changing index membership, delisted instruments and corporate actions. A present-day universe can exclude securities that failed or disappeared, while retrospective adjustments can change earlier values. Ask how the backtest handles these issues and what evidence supports the answer. Missing point-in-time information should become a stated limitation. Do not silently fill gaps with a current dataset and then describe the resulting reconstruction as a causal replay of information available to historical decision makers.

Understand selection and sample boundaries

Request the development period, tuning process, validation design and any later untouched evaluation period. Ask how many alternatives were considered and whether the final model was selected after inspecting the reported test results. A period can be outside the fitting sample yet still influence selection. The label out of sample does not eliminate that possibility. Keep the selection history attached to the result rather than reduce methodological credibility to one familiar label.

Check dependence among observations. Hundreds of trades can overlap in time, share instruments or respond to the same market event. The raw row count is therefore not automatically an independent sample size. A buyer need not invent a statistical correction without the necessary information; it should ask how uncertainty and concentration were assessed. Software tests that confirm arithmetic are useful, but they do not add independent market observations to the evidence supporting an investment claim.

Work through a hypothetical record reconciliation

A hypothetical buyer selects sixty synthetic transaction records to test a proposed evidence-review procedure. Forty-eight can be reconstructed under the documented convention, eight lack required cost fields and four have ambiguous input-availability times. The categories are mutually exclusive in this example, so 48 + 8 + 4 = 60. The fully reconciled share is 48 / 60 = 80%. No transaction returns or vendor performance claims are invented by this exercise.

The buyer records twelve unresolved records and asks targeted questions instead of calculating a reassuring average from the forty-eight that are easier to explain. Missing costs and ambiguous timing require different remedies. A later correction must retain the original record and identify the changed assumption where retention is permitted. Passing this reconciliation would establish consistency for the checked sample, not investment profitability, absence of selection bias or realistic execution across the entire backtest.

Read fill and cost conventions literally

Request the exact observation window, directional price rule, order assumptions and treatment of unavailable prices. Some research conventions deliberately report a best hypothetical price within a bounded window. Preserve that convention when reproducing the research, clearly label its optimistic nature where applicable, and do not present it as an actual fill. A separate execution sensitivity can be informative, but it should not silently replace the original rule or extend the window using later observations.

Inspect commissions, spread, market impact, borrowing, financing and capacity assumptions according to the strategy's activity. A flat cost deduction may answer a limited sensitivity question while failing to represent size-dependent execution. Keep gross and cost-adjusted results distinguishable and check any overlap or capital constraints. For external communications, obtain review of the actual presentation and applicable jurisdiction; a generic disclaimer or a technically accurate backtest does not settle every legal or regulatory question about its use.

State the evidential limit in the decision

Summarize what was reconstructed, which assumptions remain unverified and how those uncertainties affect the proposed use. A credible historical study can justify further evaluation without supporting immediate deployment. Conversely, an irreproducible headline may justify declining procurement without proving the underlying idea is impossible. Keep those conclusions distinct. Record the model identity and evidence version so later improvements do not retroactively make an earlier assessment appear more certain than it actually was.

Retain authorized sample records, questions and provider explanations with the decision. Where a critical input vintage or selection history cannot be recovered, state that limitation directly instead of filling the gap with a favorable narrative. Backtests help organize conditional evidence about a defined process. They do not establish that a buyer would have obtained the same fills, costs or returns, and they do not guarantee future results under market conditions absent from the historical study.

• Keep data availability and model selection histories visible.
• Reproduce the stated research convention without relabeling it as execution.

Source notes

1. SEC Investor.gov: Investor Bulletin on Performance Claims
2. Federal Reserve Bank of St. Louis: FRED API Real-Time Periods