Read Crypto Backtests Without Overclaiming

A backtest can produce an answer with two decimal places long before it produces strong evidence. The crucial questions concern what was measured, how many independent observations support it, and how the rule was chosen.

An event study is a useful starting point. It asks what happened after a specified condition in a historical dataset. It does not automatically model a strategy with executable entries, position sizes, costs and portfolio accounting. Confusing those two tasks can turn a descriptive table into an unsupported profit claim.

The historical query and its small sample

On 22 September 2026 at 19:31 UTC, CryptX asked the connected TraderSpy service for a BTCUSDT daily event study: RSI with a 14-period lookback crossing below 30, followed by returns after one, three and seven daily bars. We selected this familiar rule as a methodological example and did not search multiple thresholds to find a flattering result.

The response reported 1,000 daily candles, dated 27 December 2023 through 21 September 2026 by candle opening time, 985 evaluable bars, and nine occurrences. TraderSpy describes the requested market as a USDT perpetual. The returned study did not identify the underlying exchange or provide the complete raw tape and every episode needed for independent reproduction.

These are provider-reported historical event-study statistics, retrieved through TraderSpy. They are not CryptX’s independently rebuilt execution backtest. The provider explicitly warned that nine occurrences should be treated as anecdotal evidence. Its rounded one-day edge differs by 0.01 percentage point from subtracting the two displayed rounded means; we preserve the reported value rather than invent hidden precision.

A baseline changes the question

A positive mean following a condition can look encouraging even when the broader dataset rose faster. Comparing the conditional result with an unconditional result asks whether this selected event group differed from ordinary observations in the same tape.

Here, the three-day mean after the event was −1.39%, compared with an unconditional +0.28%, producing the reported −1.67 percentage-point edge. The negative difference describes this sample. It is not a forecast that the next occurrence will lose 1.67%, nor does it establish that reversing the condition creates a profitable strategy.

The baseline itself needs a definition. Does it use the same valid endpoint rules and observation period? Are its overlapping windows treated consistently? The connector’s aggregate output cannot resolve every such implementation detail. Missing methodology should narrow the claim rather than be filled with confident assumptions.

Nine episodes are not nine market regimes

The returned three-day median was −3.78%, below the mean of −1.39%. That gap shows why a mean alone can hide an uneven outcome distribution. The response also reported three positive outcomes out of nine at that horizon, expressed as a 33.3% win rate.

One additional positive or negative episode would materially change that proportion. More importantly, events can cluster. Two entries in the returned recent-episode list were only two days apart in November 2025, so their three-day and seven-day forward windows overlap. Treating every horizon observation as independent would exaggerate the amount of separate information available.

The rule is also a crossing rule. An RSI value remaining below 30 for several days is not the same as a new downward crossing each day. Replacing crossings with every below-threshold bar changes event counts, dependence and the question being tested.

A price path is not a fill history

If the condition becomes known only at a candle close, a realistic strategy must specify when an order could be placed and at what executable price. A calculation that observes the close and assumes a guaranteed fill at that same close can give itself information or execution quality it did not actually have.

A perpetual strategy would also need explicit funding, trading fees, spread, slippage and liquidation assumptions. None is established by a table of forward price changes. The CFTC’s discussion of hypothetical performance limitations highlights the gap between simulated results and actual execution, including liquidity and hindsight. Here that is a methodological caution, not a claim about which regulation governs a reader.

Research on backtest-engine correctness examines how candle-based simulations can go wrong. A candle’s high and low do not fully specify the order in which prices occurred, so assumptions about stops and targets can materially affect simulated results.

Selection can manufacture a convincing winner

Trying hundreds of coins, thresholds and holding periods and publishing only the best result creates a different evidential problem from testing one stated rule. The selected winner benefits from the search process itself.

The research on the probability of backtest overfitting develops a framework for studying that selection problem. The Deflated Sharpe Ratio paper similarly addresses selection bias and non-normal returns in evaluating performance. We have not calculated either statistic for this small connector output; citing the research does not turn the response into a validated strategy.

A stronger follow-up would preserve the original hypothesis, obtain a documented raw dataset, implement event and execution rules separately, account for overlapping windows, and evaluate genuinely unseen data. The useful conclusion from these nine episodes is that the question is testable and the limitations are visible. It is not a trading instruction.

For the mechanics behind execution differences, see our explainer on slippage and price impact.