Trading Strategy Backtesting Methodology

Stingray backtests are designed to answer a narrow question: how did a specific, typed market rule behave over a stated historical window? The result is a replay artifact for evaluating a signal, not a prediction and not a complete portfolio simulation.

This methodology explains what is fixed before a replay starts, how historical data is evaluated, how triggers and forward returns are counted, and which limitations must remain visible. Crypto markets appear in many current examples because their venue, funding, order-book, and onchain data are unusually observable. The same method applies to other markets as reliable data and settlement activity move onto programmable rails.

Methodology at a glance

Question	Method	Evidence in the artifact
What was tested?	Freeze the typed rule, market, source, interval, window, and cooldown before replay.	Rule definition and replay metadata
Was the data usable?	Resolve each source, preserve its timestamps, and disclose gaps or trimmed ranges.	Coverage notes and source metadata
When did the rule fire?	Evaluate historical samples in order and separate raw truth from distinct events and cooldown-filtered notifications.	Fire sets, counts, and event rows
What happened next?	Measure price changes at declared horizons after each eligible fire.	Forward returns with sample sizes
Can the result be trusted?	Keep the artifact reproducible and expose assumptions, unsupported inputs, and missing observations.	Semantic version, diagnostics, and limitations
What happens after review?	Reject, refine, or monitor the rule before any execution decision.	Notify, preview-confirm, and opt-in activation path

1. Freeze the rule before replay

A useful backtest begins with a rule that can be inspected. Stingray resolves a plain-English thesis into a typed definition before historical evaluation starts. The definition should identify:

• The market, symbol, and venue or data source
• Every condition and threshold
• The evaluation interval and historical window
• The cooldown or deduplication policy
• The direction and forward-return horizons used for interpretation

The replay must not silently change the rule to produce a cleaner result. If a requested source is unavailable, a time parameter is incompatible with the historical interval, or the definition cannot be evaluated faithfully, the correct result is an explicit limitation or rejection.

This separation matters because changing a threshold after seeing the result creates a new rule. The revised rule should receive a new replay rather than overwriting the evidence for the previous one.

2. Resolve data sources and time correctly

Each condition needs a replay-safe source contract. Price, funding, open interest, event, and news inputs have different clocks and coverage. The replay records the source used and evaluates observations in timestamp order.

For event or news data, the relevant time is when the information became available to the system, not an earlier timestamp later attached to the event. This avoids using information before it could have been observed.

The artifact should disclose:

• The requested and effective date range
• The native or selected interval
• Missing internal segments and trimmed edges
• Sources that were sampled, aggregated, or unavailable
• Whether the available history was sufficient to evaluate the rule

A clean headline number cannot repair incomplete coverage. When gaps split a replay into valid segments, those boundaries should remain visible rather than being filled with invented observations.

3. Warm up indicators before counting fires

Indicators and stateful conditions need history before their values are meaningful. A moving average, RSI, sequence, or cross condition cannot be evaluated fairly from the first row of a dataset.

Stingray uses warmup observations before the visible evaluation window where the source and rule require them. Warmup data initializes indicator and edge state, but warmup fires do not count as user-visible evidence. The artifact records the warmup boundary so the first visible fire is not created by an uninitialized indicator.

Evaluation granularity must also fit the rule. A five-minute condition cannot be reconstructed honestly from hourly samples. When the historical interval is too coarse for the rule’s smallest time parameter, the replay should fail clearly instead of implying false precision.

4. Separate truthy samples from actionable fires

A condition can remain true across many consecutive samples. Counting every true row as an independent signal exaggerates evidence and can make one market episode dominate the result.

Current replay artifacts distinguish three views:

• Raw: every evaluated sample where the whole rule is true. This is diagnostic.
• Primary: transitions where the whole rule moves from false to true. These represent distinct entries into the condition.
• After cooldown: primary fires that remain after the configured notification cooldown. This is the closest view of live alert behavior.

Cooldown is an operational control, not a claim that suppressed fires were economically irrelevant. Showing all three views lets a reviewer see whether the rule detects distinct events or simply stays true for long periods.

For legacy signal replays that expose raw and cooldown-filtered sets without a separate primary set, the interpretation is the same: clustered rows should be treated as one market episode unless the rule explicitly defines otherwise.

5. Measure forward returns without turning them into P&L

Forward returns describe the percentage price change after an eligible fire at declared horizons such as 1 hour, 4 hours, or 24 hours. They answer, “what tended to happen after this condition appeared?”

They do not include a complete trading strategy unless the artifact explicitly models one. Standard signal replay does not assume:

• Entry or exit execution
• Position size or leverage
• Fees, spread, slippage, or market impact
• Stop-loss or take-profit behavior
• Portfolio constraints or capital allocation

The artifact should show the number of observations behind every summary. Distinct events should contribute fairly so a long cluster of nearly identical samples does not outweigh every other episode. Average, median, positive-share, and distribution shape are more useful together than any single headline statistic.

Forward data is only read after a fire has been determined. It is an outcome measurement and never an input to the historical trigger decision.

6. Keep one replay artifact internally consistent

A replay should be reproducible from its stored inputs: rule version, source contract, market identity, interval, window, cooldown, and replay-semantics version.

Changing a chart range should read another window from the same artifact where possible. It should not quietly execute the rule again and create different counts. Charts, fire tables, summary metrics, and selected-event details should all use the same fire set and visible window.

When source history or replay semantics change materially, the result should receive a new version. Reproducibility does not mean that all future data vendors will return identical history. It means the assumptions behind a given artifact are explicit enough to audit.

7. Review the result as evidence, not a verdict

Read a replay in this order:

1. Confirm the rule is the rule you intended to test.
2. Check source coverage, interval, and warmup before reading returns.
3. Compare raw, primary, and after-cooldown counts.
4. Inspect clustering and the exact fire rows.
5. Read forward returns with sample size and regime context.
6. Test a genuinely revised hypothesis as a new rule.

Useful questions include:

• Did most fires occur in one volatility or liquidity regime?
• Is the result dominated by one event or a small sample?
• Does the chosen horizon match the thesis?
• Would another venue, source, or interval change the interpretation?
• Was the rule selected before the window was inspected, or tuned repeatedly to it?

Repeatedly optimizing a rule on one historical window increases overfitting risk. A promising result should be checked on a later or otherwise separate period before it is treated as monitoring evidence.

8. Use the trust ladder after backtesting

The normal outcome of a backtest is one of three decisions:

• Reject: the rule is noisy, rare, fragile, unsupported, or regime-dependent.
• Refine: change one explicit assumption and generate a new replay.
• Monitor: run the typed rule as a live alert and compare observed behavior with the historical artifact.

Execution remains a separate decision. Stingray’s trust ladder is notify first, preview-confirm where supported, and opt-in execution only when the user has explicitly enabled it. A positive historical average does not skip that ladder.

Known limitations

• Historical performance does not predict future results.
• Available history varies by source, venue, asset, and event type.
• Candle or sampled data cannot prove the exact intrabar order of events.
• Forward returns measure outcomes after signals, not achievable trade returns.
• Backtests can miss structural changes, delistings, liquidity constraints, and future market rules.
• Multiple tests on the same window increase the chance of finding a result by accident.
• Unsupported or insufficiently covered inputs should be rejected, not approximated silently.

Publication checklist

Before a Stingray replay is used in a guide, example, or benchmark, the publication should include:

• The exact thesis or prompt
• The typed rule or a faithful readable summary
• Market, venue, sources, interval, and date window
• Coverage and warmup notes
• Raw, primary, and after-cooldown counts where available
• Forward-return horizons and sample sizes
• Clear exclusions for fees, slippage, sizing, and execution
• A statement that the result is historical evidence, not trading advice or a performance promise

For the field-level reading guide, see How to Read a Backtest Card. For published walkthroughs, see Strategy Examples. For the wider thesis-to-monitor workflow, see How Stingray Works.