Splits and the holdout

Immutable ranges, a frozen gap policy, and one evaluation per strategy for the lifetime of that strategy.

Register once#

split_register(
  train   = ["2021-01-01", "2024-06-30"],
  test    = ["2024-07-01", "2025-06-30"],
  holdout = ["2025-07-01", "2026-06-30"],
)

Ranges are immutable. There is no split_update. If a split turns out to be wrong you register a new one, which is visible as a new split rather than as a silent revision of the old one.

The reason is that a movable boundary is not a boundary. If the test period can be adjusted after seeing test results, the test period is training data with extra steps.

The gap policy is frozen at registration#

Market data is shared between tenants — bars are the expensive resource and OHLCV for BTC-USD is the same fact for everyone. But the policy for handling gaps in that data is not shared: a split records the gap policy in force at the moment it was registered.

Without this, a later change to how missing bars are treated would retroactively alter what an already-registered backtest saw, and a result would stop being reproducible without anything in the ledger changing.

Set the policy explicitly if you care:

data_gap_policy_set(policy="forward_fill_max_3")

Purging and embargo#

Cross-validation on time series leaks unless you actively prevent it. A label computed at time t with horizon h depends on data up to t+h, so a training example at t+1 overlaps a test label at t even though t+1 is strictly later.

The server purges both edges of each test block. Test labels run to hi + h, so training events after the block up to hi + h overlap them. Purging only backwards — which is the common implementation — leaks that right edge whenever the label horizon is positive.

An embargo period after each test block further separates the folds.

The holdout is unreadable#

Holdout data cannot be read by any tool until the strategy is frozen:

gate_check(strategy_id="...")      # are you ready?
strategy_freeze(strategy_id="...") # spec hash is locked
holdout_evaluate(strategy_id="...")# once, ever

holdout_one_shot is unique on (owner_id, strategy_id). The grant is checked against the ledger, not merely held in application memory, so a retry, a reconnect or a second client cannot produce a second evaluation.

Note the owner_id in that uniqueness constraint. Scoped to strategy_id alone, one tenant could have spent another tenant's only attempt.

Why one shot#

The holdout's value is entirely in never having influenced a decision. Look at it twice and the second look is conditioned on the first — you now know something about the holdout, and any subsequent choice is fitted to it.

At that point it is a test set, and you no longer have a holdout at all.

Run gate_check first. It reports whether the in-sample evidence justifies spending the attempt, and it costs nothing to call.

Read this page as Markdown: /docs/concepts/splits.md