The trial ledger

What counts as a trial, why the count only rises, and why you cannot supply it yourself.

What a trial is#

A trial is one evaluated configuration. Concretely:

The asymmetry between rows two and three is the most important thing on this page. One specification tested across many assets is one hypothesis evaluated against a larger sample. Two thousand specifications tested to find the best one is two thousand hypotheses, and the maximum of two thousand noisy draws is large even when every underlying edge is zero.

Why the count only rises#

The trials table is append-only, enforced by database triggers that refuse DELETE and UPDATE. There is no supported path — through the MCP tools or otherwise — to lower N.

This is not distrust of the user. It is a design response to how research actually degrades:

• Sessions span weeks and many conversations.
• Context windows get compacted; what was tried is forgotten.
• Sweeps get re-run after a parameter tweak nobody wrote down.
• The person computing the deflated Sharpe is often not the person who ran the sweep.

By the time a correction is applied, the honest N is unrecoverable by memory. So the server remembers instead.

You cannot pass N#

stats_deflated_sharpe(strategy_id="...")     # no n_trials parameter exists

There is no n_trials argument. The server reads the ledger for the family that strategy belongs to. This is the single load-bearing constraint in the whole system: an API that accepted N would be an API where the correction is whatever the caller wants it to be.

Forks inherit their history#

family_fork(parent="btc-ma", justification="...")

A fork starts a new family but carries the parent's trial count forward. A sibling fork was fitted on the same data over the same hypothesis space, so its trials are part of your multiple-testing burden whether or not you rename the family.

family_fork requires a written justification. It is recorded. The intent is not to make forking hard — sometimes a genuinely new idea grows out of an old family — but to make it deliberate and auditable rather than a reflex when a hurdle gets uncomfortable.

Reading your own count#

experiment_ledger(family="btc-ma")

Returns N, the Sharpe distribution across recorded runs, and the extremes. Read-only, append-only underneath.

N is per-tenant under row-level security and monotonic. No number in this documentation is your N — a figure baked into a document would be stale for whoever produced it and simply false for everyone else.

The recorded example#

The server's own btc-ma family shows what the ledger is for.

Twenty-eight recorded training runs on Bitcoin produced a best Sharpe of 0.9497. Taken alone that is a publishable-looking number. But it is the maximum of 28 draws, and the deflated Sharpe against N=28 is a very different statistic from the raw one.

Then the same specification was run cross-sectionally over ten assets — one trial — and pooled to -1.376, with 3 of 10 positive and the best being AERO-USD at 1.19. The Spearman correlation between per-asset Sharpe and per-asset trade count was -0.61: assets that traded less scored better, so the ordering tracked turnover and its cost rather than signal.

Every figure above is recomputable from that run's stored metrics.per_asset. That is the standard the ledger holds itself to — an earlier version of the server's own instructions quoted a pooled -1.86 over 22 assets and named BTC as the distribution maximum, and none of it matched any recorded run. The ledger held one cross-sectional run, over ten assets, and BTC was not among them. The text was wrong; the ledger was not.

Read this page as Markdown: /docs/concepts/ledger.md