Anti-Overfitting

Anti-Overfitting Safeguards

CRITERION uses a complexity budget and validation framework to reduce data mining risk.

Complexity Budget

The strategy engines define approximately 27 numeric parameters (excluding enabled). Default safeguards require:

• Min trading days: 27 × 252 ≈ 6,804 days (~27 years)
• Min trades: 27 × 10 = 270 trades

These thresholds (min_trading_days_per_parameter, min_trades_per_parameter in overfitting_safeguards) are intentionally strict. Most backtests will exceed the budget; the researcher must consciously relax them in config, not have them silently pass.

Verdict Interpretation

When running criterion backtest with a config, the run summary reports:

• Parameters: From parameter inventory (strategy_engines)
• Trades / trading days: From the bundle
• Budget exceeded: true when either min trades or min days is not met
• Warnings: Human-readable descriptions of exceeded thresholds

If halt_on_budget_exceeded is true, the backtest aborts when the budget is exceeded. If false (default), it continues and warn_on_budget_exceeded emits a warning.

Worked Example

5-year backtest, 27 parameters:

• Trading days: 1,260
• Trades: 180

Required: 6,804 days, 270 trades.

Result: Budget exceeded. Both thresholds fail. The backtest may not have sufficient data to support its parameter count. Options: shorten the parameter set, extend the data period, or consciously relax safeguards in config (with documented rationale).

OOS Validation

criterion oos-validate splits trades into in-sample and out-of-sample by fraction or date. It reports performance on both periods. Degradation from IS to OOS suggests overfitting.

Walk-Forward Validation

criterion walk-forward runs rolling or anchored folds. Each fold trains on an in-sample window and evaluates on an out-of-sample window. Configuration: overfitting_safeguards.walk_forward (mode, is_window_days, oos_window_days, min_folds, step_days).