Tool lesson

Backtest: Stress-Test Against Overfitting

A practical Backtest lesson for spotting the temptation to tune history, then using period windows, cross-metal checks, and research-trail saves to keep evidence honest.

14 minIntermediate5 chapters

Educational only

The examples teach workflow and risk framing. They do not provide trade recommendations, personalized advice, leverage guidance, or guaranteed outcomes.

Chapter 01

Name the temptation to tune history

Trader question

Am I improving the rule, or teaching it to memorize one sample?

Overfitting is winning history by memorizing it. A learner should pause when every disappointing result leads to another threshold change before any wider validation.

Desk checklist

Limit parameter changes to a named research version.
Write why a threshold changed before running again.
Do not treat the prettiest chart as the final answer.

Interactive proof

Strategy Builder threshold fields and Run Backtest button

Use the Overfit Lab to tune the in-sample threshold, then reveal what happens to the holdout-style panel.

1Tune temptationA better in-sample chart can be a worse research habit.Repeatedly changing thresholds after every weak run can teach the strategy to memorize one slice of history.

2Window splitShort-window success should face a wider window.Compare the tuned result against longer periods and a holdout-style read before saving it as evidence.

3Cross-metal checkA rule that only works in one metal may be narrow, not wrong.Gold, silver, platinum, and palladium can stress the same logic in different market behavior.

4Research trailSave the hypothesis and versions, not only the prettiest chart.The Strategy Library should preserve what changed, why it changed, and what still needs validation.

Overfitting is not a moral failure. It is a research risk: the rule can memorize one slice of history. Stress it with wider windows, other metals, and a saved note that records what changed.

Interactive desk lab

Backtest Overfit Stress Lab

A practical Backtest validation lab for tuning an in-sample threshold, revealing holdout weakness, widening the period, and saving a research trail.

Native scroll

48s Remotion sceneTuneTemptationVideo

Tuning can memorize history

A threshold dial makes the in-sample chart prettier while a hidden holdout panel quietly weakens.

Storyboard beats4 cues

A simple RSI threshold starts with plain evidence.

The threshold is tuned after a disappointing result.

The in-sample curve improves with each tweak.

A holdout panel reveals that the fitted version became fragile.

Remotion code

TuneTemptationVideo

The snippet is stored with the lesson so a future Remotion project can render the chapter video.

Show component snippet

import {AbsoluteFill, Easing, Sequence, interpolate, useCurrentFrame} from "remotion";

export const TuneTemptationVideo = () => {
  const frame = useCurrentFrame();
  const dial = interpolate(frame, [20, 88], [0, 1], {
    extrapolateLeft: "clamp",
    extrapolateRight: "clamp",
    easing: Easing.bezier(0.16, 1, 0.3, 1),
  });
  const holdout = interpolate(frame, [92, 126], [0, 1], {
    extrapolateLeft: "clamp",
    extrapolateRight: "clamp",
  });

  return (
    <AbsoluteFill style={{background: "#fff8e8", color: "#071126", padding: 72}}>
      <h1 style={{fontSize: 52, margin: 0}}>A tuned chart can memorize the sample.</h1>
      <div style={{display: "grid", gridTemplateColumns: "0.8fr 1.2fr", gap: 28, marginTop: 64}}>
        <div style={{border: "1px solid #d9caa7", background: "#fffdf7", padding: 28}}>
          <p style={{fontSize: 18, color: "#5f6b7c"}}>RSI threshold</p>
          <strong style={{fontSize: 62}}>{Math.round(30 + dial * 12)}</strong>
          <div style={{height: 12, background: "#f6eedb", marginTop: 22}}>
            <div style={{height: 12, width: 40 + dial * 150, background: "#d9971f"}} />
          </div>
        </div>
        <div style={{position: "relative", border: "1px solid #d9caa7", background: "#fffdf7", padding: 28}}>
          <strong style={{fontSize: 28}}>In-sample result</strong>
          <div style={{height: 120, marginTop: 28, background: "#f6eedb", position: "relative"}}>
            <div style={{position: "absolute", left: 20, bottom: 30, width: 360 * dial, height: 5, background: "#047857", transformOrigin: "left"}} />
          </div>
          <Sequence from={94} layout="none">
            <div style={{marginTop: 22, border: "1px solid #b42318", background: "#fae4df", color: "#b42318", padding: 18, opacity: holdout}}>
              Holdout weakens after the same tuning.
            </div>
          </Sequence>
        </div>
      </div>
    </AbsoluteFill>
  );
};

Sources used for this tutorial

Backtesting and SimulationCFA InstituteUsed for validation framing around historical tests, sample selection, and disciplined interpretation.BacktestingCME Group EducationUsed for practical backtest caution around repeated parameter changes and market-period comparison.The Probability of Backtest OverfittingBailey, Borwein, Lopez de Prado, ZhuUsed for the core warning that repeated trial-and-error can produce a beautiful but fragile in-sample result.Investment Model ValidationCFA Institute Research FoundationUsed to frame holdout-style, cross-period, and cross-market checks as validation evidence.NFA Interpretive Notice 9025National Futures AssociationUsed for hypothetical-performance limitations and careful educational language around backtest results.

Next step

Open the tool with the checklist beside you.

Move from the lesson into the matching Bullion Brains tool, keep the checklist visible, and treat the output as evidence until the caveats are clear.

Open Strategy Backtester