Verifying what the AI writes — the skill that pays

This is the lesson that separates people who use AI to build real bots from people who use AI to build convincing losses. A concrete checklist for auditing AI-written trading code.

If you remember one lesson from this part, make it this one. An AI will hand you code that runs and shows a Sharpe of 3. Your entire job — the thing that makes you worth more than the tool — is knowing whether to believe it. Here is how to audit AI-written trading code, top to bottom.

1. Hunt the look-ahead

The number one killer. Anywhere a decision is made, ask: *could this line know something it would not have known in real time?* The classic tell is using bar t to trade bar t with no shift.

class="c"># WRONG — decides todayclass="s">'s position from today's signal (future knowledge)
df[class="s">'position'] = (df[class="s">'sma_fast'] > df[class="s">'sma_slow']).astype(int)
df[class="s">'ret'] = df[class="s">'position'] * df[class="s">'close'].pct_change()      class="c"># look-ahead!

class="c"># RIGHT — act on yesterdayclass="s">'s signal, on today's move
signal = (df[class="s">'sma_fast'] > df[class="s">'sma_slow']).astype(int)
df[class="s">'position'] = signal.shift(class="n">1)                            class="c"># decided at t-class="n">1
df[class="s">'ret'] = df[class="s">'position'] * df[class="s">'close'].pct_change()

2. Confirm every API call is real

Hallucinated methods and parameters are common. Do not assume df.rolling(20).sharpe() exists because it sounds like it should. Check the docs, or ask the AI to cite the signature — and then still check. If a call fails on import or run, that is the cheap kind of wrong; the dangerous kind runs anyway with a wrong default.

3. Check the numbers are physically possible

• Does the win rate / return imply something absurd (90% win rate, no losing month)? Absurd usually means a bug, not an edge.
• Are costs actually applied? Set commission to something huge and confirm the equity curve gets worse — if it does not move, costs are not wired in.
• Does turning the signal OFF produce ~0 return? If a flat strategy still "makes money", your P&L calculation is broken.

class="c"># class="n">1. Costs must bite: crank them up and the curve must fall
assert backtest(cost_bps=class="n">100).total_return < backtest(cost_bps=class="n">0).total_return

class="c"># class="n">2. A no-trade strategy must earn ~class="n">0 (not free money)
flat = backtest(signal=pd.Series(class="n">0, index=df.index))
assert abs(flat.total_return) < 1e-class="n">9

class="c"># class="n">3. Shifting the signal one bar EARLIER should HELP if you have look-ahead
class="c">#    (peeking further). If class="s">"cheating more" boosts returns, you have a leak.

4. Make it defend itself

A surprisingly powerful move: ask the AI to critique its own code. Models are often better critics than authors.

Review the backtest you just wrote as a sceptical quant. List every
assumption it makes (costs, fills, data, alignment). Point out any
look-ahead or survivorship bias. Where would this be most likely to
overstate real-world performance? Don't defend it — attack it.

Do this often enough and it becomes reflex. You will read AI-written trading code the way an editor reads a draft: quickly, sceptically, and with a nose for the one line that ruins everything.