Monitoring, logging & alerting
What to log and watch in production — fills, errors, equity, drawdown, data staleness — plus health checks and pushing real-time alerts to your phone so you are never surprised.
A live trading bot is a process running unattended on a server, making financial decisions while you sleep. The difference between a professional operation and an accident waiting to happen is observability: at any moment you can answer "what is it doing, is it healthy, and did anything just go wrong?" This lesson wires that up.
Log everything that matters (and nothing that does not)
Logs are the flight recorder. When something goes wrong at 3am — and it will — the logs are the only witness. Log *decisions and events*, not noise. A good rule: if reading this line six months from now would help you understand a trade or a bug, log it.
- Every signal — timestamp, instrument, the computed value, and the decision (enter/exit/hold).
- Every order — submitted, its price/size, and the fill you actually got (fill price vs intended = your real slippage).
- Every error and exception — with the full traceback. Silent failures are how bots quietly die.
- Equity and open positions — periodically, so you can reconstruct the equity curve and drawdown after the fact.
- Data health — the timestamp of the last received tick/bar, so you can detect a stale or dead feed.
import logging, json, sys
from datetime import datetime, timezone
class="c"># Log to both a rotating file and stdout, as machine-parseable JSON lines.
logger = logging.getLogger(class="s">"bot")
logger.setLevel(logging.INFO)
handler = logging.StreamHandler(sys.stdout)
logger.addHandler(handler)
def log_event(kind, **fields):
class="s">""class="s">"One structured line per event — greppable and machine-readable."class="s">""
record = {class="s">"ts": datetime.now(timezone.utc).isoformat(), class="s">"kind": kind, **fields}
logger.info(json.dumps(record))
class="c"># Usage throughout the bot:
log_event(class="s">"signal", symbol=class="s">"EURUSD", value=class="n">1.83, decision=class="s">"enter_long")
log_event(class="s">"order", symbol=class="s">"EURUSD", side=class="s">"buy", size=class="n">0.01, intended=class="n">1.0921)
log_event(class="s">"fill", symbol=class="s">"EURUSD", side=class="s">"buy", size=class="n">0.01,
fill_price=class="n">1.0923, slippage_bps=class="n">1.8)
log_event(class="s">"equity", balance=class="n">10234.51, open_positions=class="n">1, drawdown_pct=class="n">2.3)Health checks: is it even alive?
The scariest failure is the silent one: the process is running, but the data feed died an hour ago and it is trading on stale prices — or it crashed and is not trading at all. A health check is a heartbeat plus a few sanity gates that the bot evaluates on every loop.
import time
class HealthMonitor:
def __init__(self, max_data_age_s=class="n">120, max_loop_gap_s=class="n">60):
self.max_data_age = max_data_age_s class="c"># feed considered stale after this
self.max_loop_gap = max_loop_gap_s class="c"># bot considered stalled after this
self.last_data_ts = time.time()
self.last_loop_ts = time.time()
def mark_data(self): self.last_data_ts = time.time()
def mark_loop(self): self.last_loop_ts = time.time()
def check(self):
now = time.time()
problems = []
if now - self.last_data_ts > self.max_data_age:
problems.append(fclass="s">"STALE DATA: {now - self.last_data_ts:.0f}s since last tick")
if now - self.last_loop_ts > self.max_loop_gap:
problems.append(fclass="s">"BOT STALLED: {now - self.last_loop_ts:.0f}s since last loop")
return problems class="c"># empty list == healthyAlerting: push, do not poll
You will not sit and watch a terminal — nor should you. The system must reach *out* to you when it matters. A dead-simple, robust choice is a Telegram bot or a webhook: your code makes one HTTP request and a message lands on your phone. Alert on three things, and only three, so alerts stay meaningful:
- Fills — you took a trade. Low-urgency, but you want the record on your phone.
- Exceptions — the bot hit an error. High-urgency; something needs a human.
- Drawdown / risk breaches — the account crossed a line you set. This is the one that gets you out of bed.
import os, requests, logging
class Alerter:
class="s">""class="s">"Push alerts to Telegram. Never let a failed alert crash the bot."class="s">""
def __init__(self):
self.token = os.environ[class="s">"TG_BOT_TOKEN"] class="c"># from @BotFather
self.chat_id = os.environ[class="s">"TG_CHAT_ID"]
self.url = fclass="s">"https:class="c">//api.telegram.org/bot{self.token}/sendMessage"
def send(self, level, message):
icon = {class="s">"info": class="s">"*", class="s">"warn": class="s">"!!", class="s">"critical": class="s">"!!!"}.get(level, class="s">"*")
text = fclass="s">"[{icon}] {level.upper()}\n{message}"
try:
requests.post(self.url,
json={class="s">"chat_id": self.chat_id, class="s">"text": text},
timeout=class="n">5)
except Exception as e:
class="c"># An alerter that crashes the bot is worse than no alerter.
logging.error(fclass="s">"alert failed: {e}")
alert = Alerter()
alert.send(class="s">"info", class="s">"Filled BUY class="n">0.01 EURUSD @ class="n">1.0923")
alert.send(class="s">"critical", class="s">"Drawdown class="n">8.2% breached the class="n">8% hard limit. Flattening.")Watching the equity curve live
Beyond point alerts, keep a rolling view of the two numbers that decide whether you stay in the game: equity and drawdown from the running peak. Drawdown is the one that hurts, and the one your kill switch (next lesson) watches.
You now have a bot that records what it does, notices when it is unhealthy, and taps you on the shoulder when it matters. That observability is the foundation for the next, sharper tool: knowing exactly when — and how — to pull the plug.