At 13:42 on a Friday my bot sent an entry order that the broker rejected for insufficient margin. The rejection was handled — that part worked. What did not work was what happened next.
What happened, in order
The retry logic treated a rejection the same way it treated a network timeout, so it tried again. And again. Eleven times in nine seconds, until the position sizing check finally caught up and halted the strategy. Nine attempts were rejected too. Two went through, at prices four rupees apart, leaving me with double the intended position in a stock I had already decided to size small.
Why the retry loop existed at all
The retry was added for a real reason. Broker APIs drop connections, and a timeout does not tell you whether the order reached the exchange. The mistake was lumping every failure into one category.
- Transient failures — timeouts, connection resets, 5xx responses. Retrying is reasonable, with backoff.
- Terminal failures — insufficient margin, instrument in the ban list, price outside the circuit. Retrying cannot help and can hurt.
- Ambiguous failures — the request went out, the response never came. Never retry blindly; reconcile first by fetching the order book.
The fix
Classify the error before any retry, and let only the transient class through. Everything else raises, gets logged with the broker's own message, and halts the strategy for the day if it repeats. I also added a hard ceiling that no code path can bypass: a maximum of three attempts per signal, counted in the trade registry rather than inside the function doing the retrying. The bug happened partly because the counter lived in the wrong place.
How I found it
Forty minutes, because I had the session replay. Every decision the bot makes is recorded with the state it saw at the time, so I could rerun the afternoon and watch the loop happen. Without that I would have been reading log files and guessing.
If you build one piece of infrastructure before going live, make it this one. Not the backtester, not the dashboard — the ability to replay exactly what your bot saw and decided.