`qe_daemon` IBKR-paper smoke checklist¶

Manual end-to-end verification that the wired daemon (EPIC-62 T62.4 / T62.5 / T62.6 / T62.11 + the epic/62-finish follow-ups) connects, warms up, ticks, and stops cleanly against a real IB Gateway. Not gated by ctest — the daemon's own unit tests cover the modules in isolation.

This page was rewritten after running the smoke against IB Gateway 151+ on 2026-06-04; the quoted log lines below are verbatim from that run.

Prerequisites¶

IB Gateway running in paper mode. The daemon supports both IB Gateway and TWS Desktop — pick the right port:

App	Port
Paper IB Gateway	4002
Paper TWS Desktop	7497
Live IB Gateway	4001
Live TWS Desktop	7496

Log in to the Gateway / TWS with your paper account. Confirm: - Configure → API → Settings → Enable ActiveX and Socket Clients is on. - Trusted IPs includes 127.0.0.1 (and your dashboard machine's LAN address if remote-attaching). - Read-Only API is off — otherwise order submission fails at the broker layer (status and positions still work). - Master API client ID is left empty.

Build: cmake --build --preset=release -j.
client_id budget. The daemon opens TWO concurrent IBKR connections from one process — a data path (the .qe's ibkr_connection(client_id = N) value) and a broker / order-routing path (N + 1). Pick an N such that neither N nor N + 1 collides with your dashboard, any TWS bookkeeping client, or a second daemon. The default N = 1 uses 1 and 2; smoke fixtures in this guide use N = 0 so they don't fight the dashboard's client_id = 1 default.

The smoke fixture¶

A minimal live(...) config pointing at the paper IB Gateway:

let base = backtest(
  data = file("tests/fixtures/csv/short_60days.csv"),
  strategy = signal(
    entry  = cross_above(sma(close, 5), sma(close, 20)),
    exit   = cross_below(sma(close, 5), sma(close, 20)),
    symbol = "SPY",
  ),
  execution = execution(capital = 100_000),
)

live(
  base    = base,
  broker  = "ibkr-paper",
  symbols = ["SPY"],
  ibkr    = ibkr_connection(host = "127.0.0.1", port = 4002, client_id = 0),
)

Save to /tmp/qe_smoke.qe (or anywhere). Adjust port to 7497 for TWS Desktop.

Cold start¶

./build/release/bin/qe_daemon start /tmp/qe_smoke.qe

Expected log sequence (verbatim from the verified run):

daemon: connecting to IBKR Gateway at 127.0.0.1:4002
daemon: broker session will connect with client_id=1 (data uses 0)
ibkr_connection: TWS msg [2104, req=-1]: Market data farm connection is OK:usfarm
ibkr_connection: TWS msg [2107, req=-1]: HMDS data farm connection is inactive but should be available upon demand.ushmds
ibkr_connection: TWS msg [2158, req=-1]: Sec-def data farm connection is OK:secdefil
daemon: broker session up — name = ibkr-paper
daemon: opening data-path connection to 127.0.0.1:4002
daemon: data-path IbkrConnection constructed, client_id=0
daemon: journal dir = ~/Library/Application Support/qe_daemon/state
daemon: cold start (no prior journal events)
daemon: control socket listening at ~/Library/Application Support/qe_daemon/daemon.sock
daemon: IBKR handshake OK — starting quote subscriptions
daemon: warmup needs 20 bars (driven by rolling_vol(20) in leg 0 (SPY) entry) → requesting "39 D", "1 day"
ibkr_connection: TWS msg [2106, req=-1]: HMDS data farm connection is OK:ushmds
daemon: historical warmup = 60 bars across 1 symbols (min 60 per symbol, needed 20)
daemon: entering live event loop
disconnect_watchdog: armed (poll=1000ms, pause_after=3 ticks, trip_after=30 ticks)
ibkr_quote_source: subscribed SPY → ticker_id=10000000
live_engine: started — broker=ibkr-paper, trade_symbol=SPY, subscribed=1 symbol(s), resolution=1m

The two TWS msg [2104/2107/2158] blocks each appear twice because both the broker and data-path connections receive the farm-connection notices independently. That's expected.

Paper-account quirk. After subscribe you may see TWS error [10167, ...]: Requested market data is not subscribed. Displaying delayed market data. — that's IBKR telling us a paper account without a paid real-time data subscription will get delayed tick types (74 / 75 / 76 — delayed volume / close / open). The daemon parses these as informational; no bars close outside RTH because delayed-summary types aren't trade ticks. To exercise bar closes during the smoke you need either regular trading hours, a paid market-data subscription, or running against TWS with replay enabled.

Control socket round-trip¶

In a second shell, while the daemon runs:

./build/release/bin/qe_daemon status

Expected — the bare verb prints a grouped human report, not JSON:

qe_daemon 0.4.2  ·  pid 41207  ·  up 18s
  strategy   live_spy
  broker     ibkr-paper · connected (data up · order up)
  mkt data   DELAYED (15 min)
  marks      refreshed 41s ago  ·  its clock read 16:42

SAFETY     all clear
  kill       not tripped
  pause      running
  day loss   armed · $0 of $15,000 used

ORDERS     nothing queued
  session    0 submitted · 0 filled · 0 rejected
  next eval  Fri 18 Sep 16:00 close  (in 5h 47m)

RECONCILE  clean
  drift      none now · 0 alert(s) since start

The JSON is behind a flag — status has printed the report since the status formatter landed, and anything scripting against field names wants:

./build/release/bin/qe_daemon status --json

./build/release/bin/qe_daemon tail events

(Stays idle outside RTH — every journal append flows through the events push channel; with no bars closing yet, none have been written. Inside RTH or after a manual position change you should see one JSON line per event.)

Graceful stop¶

./build/release/bin/qe_daemon stop

Expected:

ok: {"stopping":true}

And in the daemon log:

control: stop — flagging shutdown
live_engine: stopped — 0 bars processed, 0 orders submitted
daemon: shutdown complete (bars=0, orders=0, journal_events=0)

Exit code 0.

Verifying the disconnect watchdog¶

The watchdog is a 3-state machine since EPIC-70: Healthy → Paused → Tripped. The smoke tests both transitions.

Soft-pause + auto-resume (EPIC-70)¶

While the daemon is running, stop the API listener inside Gateway (Configure → API → Settings → uncheck "Enable ActiveX and Socket Clients", then OK). Within ~3 seconds the daemon log prints:

[warn]  disconnect_watchdog: broker not connected (1/30 consecutive misses, soft-pause at 3)
[warn]  disconnect_watchdog: broker not connected (2/30 consecutive misses, soft-pause at 3)
[warn]  disconnect_watchdog: broker offline for 3 polls — entering soft-pause (trip threshold 30 polls)
[warn]  pre_trade_risk: watchdog pause armed — new orders will reject until broker reconnects

qe_daemon status from a separate shell now reports

SAFETY     ⚠ PAUSED
  pause      PAUSED by watchdog · ibkr_disconnect_soft

or, under --json, "watchdog_state": "paused", "broker_connected": false, "paused_by_watchdog": true, "pause_reason": "ibkr_disconnect_soft". F6's venue: badge flips amber: PAUSED · watchdog · N missed poll(s) — N is the count of failed polls, not seconds.

Re-enable the API listener within 30 seconds. The watchdog catches it on the next poll and resumes:

[info]  disconnect_watchdog: broker reconnected after 4 consecutive misses
[info]  pre_trade_risk: watchdog pause cleared — order submission resumed

watchdog_state returns to "healthy", F6 venue: badge returns to green connected. No daemon restart needed.

Hard trip after sustained outage¶

Repeat the test but leave the API listener disabled past the 30 s trip threshold. After 30 polls:

[error] disconnect_watchdog: broker offline for 30 consecutive polls — tripping kill-switch + invoking on_trip
[warn]  watchdog: N staged slice(s) PRESERVED across the kill    # only if N > 0
[warn]  watchdog: kill-switch tripped on disconnect. Working orders at the venue: N.
        They are NOT cancelled — cancel them by hand in TWS / IB Gateway / Client Portal,
        then `qe_daemon stop`. See docs/live-trading-safety.md Layer 3.

That last line is the expected outcome, not a smoke failure. No order is cancelled at IBKR by this path, ever. EPIC-88 T88.2 deleted the venue-cancel loop that used to live here; what remains is a warning telling you to go and cancel by hand.

The count N is whatever list_open_orders() returned. Do not assume it is unknown: the watchdog polls the DATA link while that call travels the ORDER link on client_id + 1, so a data-only death can reach this callback with the order socket alive and report a real integer. The staged-slice line above it is separate and is real work: those are locally queued slices that were never submitted.

If you had a working order at IBKR when the link died, it is still working. Cancel it in TWS / IB Gateway / Client Portal. See Live-trading safety, Layer 3.

qe_daemon status now reports

SAFETY     ⛔ KILL TRIPPED — no new orders will be submitted
  kill       TRIPPED (one-way) · ibkr_disconnect

or, under --json, "kill": true, "kill_reason": "ibkr_disconnect", "watchdog_state": "tripped". F6 venue: badge is red TRIPPED (ibkr_disconnect). Re-enabling the API does NOT auto-unkill — KillSwitch is one-way for safety. Stop the daemon (F6 → Stop daemon or qe_daemon stop) and redeploy to clear.

Verifying broker reconcile¶

Run the smoke; let it process at least one journal event (a position transition, or trigger one via the dashboard's F6 TRADE panel).
qe_daemon stop.
From the Gateway UI, flatten the position or change its quantity.

Re-launch the daemon. Expected log:

daemon: broker reconcile found 1 drift row(s) — operator review recommended before placing new orders. Full report:
broker_reconcile: 1 DRIFT row(s) (journal=1 symbols, broker=0 symbols)
  SPY  journal=100  broker=0  Δ=-100  [ghost — journal has it, broker doesn't]

The same row also lands in the journal as a Notice event (visible via qe_daemon tail events).

Restart-recovery check¶

# 1. Run for at least one journal-event-producing window.
./build/release/bin/qe_daemon start /tmp/qe_smoke.qe

# 2. Inspect the journal directly.
cat ~/Library/Application\ Support/qe_daemon/state/events.jsonl | tail

# 3. Stop + restart. Expect:
#    "daemon: replayed N journal events; restored M positions,
#     bar_count=..., last_equity=..."
./build/release/bin/qe_daemon stop
./build/release/bin/qe_daemon start /tmp/qe_smoke.qe

The broker-reconcile pass runs automatically on every restart after the IBKR handshake. Drift-clean runs log a one-liner: daemon: broker reconcile clean (journal=N sym, broker=N sym).

Verifying the cross-sectional barrier (EPIC-69)¶

Only meaningful when the smoke .qe is a signalize_universe (or otherwise uses is_top / is_bottom / quantile / rank). For a single-symbol smoke this section is a no-op.

The barrier shows up in the daemon log as [xs_batch:reason] suffixes on every ENTRY/EXIT line. Three reasons:

`reason`	When	Operator action
`quorum`	Every universe symbol produced a bar for the current `close_ts_ns`. The expected, healthy case.	None
`new_ts`	A bar at a fresh `close_ts_ns` arrived while the prior batch was still pending (some symbol never ticked at the prior ts). Prior batch force-flushed with stale slots.	Investigate why a symbol skipped a bar — usually low liquidity.
`timeout`	5 s wall-clock elapsed since the batch started without quorum. WARN-logged.	Same as `new_ts` — check the missing symbol's tick stream.

What quorum looks like for a 30-symbol cross-sectional close:

[info] live_engine: ENTRY AAPL (AAPL) qty=1 @ ts_ns=... [xs_batch:quorum]
[info] live_engine: ENTRY MSFT (MSFT) qty=1 @ ts_ns=... [xs_batch:quorum]
... up to 10 entries (top_k=10) all at the same ts_ns ...

All K=top_k entries land in one flush at the same close_ts_ns. The prepass computed ranks against fresh data for every universe symbol — orders reflect the actual top-K, not an arrival-order race.

What timeout looks like:

[warn] xs_barrier: timed out waiting for 27/30 symbols at ts_ns=... — dispatching with stale slots (rank quality degraded)
[info] live_engine: ENTRY AAPL (AAPL) qty=1 @ ts_ns=... [xs_batch:timeout]

The 3 missing symbols' contexts fall back to whatever last_ctx had (prior bar's close, or NaN if never observed). The prepass returns 0 for NaN slots in is_top / is_bottom so half-formed ranks don't fire trades against NaN-valued symbols — timeout flushes that don't produce any entry log lines are working as designed (defensive bail-out).

Staged entry / exit smoke (EPIC-74)¶

Run when the .qe you're deploying carries entry_schedule = staged_entry(...) (or exit_schedule).

Daemon log mentions the scheduler. On startup, look for:

[info] order_scheduler: enabled (entry slices=4, exit slices=0)

If the log says nothing, the schedule is unset on this .qe.

Strategy intent expands into N journal rows. After a signal fires, grep slice_scheduled events.jsonl should show one row per slice with parent_signal_id, slice_idx, slice_total, fire_at_ns, and signed_qty.
F6 TRADE STAGED panel populates. The WORKING ORDERS panel grows a "STAGED ORDERS" section listing each pending slice with its symbol, side, slice index, qty, state, and fire time. Until any slice fires the state column reads pending (green).
First slice fires at its window. When wall-clock crosses the first slice's fire_at_ns, the daemon log emits:

[info] order_scheduler: fired 1 slice(s) at ts_ns=...

A matching slice_fired row appears in the journal, the broker submits the order through the same risk-gated router as un-staged orders, and the panel's first row flips to fired (dim).

Restart recovery. Kill the daemon (Cmd-Shift-X or qe_daemon stop) after 1-2 of N slices have fired. Restart the daemon and confirm the log line order_scheduler: hydrated N slice row(s) from journal. The STAGED panel resumes with the same fired/pending split. Subsequent ticks fire ONLY the un-fired slices — the broker should NOT see a re-submission of the slices that fired pre-crash.
Kill-switch stops Pending from firing — and preserves it. Trip the kill-switch with at least one slice still Pending. The watchdog log emits watchdog: N staged slice(s) PRESERVED across the kill and the pending rows stay Pending in the panel. No broker submissions occur while the gate is closed.

They are not cancelled, and this step used to assert the opposite: the rows flipping to cancel (red) is the old behaviour and is now a smoke failure. Restart the daemon and verify the slices rehydrate — those still inside their window fire, those past it are retired and journalled window_miss.

"Pending" means never submitted. Slices already sent to IBKR are untouched and keep working — the trip closes the submit gate, it does not reach the venue. Verify this too: leave one slice fired and confirm it is still live in TWS after the trip. Cancel it by hand when you're done.

If step 5 fails — the daemon re-submits a slice the broker already filled pre-crash — STOP. That's a double-submit, not a smoke nit; file an issue with the relevant journal segment + IBKR paper account ids.

Known limitations (NOT smoke gates)¶

Bar closes during the smoke require live ticks. Paper accounts without a paid real-time data subscription get delayed-summary tick types that the bar aggregator drops as non-trades. To exercise bar closes + strategy fires, either: use the smoke during RTH, subscribe to live market data on the paper account, or wire a tick-replay source for offline testing.
Alpaca daemon support — broker = "alpaca-..." exits code 4 with a clear error. Daemon ships ibkr-paper / ibkr-live only.

qe_daemon IBKR-paper smoke checklist¶