From 7a07371fa9297b95c8047ba380844c2936e25d4f Mon Sep 17 00:00:00 2001 From: Romazes Date: Mon, 10 Aug 2026 21:15:09 +0300 Subject: [PATCH 01/25] docs: add ADR for brokerage order polling service - one core service polls order updates when the stream is missing, lost or silent - two modes by constructor: read one watched brokerage order id, or all orders - brokerages convert their wire model into a shared BrokerOrderState and one core diff emits the events - grounded in a survey of eight brokerage plugins --- .../0001-brokerage-order-polling-service.md | 810 ++++++++++++++++++ 1 file changed, 810 insertions(+) create mode 100644 Documentation/ADR/0001-brokerage-order-polling-service.md diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md new file mode 100644 index 000000000000..a08d26cad02d --- /dev/null +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -0,0 +1,810 @@ +# ADR 0001: Brokerage order polling service + +## Status + +Proposed - 2026-08-07 + +## Purpose + +Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: + +- **CharlesSchwab** wrote `Services/OrderUpdatePollingService.cs` so the algorithm keeps working when a second + algorithm takes the single streaming connection away. +- **Public.com** wrote `OrderPollingService.cs` because Public.com has no order stream at all. +- **Tradier** has a smaller version of the same idea written inline in the brokerage. +- **InteractiveBrokers** has the problem and no answer: when the broker does not reply, the algorithm stops. + +Underneath those four there is one problem, not four: **the real-time channel is the only thing telling Lean what +happened to an order, and it is not reliable.** It can be absent, it can be taken away, it can drop for 15 minutes, +and it can stay up while quietly missing an update. In all four shapes the broker still knows the answer over HTTP, +and nobody asks. + +This document proposes one helper class in Lean core, `BrokerageOrderPollingService`, that any brokerage can create +and use. The brokerage picks one of two constructors — read one order by its brokerage id, or read all orders — and +hands the service a read callback that converts each order from the broker's own model into one shared snapshot +shape. Every N seconds the service runs the read, and the snapshots travel through the brokerage's message handler +back into the service, which compares each one with the last state it has seen for that order and raises an event +with the order events that are new. The brokerage decides what to do with them. + +This document covers the service only. It does not change any brokerage on its own — each plugin adopts it in its +own pull request. + +## The problem + +### 1. The same loop, written three times + +| Plugin | Class | Size | Mode | Interval | How a result reaches Lean | +| --- | --- | --- | --- | --- | --- | +| CharlesSchwab | `Services/OrderUpdatePollingService.cs` | 245 lines | all orders | `charles-schwab-order-poll-interval-ms`, default `3000` | `_messageHandler.HandleNewMessage` | +| Public.com | `OrderPollingService.cs` | 268 lines | per order id | `OrderPollingInterval` ctor argument | `_messageHandler.HandleNewMessage` | +| Tradier | inline, `TradierBrokerage.cs:1240-1284` | ~45 lines | per order id, one shot | `Task.Delay(2s)` | direct | + +The two service classes are near copies outside their two callbacks. Both have: a background task, a +`CancellationTokenSource` recreated by `Start` and cleared by `Stop`, an idempotent `Start`, a `Stop` that cancels and +waits up to 2 seconds before disposing the source, a `Dispose` that refuses to start again, a loop that logs and +retries on a failed read instead of dying, and the same two trace lines. Even the comments match, because the second +one was written from the first. + +What actually differs is small and none of it is a design decision worth keeping twice: `Task.Run` against +`Task.Factory.StartNew(LongRunning)`, an async fetch against a sync one, `Task.Delay` against +`cancellationToken.WaitHandle.WaitOne`, and a failure counter that only Schwab has. Public has one real extra: a +registry of watched brokerage ids with the last state seen for each (`Models/OrderSnapshot.cs`: status, cumulative +filled quantity, average price). + +Tradier's version is the same idea again in miniature. When a fill arrives for a brokerage id Lean does not know, it +waits 2 seconds, re-checks `_orderProvider.GetOrdersByBrokerageId`, and re-requests the orders from the API +(`TradierBrokerage.cs:1240-1284`). + +### 2. Two brokerages block the order thread for minutes, then kill the run + +Both CharlesSchwab and InteractiveBrokers place the order, then **block inside the order method** waiting for the +broker to confirm it on the real-time channel. Neither of them ever asks the broker over HTTP instead. + +| Brokerage | Waits for | How long | Where the number comes from | When it expires | +| --- | --- | --- | --- | --- | +| CharlesSchwab | `OrderAccepted` on the account activity stream | **3 minutes** | hardcoded `TimeSpan.FromMinutes(3)`, `CharlesSchwabBrokerage.cs:478` | `Error` `MissingWebSocketResponse` (`:480`) | +| InteractiveBrokers | `openOrder` / `orderStatus` / `execDetails` callback | **5 minutes** | `ib-response-timeout`, default `300` seconds, `InteractiveBrokersBrokerage.cs:84` | `Error` `NoBrokerageResponse` (`:1659`) | +| InteractiveBrokers, `MarketOnOpen` / `ComboLegLimit` / `ComboMarket` / `ComboLimit` | same | 10 seconds | `ib-no-submission-orders-response-timeout`, `:90` | Lean **invents** a `Submitted` event (`:1649-1652`) | + +Two costs, and both are paid on every occurrence. + +**The wait itself.** Schwab's is in `PlaceOrder`; IB's is in `IBPlaceOrder` (`:1536`), which serves both `PlaceOrder` +(`:451`) and `UpdateOrder` (`:483`), and `CancelOrder` has its own copy at `:538`. So a single lost message parks an +order thread for three or five minutes while the market moves. + +**The end of the run.** A `BrokerageMessageType.Error` is not a log line. `DefaultBrokerageMessageHandler` turns it +into `SetRuntimeError` (`Common/Brokerages/DefaultBrokerageMessageHandler.cs:92-96`), which ends the deployment. + +The IB plugin already admits the cause in a comment: + +```csharp +// tracks pending brokerage order responses. In some cases we've seen orders been placed and they never get through to IB +private readonly ConcurrentDictionary _pendingOrderResponse = new(); +``` +`InteractiveBrokersBrokerage.cs:160-161` + +And the third row is the worst one, because it is not even a failure the operator can see. IB is known not to send a +submission event for those order types, so after 10 seconds Lean writes the event itself: + +```csharp +Status = OrderStatus.Submitted, +Message = "Lean Generated Interactive Brokers Order Event" +``` + +That event is a guess. The broker is never asked whether the order is there. + +So today the answer to "the broker did not reply" is: block for minutes, then stop the algorithm, or guess. + +### 3. A real-time stream can be down, or just quiet + +Section 2 was about one lost message: the order confirmation never arrives, the algorithm stops or Lean invents the +event itself. But the stream can also fail in two quieter ways: the connection drops, or the connection is fine and +a single update is just never sent. Today nothing recovers from either. + +**The socket drops.** When a brokerage reports `Disconnect` and any exchange is open, Lean does not stop immediately: +it waits `DefaultInitialDelay`, **15 minutes**, for the connection to come back +(`Common/Brokerages/DefaultBrokerageMessageHandler.cs:38` and `:117-122`). That is 15 minutes in which the algorithm +is alive, orders may fill, and no order update can arrive. Worse, nothing re-reads the orders once the socket comes +back: a search across the CharlesSchwab, Tastytrade, Alpaca and TradeStation plugins finds no path that calls +`GetOpenOrders` or re-syncs order state on reconnect. Whatever happened during the gap is lost for the rest of the +run. + +Schwab has the sharper version of this: the account activity stream can be taken away permanently, because Schwab +allows one streaming connection per user and a second algorithm on the same account takes the slot. That is the +reason its polling fallback exists at all. + +**The socket is up but incomplete.** A live connection is not proof that every update arrived. IB +[documents this about its own API](https://interactivebrokers.github.io/tws-api/order_submission.html#order_status#:~:text=There%20are%20not%20guaranteed%20to%20be%20orderStatus%20callbacks%20for%20every%20change%20in%20order%20status.), +and the plugin copies the warning into the code: + +```csharp +// There are not guaranteed to be orderStatus callbacks for every change in order status. For example with market orders +// when the order is accepted and executes immediately, there commonly will not be any corresponding orderStatus callbacks. +// For that reason it is recommended to monitor the IBApi.EWrapper.execDetails function in addition to +// IBApi.EWrapper.orderStatus. From IB API docs +``` +`InteractiveBrokersBrokerage.cs:2553-2555` + +A dropped or never-sent update leaves a Lean order open forever while the broker has closed it. The algorithm then +sizes its next trade against holdings that are wrong, and any cancel or update it sends on that order is rejected. + +A poll answers all three cases with the same question — *does the broker still have this order, and at what +status?* — which is why they belong in one service and not three: + +| Case | Mode that covers it | +| --- | --- | +| No order stream at all (Public.com) | all orders, running for the whole session | +| Stream lost or taken away (Schwab, any disconnect) | all orders, started when the stream goes down and stopped when it returns | +| Stream up but an update never arrived (IB, Schwab acks) | watched ids only, running while something is unconfirmed | + +## The shared input: a snapshot, not a Lean order + +The obvious shared input would be `IBrokerage.GetOpenOrders()` (`Common/Interfaces/IBrokerage.cs:94`) — the one +order read every brokerage already implements. It returns Lean `Order` objects, and that is exactly why it does not +work. + +An `Order` carries `BrokerId`, `Symbol`, `Quantity`, `Price`, `Time`, `LastFillTime`, `LastUpdateTime`, +`CanceledTime`, `Type`, `Status` and `Tag` (`Common/Orders/Order.cs`). **There is no filled quantity and no fill +price on it.** Lean core already says this out loud in the startup path that adopts these orders: + +> Beware that this order ticket may not accurately reflect the quantity of the order if the open order is partially +> filled. + +`Engine/Setup/BrokerageSetupHandler.cs:504` + +The plugins prove the gap themselves. Each one sets `Status` on the orders it returns, and each one does it +differently: IB maps the IB order state (`InteractiveBrokersBrokerage.cs:3318`), Alpaca writes `Submitted`, or +`PartiallyFilled` when `brokerageOrder.FilledQuantity` is between zero and the order quantity +(`AlpacaBrokerage.cs:386-390`), TradeStation does the same from `leg.ExecQuantity` +(`TradeStationExtensions.cs:378`). Two of them **read the broker's filled quantity and then throw the number +away**, because a Lean `Order` has nowhere to put it. All that survives is the word `PartiallyFilled`. + +So the plugins already have these numbers. There is just no field to store them in. That is the whole +fix: instead of the service reading Lean `Order` objects, **the brokerage converts its +own order model into one small shared snapshot and passes that to the service.** The snapshot has fields for the +status, the total filled quantity and the fill price — the three numbers Public's poller already tracks +per order (`Models/OrderSnapshot.cs`) — so one compare in Lean core works for every brokerage and can report real +fills, not only that the order exists. + +Two rules shape what the service does with a snapshot: + +1. **The service acts only on what a snapshot says.** The plugin fills the snapshot's status by mapping its + broker's own status, the same way it already does for stream messages. So a `Canceled` in a snapshot is a fact, + not a guess. +2. **A missing order proves nothing.** The service never emits an event because an order stopped appearing in the + reads. An order can be missing because it was filled, canceled, rejected, replaced, or never reached the broker + at all — and each of those needs a different event. If the service guessed `Canceled`, a fill would be lost and + the holdings would be wrong. So when an order stays missing, the service does not decide anything: after a + timeout it tells the brokerage the order was never seen, and the brokerage checks with the broker what happened. + +One more fact, this one about the modes: IB **cannot ask the broker about one order id**. Its API only returns all +open orders (`reqOpenOrders` / `reqAllOpenOrders`, with a 15 second wait — `InteractiveBrokersBrokerage.cs:626`). +So the service cannot require per-order reads from every brokerage. Which orders one read covers is the brokerage's +choice, made when it picks the constructor. + +## Design + +### Where it lives + +`Lean/Brokerages/Services/BrokerageOrderPollingService.cs`, namespace `QuantConnect.Brokerages.Services`, with +`BrokerOrderState.cs` next to it. + +`Services` is a new folder under `Brokerages`, and it follows the convention the other subfolders there already use: +`Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. It also matches where +the two plugins keep this class today — CharlesSchwab has it in `QuantConnect.CharlesSchwabBrokerage/Services/`, so +the move up to core keeps the same path. + +Tests go to `Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs`. + +### How a poll flows + +``` +poll thread — one sweep every PollInterval + service calls the read: once per watched id, or once for all orders (by constructor) + brokerage the read asks the broker and converts each order to a BrokerOrderState + service sends each state to the brokerage's message handler + (waits in the queue while an order request holds the lock) + service counts failed reads: three sweeps in a row -> one Warning + service checks the watched orders: one the broker never reported is flagged after a timeout + +handler thread — when the message handler takes a state from the queue + brokerage hands it back: service.ProcessOrderState(orderState) + service compares it with the last state seen and keeps only what is new + service raises OrderEvents: the submit first, then fills, then a close + brokerage forwards them: OnOrderEvents(events) -> Lean applies them +``` + +The service does the start and the end: it runs the loop and it compares the states. The brokerage does the middle: +it reads the broker, converts the orders, and says where the results go. That split is what makes the class +generic: nothing broker-specific ever enters it, because the brokerage translated everything into the shared shape +before the service looks at it. And because the service is the one calling the read, a read that throws is caught +and counted by the service itself — the repeated-failure warning below works without any extra code in the plugin. + +Both existing pollers already have exactly this flow. Schwab's loop hands each polled order to +`_messageHandler.HandleNewMessage` and the diff runs when the handler dequeues it +(`CharlesSchwabBrokerage.OrderUpdatePolling.cs`), and Public wires its poller the same way +(`PublicBrokerage.cs:182`). + +### The broker order state + +```csharp +namespace QuantConnect.Brokerages.Services; + +/// +/// One order, as the brokerage last saw it. The brokerage converts its own order model into this shape +/// and passes it to the service, which compares it with the last snapshot seen for the same order and +/// reports only what is new. +/// +public class BrokerOrderState +{ + /// The brokerage order id. Some brokers give every combo leg its own id, some give the + /// whole combo one id; the snapshot carries whatever the broker uses. + public string BrokerageOrderId { get; set; } + + /// The Lean status the brokerage maps its broker's own status to. + public OrderStatus Status { get; set; } + + /// The total absolute quantity filled so far. Null when the read does not carry it. + public decimal? FilledQuantity { get; set; } + + /// The price the broker reports for the fills. Null when the read does not carry it. + public decimal? FillPrice { get; set; } + + /// When the brokerage reported this state, in UTC. + public DateTime TimeUtc { get; set; } + + /// The broker's own words for a closing status, e.g. the reject reason. + public string Message { get; set; } +} +``` + +Every field except the id and the status is optional, and null means "my read does not know", never "zero". A +brokerage whose read is `GetOpenOrders()` fills only the id and the status, and the service will emit only what +those two can prove. A brokerage whose endpoint returns fill numbers fills `FilledQuantity` and `FillPrice` too and gets fill events. +A brokerage whose endpoint returns a full execution history, like Schwab, reduces it in the mapping: the quantities +sum into `FilledQuantity`, and the newest execution's price becomes `FillPrice`. The service never invents a number +to cover a null. + +The state does not carry an average price or a list of executions, and the service does no price math. +`FillPrice` is used as the broker reported it, and Lean's portfolio averages the fills on its own, like it does +for every other fill event. + +Fees never travel through a poll: both existing pollers report `OrderFee.Zero`, and the snapshot keeps that rule +instead of carrying a fee field nobody fills. + +Combos come in two shapes, and the state carries both without any extra field. Schwab gives every leg its own +brokerage id (`mainId + legId - 1`), so the plugin passes one state per leg. Public.com gives the whole combo +**one** id (`PublicBrokerage.Brokerage.cs:328`), so the plugin passes one state and the service fans it out: +`GetOrdersByBrokerageId` returns every Lean leg order behind the id, and each leg's share of a new fill is + +``` +legFill = leanOrder.Quantity * newPart / abs(leanOrder.GroupOrderManager.Quantity) +``` + +The state has no quantity field because the service does not need one: it already knows the brokerage id, so it +reads the group quantity from the Lean orders themselves. That number equals the broker's own order quantity — the +group quantity is exactly what the combo was placed with. Public's diff already splits fills this way today +(`PublicBrokerage.Brokerage.cs:700-711`). One rule for the mapping follows: +`FilledQuantity` of a shared-id combo is in strategy units, the same units the group quantity counts in. + +A worked example, from a real Public.com order — 5 AAPL strangles, one brokerage id, a put leg and a call leg with +ratio 1 each: + +``` +Lean orders behind the id: put leg Quantity 5 (ratio 1 x group quantity 5) + call leg Quantity 5 (ratio 1 x group quantity 5) +group quantity: 5 (abs(GroupOrderManager.Quantity)) + +broker reports FilledQuantity 2 -> newPart = 2 - 0 = 2 strangles + +put leg: 5 * 2 / 5 = 2 contracts filled +call leg: 5 * 2 / 5 = 2 contracts filled +``` + +The proportion is the point. If the call leg had ratio 2 (Lean quantity 10), the same order-level "2 filled" would +give it 10 * 2 / 5 = 4 contracts — each leg fills at its own ratio, all from one broker number. + +And the other shape, from a real Schwab recording — one combo `OrderResponse`, main id `1002667707949`, three legs, +each leg with its own brokerage id counting up from the main one (`WS_ACCT_ACTIVITY_COMBO_MARKET_FILLED.json` in +the Schwab repo): + +``` +one OrderResponse -> three states, one per leg: + leg 1 -> BrokerageOrderId "1002667707949" (main id + 1 - 1) + leg 2 -> BrokerageOrderId "1002667707950" (main id + 2 - 1) + leg 3 -> BrokerageOrderId "1002667707951" (main id + 3 - 1) + +per state: FilledQuantity = the sum of that leg's execution legs, + FillPrice = that leg's newest execution price +``` + +Here every state finds exactly one Lean order, so there is no split at all — the fan-out and the group quantity +never come into play. The two shapes meet the same diff; only the mapping differs. + +The shape is not guessed — it is what a survey of eight plugins' order reads actually returns. Each column maps to +one field or rule of the state: **filled qty** feeds `FilledQuantity`, **fill price** feeds `FillPrice`, **reason +text** feeds `Message`, and **one id, many Lean orders** is the case the fan-out exists for. A "no" in a cell is +what the nullable fields are for — that broker's state simply carries less, and the service emits less. + +| Broker read | filled qty | fill price | reason text | one id, many Lean orders | +| --- | --- | --- | --- | --- | +| InteractiveBrokers (`reqAllOpenOrders`) | yes — captured but never read | on the paired `orderStatus` callback, not hooked today | no — error callback only | yes, combo legs share the id | +| CharlesSchwab (`GetAllOrders`) | yes | from its execution legs | yes | yes, `mainId + legId - 1` | +| Public.com (`GetOrderById`) | yes | yes | rejects only | yes, combo legs share the id | +| Webull (`GetOpenOrders`) | yes | yes | no | no | +| TradeStation (`GetOrders`) | per leg | yes, as a string | yes | yes, combo legs share the id | +| Alpaca (`ListOrdersAsync`) | yes | yes | no | no | +| Binance (`GetOpenOrders`) | yes | no | no | no | +| Tradier (`GetOrder`) | yes | yes | yes | no | + +The IB row deserves its footnote: an open-orders request is answered with an `openOrder` **and** an `orderStatus` +callback per order +([TWS API docs](https://interactivebrokers.github.io/tws-api/open_orders.html)). The plugin already keeps the whole +`openOrder` payload — `orders.Add((args.Order, args.Contract, args.OrderState))` +(`InteractiveBrokersBrokerage.cs:598`) — and that `IBApi.Order` carries a `FilledQuantity` field. The paired +`orderStatus` callback adds the filled quantity and the average fill price (`Client/OrderStatusEventArgs.cs:38-50`). +Only the last step is missing today: the conversion never reads `FilledQuantity` (no reference anywhere in the +plugin), and `GetOpenOrdersInternal` hooks only `OpenOrder`/`OpenOrderEnd` (`:611-612`), not `OrderStatus`. So the +numbers are already in hand when IB's mapping wants them. + +Three regularities fall out. Every read fills the id and a broker status — the two required fields. Almost every +read fills cumulative quantities, while a full execution list exists at exactly one broker (Schwab) — which is why +the state carries cumulative numbers only and Schwab reduces its legs to them in the mapping. And half the brokers +map one wire order to several Lean orders, so the fan-out is not an edge case. +Nothing common enough to add is missing; the nullable fields cover every "my read does not have it" hole in the +table. Even the ordered quantity needs no field — for the split, the service reads the group quantity from the +Lean orders it already looks up. + +### The class + +```csharp +namespace QuantConnect.Brokerages.Services; + +/// +/// Reads orders from the brokerage on an interval and turns the returned snapshots into order events. +/// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order +/// the broker never replied about. +/// +public class BrokerageOrderPollingService : IDisposable +{ + /// + /// For a broker with a get-order endpoint. Every the service calls + /// once per watched brokerage id — no request when nothing is watched. + /// A null return means the broker does not know the id, so the watchdog keeps counting. + /// Each snapshot is handed to , normally the brokerage's message handler. + /// + public BrokerageOrderPollingService(Func readOrder, Action route, + IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + + /// + /// For a broker with only a bulk endpoint. Every the service calls + /// once, whatever is watched. + /// Each snapshot is handed to , normally the brokerage's message handler. + /// + public BrokerageOrderPollingService(Func> readAllOrders, Action route, + IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + + /// Initializes what both modes share: the route, the order provider, and the two time + /// settings with their defaults. The public constructors chain here and only set their read. + private BrokerageOrderPollingService(Action route, IOrderProvider orderProvider, + TimeSpan? pollInterval, TimeSpan? watchTimeout); + + /// The order events one snapshot produced. Raised inside , never empty. + public event EventHandler> OrderEvents; + + /// A watched order that nothing reported for watchTimeout of polling. + /// Raised once; the id is unwatched with it. + public event EventHandler OrderNotAcknowledged; + + /// Several reads in a row failed, so the run currently has no order updates. + public event EventHandler Message; + + /// True while the polling task is running. + public bool IsPolling { get; } + + /// Watches a brokerage order id, with nothing seen for it yet. Idempotent: watching an + /// already-watched id never overwrites its state. + public void Watch(string brokerageId); + + /// Watches a brokerage order id, seeded with what another path already reported, so the + /// next poll does not repeat it. Used for orders adopted at startup, for a submit reported from + /// the request path, and to move state onto the new id of a replace. + public void Watch(string brokerageId, BrokerOrderState lastSeen); + + /// Stops watching an order and drops its state. + public void Unwatch(string brokerageId); + + /// Records what another path already reported for an order, so the next poll does not + /// repeat it. Called by the streaming path while the stream lives. + public void UpdateOrderState(string brokerageId, BrokerOrderState orderState); + + /// The last state seen for an order, from any path. The streaming path reads it for its + /// own duplicate check, and a replace reads it to move the state to the new id. + public bool TryGetLastOrderState(string brokerageId, out BrokerOrderState lastSeen); + + /// + /// Compares a snapshot with the last state seen for the same order and raises + /// with what is new. Call it from the message handler, so polled orders + /// queue behind an order request that holds the stream lock. + /// + public void ProcessOrderState(BrokerOrderState orderState); + + public void Start(); + public void Stop(); + public void Dispose(); +} +``` + +The loop, `Start`, `Stop`, `Dispose` and the failure counter are lifted from Schwab's service, which is the more +complete of the two. The watch registry, the snapshot compare and the seed-on-adopt come from Public's. + +### The two modes + +The mode is the constructor. Both run the same diff; they differ only in what one sweep reads: + +- **Per order id** — `Func`: the service loops the watched ids and calls the read once per + id. Nothing watched, nothing requested. Public.com's own service has exactly this constructor today — + `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)`. +- **All orders** — `Func>`: the service calls the read once per sweep, and the read + returns everything the broker lists. + +Two constructors instead of one `Func` for both, because the bulk case does not fit a per-id +signature: called once per watched id, a bulk read would repeat the full-account request for every id in the same +sweep, and with nothing watched (Schwab's fallback watches nothing — it is the whole order path) it would never run +at all. + +The watch registry serves both modes. In per-id mode it is also the read list. In all-orders mode it is only the +watchdog: the service still checks that every watched id eventually shows up in the snapshots. + +| Plugin | The action reads | Scope | Why | +| --- | --- | --- | --- | +| CharlesSchwab | bulk | all | one request returns the whole account, and Schwab is the order path for the run | +| Public.com | per id | watched | Public.com has a get-order endpoint and only cares about its own orders | +| InteractiveBrokers | bulk | watched | IB has **no** per-order request — `reqOpenOrders` always returns everything | +| Tradier | bulk | watched | same as IB: the unknown ids are re-checked against a full read | + +### The service never raises Lean order events itself + +The service raises its own `OrderEvents` event, from inside `ProcessOrderState`. It never calls `OnOrderEvents`. The +brokerage routes snapshots through its `BrokerageConcurrentMessageHandler` and forwards the events. + +This is not style. The poll runs on its own thread, so it can see an order already filled while `PlaceOrder` is +still reporting `Submitted` for the same order. If the fill goes out first, the late submit flips a filled order +back to open, and Lean then accepts a cancel or update on it that the broker rejects. Schwab hit exactly this and +fixed it by pushing polled orders through the same message handler as stream messages, so they wait in the queue +while an order request holds the stream lock (`WithLockedStream`, +`Brokerages/BrokerageConcurrentMessageHandler.cs:99`). That is why `ProcessOrderState` is a separate public method +instead of something the loop calls itself: the brokerage puts its message handler between the read and the diff, +and the queue keeps the order right. The handler's message type just has to cover both the stream's model and the +snapshot — Schwab uses a small marker interface for exactly this. + +One requirement travels with the queue: the place request, the `BrokerId` assignment and the `Submitted` report +must all happen inside the same `WithLockedStream` block that the snapshots queue behind. That is what guarantees +a queued snapshot always resolves the id and finds the status the request already reported. Schwab's fallback path +does all three inside the lock today. + +### The diff + +`ProcessOrderState` compares the snapshot with the last state it has seen for that brokerage id: + +``` +record the id as seen, for the watchdog +find the Lean orders by brokerage id (IOrderProvider.GetOrdersByBrokerageId — a list, +because combo legs can share one id and each leg is its own Lean order) + none found -> skip and write nothing: not ours, or ours with the id not on the + Lean order yet — the next sweep sees it again + closed in Lean -> skip, unwatch, drop its state + +the submit first, once: Submitted is emitted when nothing was emitted for the id yet, +the Lean order is still New, and the snapshot is not a reject. Lean requires it before +any fill, and a market order can already be Filled the first time a poll sees it. The +second gate matters in bulk mode beside a live stream: orders the stream already +confirmed are not New in Lean, so a sweep seeing them for the first time stays quiet. + +then the fills, so a close can never outrun a fill of the same order: + the new part is FilledQuantity - alreadyReported; nothing at zero or below + one fill event, priced at the state's FillPrice + alreadyReported never shrinks, and it moves only when a part was actually emitted + legs sharing one id split the new part by each leg's share of the group quantity, + read from the Lean order's GroupOrderManager + fill quantities are signed by the Lean order's direction; the state stays absolute + an order is Filled once its total covers abs(leanOrder.Quantity), else PartiallyFilled + +the end of the order last: + Canceled / Invalid -> emitted once, with the snapshot's Message; the id leaves the + read list, but its state stays until a compare sees the Lean + order closed + anything already emitted -> skipped +``` + +The cumulative compare is the rule both existing pollers already share: everything at or below `alreadyReported` +was seen before, so a re-read of the same history reports nothing, and a fill the stream already delivered is not +repeated. It only works because `alreadyReported` never moves backwards. + +A worked example — long 1000 AAPL, two 100-share fills at the same price: + +| Broker reports (cumulative) | alreadyReported | New part | Event | +| --- | --- | --- | --- | +| FilledQuantity 100, FillPrice 310 | 0 | 100 | PartiallyFilled +100 at 310 | +| FilledQuantity 200, FillPrice 310 | 100 | 100 | PartiallyFilled +100 at 310 | +| FilledQuantity 200 again, next sweep | 200 | 0 | nothing | + +Two fills at the same price never look alike to the service, because `FilledQuantity` is the running total and +totals only grow. This is also the field's contract for the mapping: the plugin fills in the broker's cumulative +number, never the size of the last fill — with the increment in that field, the second fill above would look +identical to the first and be lost. + +Pricing is deliberately simple: the new part takes the state's `FillPrice`, as the broker reported it. When several +fills land inside one sweep, the quantity is still exact and the price is the broker's reported price at sweep +time, not each fill's own — Tradier's poller ships exactly this trade-off today and documents it +(`TradierBrokerage.cs:1469-1472`). Public today recovers the exact increment price from the change of the average +(`PublicBrokerage.Brokerage.cs:697`); the service drops that arithmetic on purpose — it amplifies broker rounding +and can even go negative on a tiny part, while the simple price needs no guard at all. + +One more detail is load-bearing. **State outlives the terminal event.** Forgetting an order the moment its +`Canceled` goes out re-reports every fill if the next sweep lands before Lean applies the event — Schwab's own ADR +documents exactly this race. So state is dropped only when a compare sees the order closed **in Lean**, never at +emission. + +### Later: orders placed outside Lean + +The "none found" branch skips today, but it is also an opening. An id the order provider keeps not knowing is most +likely an order the user placed outside Lean, in the broker's own app — and Lean already has a door for those: +`OnNewBrokerageOrderNotification` (`Brokerages/Brokerage.cs:256`). The transaction handler picks it up +(`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:190`, `:1674`), asks the algorithm's brokerage message +handler whether to accept the order, and adopts it with `AddOpenOrder`. TradeStation already raises it from its +stream (`TradeStationBrokerage.cs:1088`); a poll can feed the same door. + +Not in this PR, for one reason: telling "placed outside Lean" apart from "ours, id not assigned yet" needs care — +Tradier's 2-second recheck exists exactly because an unknown id can turn out to be ours a moment later. The safe +shape is: only an id that stays unknown across several sweeps, and is not inside any order request, gets raised as +a new brokerage-side order. That rule can be added to the diff later without changing the snapshot or the API, so +it is future work, not part of this design. + +This is the piece the snapshot buys. In the first draft of this document the diff could only ever emit `Submitted`, +because a Lean `Order` carries no fill numbers, and Schwab and Public had to subclass the service to keep their own +diffs. With the snapshot the diff is shared, whole, and the plugins keep only their mapping. + +### Watch mode and the give-up rule + +`Watch(brokerageId)` is called by `PlaceOrder` right after the request returns an id. From then on: + +- A snapshot arrives for the id, or the stream records one through `UpdateOrderState` → the order is acknowledged, + and the id stays watched until the order closes. +- The order is closed in Lean → `Unwatch`, and its state is dropped. +- `watchTimeout` of polling passes and nothing ever carried the id → `OrderNotAcknowledged` is raised once, with + the id and how long it was watched, and the id is unwatched. + +The timeout clock runs only while the service is polling. A watch set while the service is stopped does not count +down — otherwise every healthy order would trip the watchdog the moment polling starts. So a watch-mode plugin +calls `Start` together with `Watch` (`Start` is idempotent) and may `Stop` once nothing is watched. + +`OrderNotAcknowledged` is a question, not a verdict — this is rule 2 again, a missing order proves nothing. The service does +not know whether the order never arrived or filled instantly, so it does not decide. The brokerage handles it: +Public can call its get-order endpoint, Schwab can read the order with its executions, IB can use `reqExecutions`. +A brokerage that handles nothing raises a `Warning` and the run keeps going, which is already better than today's +`Error` that stops it. + +### One registry for the stream and the poll + +In watch mode the stream is alive **while** the service polls, so both paths see the same fills, and each must know +what the other already reported. The registry is that shared memory, in both directions: + +- **The stream writes what it reports.** After reporting a stream fill, the plugin calls `UpdateOrderState` with the + new cumulative state. The next poll compare starts from it and repeats nothing. +- **The stream reads before it reports.** Schwab's stream handler already keeps a cumulative-quantity dictionary to + drop duplicate stream events (`CharlesSchwabBrokerage.DataQueueHandler.cs:419-425`); `TryGetLastOrderState` is that + dictionary, shared. A stream update at or below the registry's quantity was already reported — by either path — + and is dropped. + +Without the write half, watch mode double-counts: the poll reports a fill, the stream reports the same fill a +moment later, and Lean applies both. This is the one wiring rule that is not optional for a plugin that polls +while its stream lives. + +A replace moves the state, because the registry is keyed by brokerage id and a replace gives the order a new one. +Inside the same locked block that reports `UpdateSubmitted`, the plugin moves it across: +`TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. The same seed +rule covers a plugin that reports `Submitted` from the request path, as Schwab does without the stream: watch the +new id with a `Submitted` snapshot in that block, so the next sweep does not repeat the submit. + +### Start and Stop follow the stream + +`Start` and `Stop` are the hook for the disconnect case. The brokerage calls `Start` when the real-time channel goes +down and `Stop` when it comes back, so polling covers exactly the window in which the stream cannot deliver. Both are +idempotent, and `Stop` does not dispose the service, so a run can switch back and forth as many times as the socket +does. `Dispose` is the one-way door. + +Two rules for adopters: + +- **Sweep once after the stream returns, before stopping.** The socket coming back does not replay what it missed, + and no plugin re-reads orders on reconnect today. One last sweep closes the gap. +- **Coming back is not always allowed.** Schwab must stay on polling for the rest of the run, because reconnecting + takes the single streaming slot back from the other algorithm. That decision belongs to the plugin, not to this + service, which is why the service only offers `Start` and `Stop` and never reconnects anything itself. + +What a gap sweep recovers is decided by what the plugin's read carries. A read that only lists open orders recovers +missed submissions. A read that carries fill numbers recovers the missed fills too — Schwab's does. The service +reports whatever the states can prove, and nothing more. + +### Repeated read failures + +A single failed read is logged and retried on the next sweep. Three failures in a row raise one +`BrokerageMessageType.Warning` through the `Message` event, and a later successful read arms the warning again. This +is Schwab's rule, kept as is: while the sweeps are failing the run may have no order path at all, and a log line +alone leaves the algorithm looking idle for no visible reason. + +Never an `Error`. An `Error` ends the run, which is the outcome this service exists to avoid. + +### State the service owns, and what it does not + +The service owns the watch registry, the last snapshot seen and the already-reported quantity per order, the failure +counter, and the polling task with its cancellation source. The streaming path shares the snapshot registry through +`UpdateOrderState` and `TryGetLastOrderState`, so the poll and the stream never report the same fill twice. The service +does **not** own the Lean order state: that is read from `IOrderProvider` on every compare, so the service never +drifts from what Lean actually knows. + +Four kinds of thread touch that state: the poll loop, the handler thread inside `ProcessOrderState`, the order +threads through `Watch`/`Unwatch`/`UpdateOrderState`, and the watchdog. One internal lock guards the registry against +all of them, and a per-id sweep copies the watched ids under that lock before reading, so `PlaceOrder` can watch a +new id mid-sweep. `ProcessOrderState` itself is not reentrant and does not need to be — the message handler already +serializes it, and an adopter without a handler must serialize the calls itself. + +### Configuration + +Both time settings are optional constructor arguments. A plugin that has its own configuration key — Schwab keeps +`charles-schwab-order-poll-interval-ms` — reads it and passes the value in, so no existing deployment changes. A +plugin that passes nothing gets the shared defaults, resolved once in the shared private constructor both public +ones chain to: + +```csharp +PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3000)); +WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); +``` + +`brokerage-order-poll-interval-ms` is one new generic config entry, so the interval can be tuned once for every +brokerage that uses the default. Core helpers already work this way: `BrokerageConcurrentMessageHandler` reads +`brokerage-concurrent-message-handler-buffer-size` in its constructor +(`Brokerages/BrokerageConcurrentMessageHandler.cs:56`). The 3000 ms default is the value Schwab runs today. + +## Wiring, per plugin + +The general shape, for a streaming brokerage with a bulk endpoint — the constructor is the mode: + +```csharp +// bulk broker: one request per sweep reads the whole account +_orderPollingService = new BrokerageOrderPollingService( + () => _apiClient.GetAllOrders().Select(ToOrderState), // read: model -> snapshot + _messageHandler.HandleNewMessage, // route: through the message handler + _orderProvider, pollInterval: TimeSpan.FromSeconds(3), watchTimeout: TimeSpan.FromMinutes(1)); +_orderPollingService.OrderEvents += (_, orderEvents) => OnOrderEvents(orderEvents); + +// in the message handler callback, next to the stream messages +_orderPollingService.ProcessOrderState(orderState); +``` + +A per-id broker only changes the read — this is Public.com's own constructor today, snapshot instead of its DTO: + +```csharp +_orderPollingService = new BrokerageOrderPollingService( + brokerageId => ToOrderState(_apiClient.GetOrderById(brokerageId)), + _messageHandler.HandleNewMessage, + _orderProvider); // nothing passed: brokerage-order-poll-interval-ms decides, default 3000 ms +``` + +InteractiveBrokers, the plugin with no answer today, adopts the bulk constructor (it has no per-id request) with +the thinnest possible mapping — id and status from the orders `reqAllOpenOrders` returns, fill fields left null. +The fill numbers are not out of reach: the captured `IBApi.Order` already carries a `FilledQuantity` field, and the +paired `orderStatus` callback adds the average fill price, so the mapping can grow later without a new endpoint. +The first step stays thin: + +```csharp +// IBPlaceOrder, instead of blocking up to 5 minutes and then killing the run +_orderPollingService.Watch(ibOrderId.ToStringInvariant()); +_orderPollingService.Start(); // idempotent; Stop once nothing is watched +``` + +One honest cost first: IB has no `BrokerageConcurrentMessageHandler` today — only Schwab and Public do — so its +adoption starts by putting one, or an equivalent lock, between its order path and `ProcessOrderState`. Without that, +the fill-before-submit ordering this design depends on does not exist in IB. + +The 5 minute block goes away: `IBPlaceOrder` returns as soon as the request is out, and the watch resolves the order +in the background. The `_noSubmissionOrderTypes` guess (`MarketOnOpen`, `ComboLegLimit`, `ComboMarket`, `ComboLimit`) +becomes a real check: the broker either lists the order, and `Submitted` is true, or it does not, and the brokerage +is asked instead of Lean inventing an event. + +The same two lines cover a dropped socket, in any streaming plugin: + +```csharp +// where the brokerage already handles the connection going down +_orderPollingService.Start(); + +// on reconnect: one last sweep for the gap, then hand the job back to the stream +_orderPollingService.Stop(); +``` + +CharlesSchwab and Public.com delete their own service class **and their diff**, and keep only the mapping from +their models to the snapshot. Schwab keeps its own rule of never going back to the stream. Tradier replaces the +inline `Task.Delay` block with `Watch` on the unknown ids. + +## What stays in the plugin + +- Reading the broker: the read callback and the constructor choice between per-id and all orders, plus how far + back a bulk read reaches (Schwab reads from its oldest open Lean order). +- Converting model to state: the status mapping, combo leg ids (Schwab's `mainId + legId - 1`), reducing an + execution history to the two numbers (Schwab sums its execution legs and takes the newest leg's price). A plugin + whose leg ids are derived rather than returned by the broker should verify they resolve and warn when they do + not — the service skips silently. +- Routing: passing snapshots through its message handler and forwarding `OrderEvents` to `OnOrderEvents`. +- Reporting without the stream: Schwab's submit and replace events from the REST reply stay in the plugin; the + service only asks that they seed the registry (see "One registry for the stream and the poll"). +- Deciding what an unacknowledged order means. + +## Alternatives not taken + +- **Hand the service Lean `Order` objects from `GetOpenOrders()` and let it diff those.** The first draft of this + document. It dies at the boundary: a Lean `Order` has no filled quantity and no fill price, so the shared diff + could only emit `Submitted`, and every brokerage with richer data had to subclass the service and override the + diff. The snapshot carries the same numbers the plugins already read and today throw away, so the subclass and + the override are gone. +- **A core interface the wire model implements** (`OrderResponse : IBrokerOrderState`), so the plugin passes its + API model straight in with no conversion. Checked against all eight surveyed plugins and rejected on the + evidence. Two cannot implement it at all: IB's order model is compiled into the vendor `CSharpAPI.dll`, and + Alpaca's `JsonOrder` is `internal sealed` inside the SDK — both would need a wrapper class, which is the same + work as filling the snapshot. Tradier's model exposes public **fields**, which cannot implement interface + properties, so its whole serialization surface would have to change. TradeStation's model is a struct, boxed on + every interface use. And for the four brokers where one wire order becomes several Lean orders, one object + cannot be several states — the per-leg conversion survives anyway. The interface would also pull the + broker-to-Lean status mapping inside the wire DTOs as computed properties, and the service registry would hold + whole wire objects alive as stored state (Schwab's `OrderResponse` carries the full execution history). A plain + class the plugin fills is one pattern that works for all eight. +- **Put it on the `Brokerage` base class, driven by the engine, like the cash sync.** + `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:480`, called from + `Engine/TransactionHandlers/BrokerageTransactionHandler.cs:731`) is the existing shape for core-driven periodic + work, and it was the obvious candidate. Rejected: cash sync runs once a day on a schedule core can decide, while + the right poll interval here is a property of the broker's rate limits and of whether its stream is alive. It also + has to be off by default — most plugins must not poll — and a base-class hook that nearly everyone turns off is + worse than a class you create when you need it. +- **Let the service call `OnOrderEvents` directly.** Simpler to wire and reintroduces the fill-before-submit bug in + every adopter. See "The service never raises Lean order events itself". +- **Emit `Canceled` when an order leaves the open list.** This is the tempting one, and it is wrong: an order that + filled also leaves the open list, so this drops fills and desynchronizes holdings. It is why rule 2 exists — the + service acts on what a snapshot says, never on an order going missing. +- **Carry the execution history in the state** — a per-execution list of quantity, price and time, so every + execution becomes its own event at its own price. The survey killed it: exactly one of the eight brokers (Schwab) + returns such a list, and the cumulative compare already keeps quantities exact. The list would buy per-execution + price precision for one broker at the cost of a second diff branch every adopter has to reason about. If it is + ever wanted, it comes back as one additive nullable field without breaking anyone — the same goes for a + `GetOrderExecutions` API on `IBrokerage`. +- **Leave it in the plugins.** It is already written three times, and the fourth copy would be IB's. + +## Risks + +| Risk | What we do about it | +| --- | --- | +| A plugin maps a broker status to the wrong Lean status | The mapping is the same one its streaming path already needs, written once per plugin and covered by its own tests. The service only emits transitions, so a wrong mapping surfaces once, not as a flood. | +| A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the watchdog still fires. | +| Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier ships this trade-off today (`TradierBrokerage.cs:1469-1472`); a shorter poll interval narrows it. | +| Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order is unacknowledged, and the interval is a constructor argument the plugin picks. | +| A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | +| An adopter passes `ProcessOrderState` as the route and gets fill-before-submit | The route is documented as "your message handler". Schwab and Public have one; IB and Tradier do not, and adding one is named in their rollout steps. | +| The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | +| The watchdog cannot tell "never arrived" from "filled instantly" | It does not try. `OrderNotAcknowledged` hands the question to the brokerage, which has the endpoints to answer it. | +| Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | +| A plugin starts polling on disconnect and forgets to stop on reconnect | Both paths are one line and sit next to the connection handling the plugin already has. The poll side repeats nothing the registry already holds, so the cost of forgetting is extra requests, not extra events. | + +## Rollout + +1. This PR: the service, the snapshot, and their unit tests in `Tests/Brokerages/Services/`, no brokerage changes. +2. InteractiveBrokers: add the message handler it does not have, then replace the `NoBrokerageResponse` error and + the invented `Submitted` with a watch. This is the proof that the abstraction holds for a plugin that did not + write it. +3. CharlesSchwab and Public.com: delete their service class and their fill/close diff. What stays is real and + named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and its + without-stream submit and replace reporting. Two behavior changes are intentional: a Public poll that shows a + new fill and the cancel together now emits both — today's code drops the cancel + (`PublicBrokerage.Brokerage.cs:629-638`) — and polled fills are priced at the broker's reported price of the + sweep, so Public's change-of-average recovery and Schwab's per-execution prices become per-sweep prices while + the quantities stay exact. +4. Tradier: replace the inline re-check block with a watch. Tradier splits orders across zero, so one brokerage + order can cover only part of the Lean quantity — its watch resolves submissions only, and fills stay on its + existing path. From aa41f1c6d4546358d998f29a74be0548fc3b8b0e Mon Sep 17 00:00:00 2001 From: Romazes Date: Tue, 11 Aug 2026 20:44:01 +0300 Subject: [PATCH 02/25] refactor: split BrokerageConcurrentMessageHandler into a shared message queue - BrokerageMessageQueue owns the lock, buffer and dispatch, raised through a MessageReceived event - BrokerageConcurrentMessageHandler becomes a thin wrapper with the same public API - add RegisterMessageType and a generic HandleNewMessage, so a second message source can queue behind the same lock instead of a separate, independent one - add BrokerageMessageQueueTests covering the shared lock, an unmatched message type, and a registered second type --- .../BrokerageConcurrentMessageHandler.cs | 212 +++------------ Brokerages/BrokerageMessageQueue.cs | 249 ++++++++++++++++++ .../Brokerages/BrokerageMessageQueueTests.cs | 151 +++++++++++ 3 files changed, 432 insertions(+), 180 deletions(-) create mode 100644 Brokerages/BrokerageMessageQueue.cs create mode 100644 Tests/Brokerages/BrokerageMessageQueueTests.cs diff --git a/Brokerages/BrokerageConcurrentMessageHandler.cs b/Brokerages/BrokerageConcurrentMessageHandler.cs index d0b896f47996..2cfad5b1a3df 100644 --- a/Brokerages/BrokerageConcurrentMessageHandler.cs +++ b/Brokerages/BrokerageConcurrentMessageHandler.cs @@ -14,24 +14,19 @@ */ using System; -using System.Threading; -using QuantConnect.Logging; -using System.Collections.Generic; -using QuantConnect.Configuration; namespace QuantConnect.Brokerages { /// /// Brokerage helper class to lock message stream while executing an action, for example placing an order /// + /// A thin wrapper around : the lock, buffer and dispatch all + /// live there now, so a second source with its own message type can be handled through the exact same + /// lock this handler uses - see . public class BrokerageConcurrentMessageHandler : IDisposable where T : class { - private readonly Action _processMessages; - private readonly Queue _messageBuffer; - private readonly ILock _lock; - private readonly ManualResetEventSlim _messagesProcessedEvent; - private readonly int _maxMessageBufferSize; + private readonly BrokerageMessageQueue _queue; /// /// Creates a new instance @@ -49,202 +44,59 @@ public BrokerageConcurrentMessageHandler(Action processMessages) /// Whether to enable concurrent order submission public BrokerageConcurrentMessageHandler(Action processMessages, bool concurrencyEnabled) { - _processMessages = processMessages; - _messageBuffer = new Queue(); - _lock = concurrencyEnabled ? new ReaderWriterLockWrapper() : new MonitorWrapper(); - _messagesProcessedEvent = new ManualResetEventSlim(false); - _maxMessageBufferSize = Config.GetInt("brokerage-concurrent-message-handler-buffer-size", 20); + _queue = new BrokerageMessageQueue(concurrencyEnabled); + RegisterMessageType(processMessages); } /// - /// Disposes of the resources used by this instance + /// Registers another message type this handler also processes, alongside . + /// A source with its own message type - for example the order polling service - registers here once, + /// then calls like anything else, and its messages queue + /// behind the exact same lock as instead of getting a second, independent + /// lock that would not actually serialize against this one. /// - public void Dispose() + /// The action to call for each new message of this type + public void RegisterMessageType(Action processMessages) + where TMessage : class { - _lock.Dispose(); - _messagesProcessedEvent.Dispose(); - } - - /// - /// Will process or enqueue a message for later processing it - /// - /// The new message - public void HandleNewMessage(T message) - { - lock (_messageBuffer) + _queue.MessageReceived += message => { - if (_lock.TryEnterReadLockImmediately()) - { - try - { - ProcessMessages(message); - } - finally - { - _lock.ExitReadLock(); - } - } - else if (message != default) + if (message is TMessage typed) { - // if someone has the lock just enqueue the new message they will process any remaining messages - // if by chance they are about to free the lock, no worries, we will always process first any remaining message first see 'ProcessMessages' - _messageBuffer.Enqueue(message); + processMessages(typed); } - } + }; } /// - /// Lock the streaming processing while we're sending orders as sometimes they fill before the call returns. + /// Disposes of the resources used by this instance /// - public void WithLockedStream(Action code) + public void Dispose() { - // Let's limit the amount of messages we can buffer, so we wait until - // consumers process a full queue of messages before we potentially add more - var queueIsFull = false; - lock (_messageBuffer) - { - queueIsFull = _messageBuffer.Count >= _maxMessageBufferSize; - } - if (queueIsFull) - { - _messagesProcessedEvent.Wait(); - _messagesProcessedEvent.Reset(); - } - - _lock.EnterWriteLock(); - try - { - code(); - } - finally - { - // once we finish our 'code' we will process any message that come through, - // to make sure no message get's left behind (race condition between us finishing 'ProcessMessages' - // and some message being enqueued to it, we just take a lock on the buffer - lock (_messageBuffer) - { - var lockedStreams = _lock.CurrentWriteCount; - - // we release the semaphore first so by the time we release '_messageBuffer' any new message is processed immediately and not enqueued - _lock.ExitWriteLock(); - // only process if no other threads will process them after us - if (lockedStreams == 1) - { - ProcessMessages(); - } - } - } + _queue.Dispose(); } /// - /// Process any pending message and the provided one if any + /// Will process or enqueue a message for later processing it. Works for and + /// for any other type registered through - the type is + /// inferred from the argument, so the call looks the same either way. /// - /// To be called owing the stream lock - private void ProcessMessages(T message = null) + /// The new message + public void HandleNewMessage(TMessage message) + where TMessage : class { - try - { - if (message != null) - { - _messageBuffer.Enqueue(message); - } - - // double check there isn't any pending message - while (_messageBuffer.TryDequeue(out var e)) - { - try - { - _processMessages(e); - } - catch (Exception ex) - { - Log.Error(ex); - } - } - } - finally + if (message != null) { - _messagesProcessedEvent.Set(); + _queue.Enqueue(message); } } - private interface ILock : IDisposable - { - int CurrentWriteCount { get; } - - void ExitReadLock(); - - bool TryEnterReadLockImmediately(); - - void EnterWriteLock(); - - void ExitWriteLock(); - } - /// - /// A simple reader/writer lock implementation that allows us to switch the meaning of read and write locks - /// so that it can be used for single reader and multiple writers scenario. - /// - /// We want to allow multiple producers so, for example, a brokerage can be placing multiple orders concurrently, - /// since the transaction handler can have multiple threads processing orders. - /// But, on the other side, we need to ensure that messages are processed only when no producers are writing - /// to the stream (hence only one reader). For example, a brokerage needs the to lock the stream and - /// only handle incoming order event messages after it releases the lock, but we now support multiple streams - /// (so multiple orders) so we wait for all the current producers to release the lock before processing any messages. + /// Lock the streaming processing while we're sending orders as sometimes they fill before the call returns. /// - private class ReaderWriterLockWrapper : ILock - { - private readonly ReaderWriterLockSlim _lock; - - public int CurrentWriteCount => _lock.CurrentReadCount; - - public ReaderWriterLockWrapper() - { - _lock = new ReaderWriterLockSlim(LockRecursionPolicy.NoRecursion); - } - public void ExitReadLock() => _lock.ExitWriteLock(); - public bool TryEnterReadLockImmediately() => _lock.TryEnterWriteLock(0); - public void EnterWriteLock() => _lock.EnterReadLock(); - public void ExitWriteLock() => _lock.ExitReadLock(); - - public void Dispose() - { - _lock.Dispose(); - } - } - - private class MonitorWrapper : ILock + public void WithLockedStream(Action code) { - private readonly object _lockObject; - - private long _currentWriteCount; - - public int CurrentWriteCount => (int)Interlocked.Read(ref _currentWriteCount); - - public MonitorWrapper() - { - _lockObject = new object(); - } - - public void ExitReadLock() => Monitor.Exit(_lockObject); - - public bool TryEnterReadLockImmediately() => Monitor.TryEnter(_lockObject); - - public void EnterWriteLock() - { - Monitor.Enter(_lockObject); - Interlocked.Exchange(ref _currentWriteCount, 1); - } - - public void ExitWriteLock() - { - Interlocked.Exchange(ref _currentWriteCount, 0); - Monitor.Exit(_lockObject); - } - - public void Dispose() - { - } + _queue.WithLockedStream(code); } } } diff --git a/Brokerages/BrokerageMessageQueue.cs b/Brokerages/BrokerageMessageQueue.cs new file mode 100644 index 000000000000..bfb487715ebe --- /dev/null +++ b/Brokerages/BrokerageMessageQueue.cs @@ -0,0 +1,249 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using System.Threading; +using QuantConnect.Logging; +using System.Collections.Generic; +using QuantConnect.Configuration; + +namespace QuantConnect.Brokerages +{ + /// + /// Owns the lock and buffer that used to keep to itself. + /// A message of any type can be enqueued here; raises it once it is this + /// message's turn, and each subscriber picks out the type it cares about. This is what lets more than one + /// source share a single lock: a brokerage's own stream messages and, separately, the order polling + /// service's snapshots, so a stream message and a polled order update can never run at the same time, and + /// neither can run while an order request holds . + /// + public class BrokerageMessageQueue : IDisposable + { + private readonly Queue _messageBuffer; + private readonly ILock _lock; + private readonly ManualResetEventSlim _messagesProcessedEvent; + private readonly int _maxMessageBufferSize; + + /// + /// Raised for every dequeued message, in arrival order, one at a time. A brokerage's stream handler + /// and the order polling service each subscribe and filter for their own message type, e.g. + /// MessageReceived += message => { if (message is OrderResponse r) OnPolledOrder(r); }; + /// + public event Action MessageReceived; + + /// + /// Creates a new instance + /// + /// Whether to enable concurrent order submission + public BrokerageMessageQueue(bool concurrencyEnabled = false) + { + _messageBuffer = new Queue(); + _lock = concurrencyEnabled ? new ReaderWriterLockWrapper() : new MonitorWrapper(); + _messagesProcessedEvent = new ManualResetEventSlim(false); + _maxMessageBufferSize = Config.GetInt("brokerage-concurrent-message-handler-buffer-size", 20); + } + + /// + /// Disposes of the resources used by this instance + /// + public void Dispose() + { + _lock.Dispose(); + _messagesProcessedEvent.Dispose(); + } + + /// + /// Will process or enqueue a message for later processing it + /// + /// The new message + public void Enqueue(object message) + { + lock (_messageBuffer) + { + if (_lock.TryEnterReadLockImmediately()) + { + try + { + ProcessMessages(message); + } + finally + { + _lock.ExitReadLock(); + } + } + else if (message != null) + { + // if someone has the lock just enqueue the new message they will process any remaining messages + // if by chance they are about to free the lock, no worries, we will always process first any remaining message first see 'ProcessMessages' + _messageBuffer.Enqueue(message); + } + } + } + + /// + /// Lock the streaming processing while we're sending orders as sometimes they fill before the call returns. + /// + public void WithLockedStream(Action code) + { + // Let's limit the amount of messages we can buffer, so we wait until + // consumers process a full queue of messages before we potentially add more + var queueIsFull = false; + lock (_messageBuffer) + { + queueIsFull = _messageBuffer.Count >= _maxMessageBufferSize; + } + if (queueIsFull) + { + _messagesProcessedEvent.Wait(); + _messagesProcessedEvent.Reset(); + } + + _lock.EnterWriteLock(); + try + { + code(); + } + finally + { + // once we finish our 'code' we will process any message that come through, + // to make sure no message get's left behind (race condition between us finishing 'ProcessMessages' + // and some message being enqueued to it, we just take a lock on the buffer + lock (_messageBuffer) + { + var lockedStreams = _lock.CurrentWriteCount; + + // we release the semaphore first so by the time we release '_messageBuffer' any new message is processed immediately and not enqueued + _lock.ExitWriteLock(); + // only process if no other threads will process them after us + if (lockedStreams == 1) + { + ProcessMessages(); + } + } + } + } + + /// + /// Process any pending message and the provided one if any + /// + /// To be called owing the stream lock + private void ProcessMessages(object message = null) + { + try + { + if (message != null) + { + _messageBuffer.Enqueue(message); + } + + // double check there isn't any pending message + while (_messageBuffer.TryDequeue(out var e)) + { + try + { + MessageReceived?.Invoke(e); + } + catch (Exception ex) + { + Log.Error(ex); + } + } + } + finally + { + _messagesProcessedEvent.Set(); + } + } + + private interface ILock : IDisposable + { + int CurrentWriteCount { get; } + + void ExitReadLock(); + + bool TryEnterReadLockImmediately(); + + void EnterWriteLock(); + + void ExitWriteLock(); + } + + /// + /// A simple reader/writer lock implementation that allows us to switch the meaning of read and write locks + /// so that it can be used for single reader and multiple writers scenario. + /// + /// We want to allow multiple producers so, for example, a brokerage can be placing multiple orders concurrently, + /// since the transaction handler can have multiple threads processing orders. + /// But, on the other side, we need to ensure that messages are processed only when no producers are writing + /// to the stream (hence only one reader). For example, a brokerage needs the to lock the stream and + /// only handle incoming order event messages after it releases the lock, but we now support multiple streams + /// (so multiple orders) so we wait for all the current producers to release the lock before processing any messages. + /// + private class ReaderWriterLockWrapper : ILock + { + private readonly ReaderWriterLockSlim _lock; + + public int CurrentWriteCount => _lock.CurrentReadCount; + + public ReaderWriterLockWrapper() + { + _lock = new ReaderWriterLockSlim(LockRecursionPolicy.NoRecursion); + } + public void ExitReadLock() => _lock.ExitWriteLock(); + public bool TryEnterReadLockImmediately() => _lock.TryEnterWriteLock(0); + public void EnterWriteLock() => _lock.EnterReadLock(); + public void ExitWriteLock() => _lock.ExitReadLock(); + + public void Dispose() + { + _lock.Dispose(); + } + } + + private class MonitorWrapper : ILock + { + private readonly object _lockObject; + + private long _currentWriteCount; + + public int CurrentWriteCount => (int)Interlocked.Read(ref _currentWriteCount); + + public MonitorWrapper() + { + _lockObject = new object(); + } + + public void ExitReadLock() => Monitor.Exit(_lockObject); + + public bool TryEnterReadLockImmediately() => Monitor.TryEnter(_lockObject); + + public void EnterWriteLock() + { + Monitor.Enter(_lockObject); + Interlocked.Exchange(ref _currentWriteCount, 1); + } + + public void ExitWriteLock() + { + Interlocked.Exchange(ref _currentWriteCount, 0); + Monitor.Exit(_lockObject); + } + + public void Dispose() + { + } + } + } +} diff --git a/Tests/Brokerages/BrokerageMessageQueueTests.cs b/Tests/Brokerages/BrokerageMessageQueueTests.cs new file mode 100644 index 000000000000..df06a457b966 --- /dev/null +++ b/Tests/Brokerages/BrokerageMessageQueueTests.cs @@ -0,0 +1,151 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using NUnit.Framework; +using System.Threading; +using System.Threading.Tasks; +using QuantConnect.Brokerages; +using System.Collections.Generic; + +namespace QuantConnect.Tests.Brokerages +{ + [TestFixture] + public class BrokerageMessageQueueTests + { + [Test] + public void TwoSubscribedMessageTypesShareOneLock() + { + var processed = new List(); + var queue = new BrokerageMessageQueue(concurrencyEnabled: true); + queue.MessageReceived += message => + { + switch (message) + { + case StreamMessage stream: + processed.Add($"stream:{stream.Value}"); + break; + case PolledMessage polled: + processed.Add($"poll:{polled.Value}"); + break; + } + }; + + using var lockedEvent = new ManualResetEventSlim(false); + using var releaseEvent = new ManualResetEventSlim(false); + + // Simulates an order request holding WithLockedStream while a place/update/cancel call is in flight. + var writer = Task.Run(() => + { + queue.WithLockedStream(() => + { + lockedEvent.Set(); + releaseEvent.Wait(); + }); + }); + + lockedEvent.Wait(); + + // A "stream" message and a "poll" message both queue behind the write lock instead of running concurrently with it. + queue.Enqueue(new StreamMessage("s1")); + queue.Enqueue(new PolledMessage("p1")); + + Assert.AreEqual(0, processed.Count); + + releaseEvent.Set(); + writer.Wait(); + + CollectionAssert.AreEqual(new[] { "stream:s1", "poll:p1" }, processed); + } + + [Test] + public void MessageWithNoInterestedSubscriberIsSkippedWithoutStoppingLaterMessages() + { + var processed = new List(); + var queue = new BrokerageMessageQueue(); + queue.MessageReceived += message => + { + if (message is StreamMessage stream) + { + processed.Add(stream.Value); + } + }; + + // Nobody filters for PolledMessage: must not throw, and must not block s1 behind it. + Assert.DoesNotThrow(() => queue.Enqueue(new PolledMessage("ignored"))); + queue.Enqueue(new StreamMessage("s1")); + + CollectionAssert.AreEqual(new[] { "s1" }, processed); + } + + [Test] + public void ConcurrentMessageHandlerHandlesARegisteredSecondMessageType() + { + var processed = new List(); + var handler = new BrokerageConcurrentMessageHandler(m => processed.Add($"stream:{m}")); + + // A second, unrelated message type registered on the same handler - standing in for the order + // polling service registering BrokerOrderState alongside the brokerage's own stream type. + handler.RegisterMessageType(m => processed.Add($"poll:{m.Value}")); + + // Same method name, either type - the generic argument is inferred from what is passed in. + handler.HandleNewMessage("s1"); + handler.HandleNewMessage(new PolledMessage("p1")); + + CollectionAssert.AreEqual(new[] { "stream:s1", "poll:p1" }, processed); + } + + [Test] + public void ConcurrentMessageHandlerStillBlocksOnWithLockedStream() + { + var processed = new List(); + var handler = new BrokerageConcurrentMessageHandler(processed.Add, concurrencyEnabled: true); + + using var lockedEvent = new ManualResetEventSlim(false); + using var releaseEvent = new ManualResetEventSlim(false); + + var writer = Task.Run(() => + { + handler.WithLockedStream(() => + { + lockedEvent.Set(); + releaseEvent.Wait(); + }); + }); + + lockedEvent.Wait(); + + handler.HandleNewMessage("s1"); + Assert.AreEqual(0, processed.Count); + + releaseEvent.Set(); + writer.Wait(); + + CollectionAssert.AreEqual(new[] { "s1" }, processed); + } + + private sealed class StreamMessage + { + public string Value { get; } + public StreamMessage(string value) => Value = value; + } + + private sealed class PolledMessage + { + public string Value { get; } + public PolledMessage(string value) => Value = value; + } + } +} From 74e0a451144d899eb5e1054db5060a3ee57c36e1 Mon Sep 17 00:00:00 2001 From: Romazes Date: Thu, 13 Aug 2026 00:32:59 +0300 Subject: [PATCH 03/25] feature: brokerage order polling service - abstract BrokerageOrderPollingService with per-order-id and all-orders modes: watch registry, state compare, watch timeout, repeated-failure warning - plugins seed the registry before Start, so the first sweep repeats nothing the stream already reported - ADR: seed-before-start handover and the survey of streaming plugins that fit it - tests: 25 polling service, 4 message queue --- .../BrokerageConcurrentMessageHandler.cs | 18 +- .../Services/AllOrdersPollingService.cs | 58 ++ Brokerages/Services/BrokerOrderState.cs | 64 ++ .../Services/BrokerageOrderPollingService.cs | 665 ++++++++++++++++++ .../Services/OrderNotAcknowledgedEventArgs.cs | 48 ++ .../Services/PerOrderIdPollingService.cs | 86 +++ .../0001-brokerage-order-polling-service.md | 311 +++++--- .../BrokerageOrderPollingServiceTests.cs | 581 +++++++++++++++ 8 files changed, 1738 insertions(+), 93 deletions(-) create mode 100644 Brokerages/Services/AllOrdersPollingService.cs create mode 100644 Brokerages/Services/BrokerOrderState.cs create mode 100644 Brokerages/Services/BrokerageOrderPollingService.cs create mode 100644 Brokerages/Services/OrderNotAcknowledgedEventArgs.cs create mode 100644 Brokerages/Services/PerOrderIdPollingService.cs create mode 100644 Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs diff --git a/Brokerages/BrokerageConcurrentMessageHandler.cs b/Brokerages/BrokerageConcurrentMessageHandler.cs index 2cfad5b1a3df..aa770f5ed1d5 100644 --- a/Brokerages/BrokerageConcurrentMessageHandler.cs +++ b/Brokerages/BrokerageConcurrentMessageHandler.cs @@ -77,9 +77,21 @@ public void Dispose() } /// - /// Will process or enqueue a message for later processing it. Works for and - /// for any other type registered through - the type is - /// inferred from the argument, so the call looks the same either way. + /// Will process or enqueue a message for later processing it + /// + /// The new message + public void HandleNewMessage(T message) + { + if (message != null) + { + _queue.Enqueue(message); + } + } + + /// + /// Same as , for any other type registered through + /// - the type is inferred from the argument, + /// so the call looks the same either way. /// /// The new message public void HandleNewMessage(TMessage message) diff --git a/Brokerages/Services/AllOrdersPollingService.cs b/Brokerages/Services/AllOrdersPollingService.cs new file mode 100644 index 000000000000..aea112ed20e2 --- /dev/null +++ b/Brokerages/Services/AllOrdersPollingService.cs @@ -0,0 +1,58 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using QuantConnect.Securities; +using System.Collections.Generic; + +namespace QuantConnect.Brokerages.Services +{ + /// + /// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched, and the + /// read returns everything the broker lists. + /// + public class AllOrdersPollingService : BrokerageOrderPollingService + { + /// + /// Reads every order the broker lists. + /// + private readonly Func> _readAllOrders; + + /// + /// Creates a new . + /// + /// Reads every order the broker lists, one state per brokerage order id. + /// Where each state a sweep returns goes, normally the brokerage's message handler. + /// Resolves brokerage order ids to Lean orders. + /// How long the loop sleeps between sweeps. Null falls back to the + /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. + /// How long a watched order may stay unreported before + /// is raised. Null falls back to one minute. + public AllOrdersPollingService(Func> readAllOrders, Action route, + IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + : base(route, orderProvider, pollInterval, watchTimeout) + { + _readAllOrders = readAllOrders; + } + + /// + /// Calls the read once for the whole sweep. + /// + protected override IEnumerable Sweep() + { + return _readAllOrders(); + } + } +} diff --git a/Brokerages/Services/BrokerOrderState.cs b/Brokerages/Services/BrokerOrderState.cs new file mode 100644 index 000000000000..de3136515d8b --- /dev/null +++ b/Brokerages/Services/BrokerOrderState.cs @@ -0,0 +1,64 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using QuantConnect.Orders; + +namespace QuantConnect.Brokerages.Services +{ + /// + /// One order, as the brokerage last saw it. The brokerage converts its own order model into this shape + /// and passes it to , which compares it with the last state + /// seen for the same order and reports only what is new. Every field except the id and the status is + /// optional, and null means "my read does not know", never "zero". + /// + public class BrokerOrderState + { + /// + /// The brokerage order id. Some brokers give every combo leg its own id, some give the whole + /// combo one id; the state carries whatever the broker uses. + /// + public string BrokerageOrderId { get; set; } + + /// + /// The Lean status the brokerage maps its broker's own status to. + /// + public OrderStatus Status { get; set; } + + /// + /// The total absolute quantity filled so far, never the size of the last fill. Null when the + /// read does not carry it - a status without this number can + /// not produce the fill event that closes the order. For a combo that shares one brokerage id + /// across its legs, this counts in strategy units, the same units the group quantity counts in. + /// + public decimal? FilledQuantity { get; set; } + + /// + /// The price the broker reports for the fills. Null when the read does not carry it, and no + /// fill event goes out without it - the service never invents a number. + /// + public decimal? FillPrice { get; set; } + + /// + /// When the brokerage reported this state, in UTC. + /// + public DateTime TimeUtc { get; set; } + + /// + /// The broker's own words for a closing status, e.g. the reject reason. + /// + public string Message { get; set; } + } +} diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/BrokerageOrderPollingService.cs new file mode 100644 index 000000000000..cf1d37c5a707 --- /dev/null +++ b/Brokerages/Services/BrokerageOrderPollingService.cs @@ -0,0 +1,665 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using System.Threading; +using QuantConnect.Util; +using QuantConnect.Orders; +using QuantConnect.Logging; +using System.Threading.Tasks; +using QuantConnect.Securities; +using QuantConnect.Orders.Fees; +using System.Collections.Generic; +using QuantConnect.Configuration; + +namespace QuantConnect.Brokerages.Services +{ + /// + /// Reads orders from the brokerage on an interval and turns the returned states into order events. + /// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order + /// the broker never replied about. This base class owns everything both modes share - the loop, the + /// watch registry, the compare and the events. What one sweep reads is the subclass: + /// or . + /// + public abstract class BrokerageOrderPollingService : IDisposable + { + /// + /// How many sweeps in a row have to fail before the failure is reported through . + /// + private const int ConsecutiveFailuresBeforeReport = 3; + + /// + /// Guards the registry and the polling task against the poll loop, the handler thread inside + /// , the order threads through / + /// / , and the watch-timeout check. + /// + private readonly object _lock = new object(); + + /// + /// The registry: per brokerage order id, the last state seen and what was already reported for it. + /// + private readonly Dictionary _orderStates = new(); + + /// + /// Where each state a sweep returns goes, normally the brokerage's message handler. The handler + /// dispatches it back into , so polled states queue behind an order + /// request that holds the stream lock. + /// + private readonly Action _route; + + /// + /// Resolves brokerage order ids to Lean orders on every compare, so the service never drifts from + /// what Lean actually knows. + /// + private readonly IOrderProvider _orderProvider; + + /// + /// Cancels the current polling task. Recreated by and cleared by . + /// + private CancellationTokenSource _cancellationTokenSource; + + /// + /// The background polling task for the current run. waits on it briefly, so the + /// cancellation source is only disposed once the loop no longer uses it. + /// + private Task _pollingTask; + + /// + /// Set by so a disposed service refuses to start again. + /// + private bool _disposed; + + /// + /// Backing field of , read from the order placement and connection paths + /// while the polling task is being started or stopped. + /// + private volatile bool _isPolling; + + /// + /// The order events one state produced, in order: the submit first, then fills, then a close. + /// Raised inside , never empty. The brokerage forwards them to + /// its own OnOrderEvents. + /// + public event EventHandler> OrderEvents; + + /// + /// A watched order that nothing reported for of polling. Raised once; + /// the id is unwatched with it. The brokerage decides what the silence means. + /// + public event EventHandler OrderNotAcknowledged; + + /// + /// Several reads in a row failed, so the run currently has no order updates. Raised once per + /// outage, as a , never an error. + /// + public event EventHandler Message; + + /// + /// True while the polling task is running. + /// + public bool IsPolling => _isPolling; + + /// + /// How long the loop sleeps between sweeps. + /// + public TimeSpan PollInterval { get; } + + /// + /// How long a watched order may stay completely unreported, in polling time, before + /// is raised for it. + /// + public TimeSpan WatchTimeout { get; } + + /// + /// Initializes what both modes share: the route, the order provider, and the two time settings + /// with their defaults. + /// + /// Where each state a sweep returns goes, normally the brokerage's message handler. + /// Resolves brokerage order ids to Lean orders. + /// How long the loop sleeps between sweeps. Null falls back to the + /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. + /// How long a watched order may stay unreported before + /// is raised. Null falls back to one minute. + protected BrokerageOrderPollingService(Action route, IOrderProvider orderProvider, + TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + { + _route = route; + _orderProvider = orderProvider; + PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3000)); + WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); + } + + /// + /// One read of the broker, giving the states the sweep saw. The loop calls it every + /// , hands each state to the route, and counts a throw as one failed sweep. + /// A null state is skipped: in per-id mode it means the broker does not know the id yet. + /// + protected abstract IEnumerable Sweep(); + + /// + /// A copy of the brokerage order ids a sweep still has to read: everything tracked whose end was + /// not reported yet. Taken under the registry lock, so an order placed mid-sweep is picked up by + /// the next one. + /// + protected List GetWatchedBrokerageIds() + { + lock (_lock) + { + var brokerageIds = new List(_orderStates.Count); + foreach (var (brokerageId, entry) in _orderStates) + { + if (!entry.TerminalReported) + { + brokerageIds.Add(brokerageId); + } + } + return brokerageIds; + } + } + + /// + /// Watches a brokerage order id, with nothing seen for it yet, so the first state to carry the id + /// acknowledges the order and of silence raises + /// . Idempotent: watching an already-watched id never overwrites + /// its state. + /// + /// The brokerage order id to watch. + public void Watch(string brokerageId) + { + Watch(brokerageId, lastSeen: null); + } + + /// + /// Watches a brokerage order id, seeded with what another path already reported, so the next poll + /// does not repeat it. Used for orders adopted at startup, for a submit reported from the request + /// path, and to move state onto the new id of a replace. Idempotent: watching an already-watched + /// id never overwrites its state. + /// + /// The brokerage order id to watch. + /// The state another path already reported for the order. + public void Watch(string brokerageId, BrokerOrderState lastSeen) + { + lock (_lock) + { + if (!_orderStates.TryGetValue(brokerageId, out var entry)) + { + entry = new OrderStateEntry(); + if (lastSeen != null) + { + entry.LastSeen = lastSeen; + entry.ReportedFilledQuantity = lastSeen.FilledQuantity ?? 0m; + // seeded means another path already heard from the broker about this order, + // and a seed carrying the order's end means the end was already reported + entry.Acknowledged = true; + entry.SubmitReported = lastSeen.Status != OrderStatus.New; + entry.TerminalReported = lastSeen.Status == OrderStatus.Canceled || lastSeen.Status == OrderStatus.Invalid; + } + _orderStates[brokerageId] = entry; + } + entry.Watched = true; + } + } + + /// + /// Stops watching an order and drops its state. + /// + /// The brokerage order id to stop watching. + public void Unwatch(string brokerageId) + { + lock (_lock) + { + _orderStates.Remove(brokerageId); + } + } + + /// + /// Records what another path already reported for an order, so the next poll does not repeat it. + /// Called by the streaming path while the stream lives, after it reports its own event. + /// + /// The brokerage order id the state belongs to. + /// The cumulative state the other path reported. + public void UpdateOrderState(string brokerageId, BrokerOrderState orderState) + { + lock (_lock) + { + if (!_orderStates.TryGetValue(brokerageId, out var entry)) + { + entry = new OrderStateEntry(); + _orderStates[brokerageId] = entry; + } + + entry.LastSeen = orderState; + entry.Acknowledged = true; + // the already-reported quantity never shrinks + var filledQuantity = orderState.FilledQuantity ?? 0m; + if (filledQuantity > entry.ReportedFilledQuantity) + { + entry.ReportedFilledQuantity = filledQuantity; + } + // a state written by the other path means its submit is out, and a terminal state means + // the end was already reported - a later sweep must not repeat either + if (orderState.Status != OrderStatus.New) + { + entry.SubmitReported = true; + } + if (orderState.Status == OrderStatus.Canceled || orderState.Status == OrderStatus.Invalid) + { + entry.TerminalReported = true; + } + } + } + + /// + /// The last state seen for an order, from any path. The streaming path reads it for its own + /// duplicate check, and a replace reads it to move the state to the new id. + /// + /// The brokerage order id to look up. + /// When this method returns true, the last state seen; otherwise null. + /// true when a state was ever recorded for the id; otherwise false. + public bool TryGetLastOrderState(string brokerageId, out BrokerOrderState lastSeen) + { + lock (_lock) + { + lastSeen = null; + if (_orderStates.TryGetValue(brokerageId, out var entry)) + { + lastSeen = entry.LastSeen; + } + return lastSeen != null; + } + } + + /// + /// Compares a state with the last one seen for the same order and raises + /// with what is new: the submit first, then fills, then a close. Register it on the message handler + /// for , so polled orders queue behind an order request that holds the + /// stream lock. Not safe to run twice at the same time - the handler already runs it one call at a + /// time, and a caller without a handler must do the same. + /// + /// The state a sweep read from the broker. + public void ProcessOrderState(BrokerOrderState orderState) + { + if (orderState == null || string.IsNullOrEmpty(orderState.BrokerageOrderId)) + { + return; + } + + var brokerageId = orderState.BrokerageOrderId; + + // record the id as seen: the broker knows the order, so the watch timeout stops counting + lock (_lock) + { + if (_orderStates.TryGetValue(brokerageId, out var seenEntry)) + { + seenEntry.Acknowledged = true; + } + } + + // a list, because combo legs can share one brokerage id and each leg is its own Lean order + var leanOrders = _orderProvider?.GetOrdersByBrokerageId(brokerageId); + if (leanOrders == null || leanOrders.Count == 0) + { + // not ours, or ours with the id not on the Lean order yet - the next sweep sees it again + return; + } + + if (leanOrders.TrueForAll(order => order.Status.IsClosed())) + { + // nothing left to report; dropping the state here is the only safe moment, because Lean + // has already applied the end of the order + Unwatch(brokerageId); + return; + } + + var timeUtc = orderState.TimeUtc == default ? DateTime.UtcNow : orderState.TimeUtc; + var orderEvents = new List(); + + // the whole compare runs under the registry lock, so the streaming path writing the same entry + // through UpdateOrderState can never interleave with the diff's read-then-write bookkeeping + lock (_lock) + { + if (!_orderStates.TryGetValue(brokerageId, out var entry)) + { + entry = new OrderStateEntry(); + _orderStates[brokerageId] = entry; + } + entry.Acknowledged = true; + + // the submit first, once: when nothing was emitted for the id yet, the Lean order is still New, + // and the state is not a reject. Lean requires it before any fill, and a market order can + // already be Filled the first time a poll sees it. + if (!entry.SubmitReported + && (entry.LastSeen == null || entry.LastSeen.Status == OrderStatus.New) + && orderState.Status != OrderStatus.Invalid) + { + foreach (var leanOrder in leanOrders) + { + if (leanOrder.Status == OrderStatus.New) + { + orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero, "Submitted by order polling") + { + Status = OrderStatus.Submitted + }); + entry.SubmitReported = true; + } + } + } + + // then the fills, so a close can never outrun a fill of the same order. A fill needs both + // numbers: without a price the service would have to invent one, and it never invents a + // number - a read without prices simply reports less. + if (orderState.FilledQuantity.HasValue && orderState.FillPrice.HasValue) + { + var cumulativeFilled = orderState.FilledQuantity.Value; + var newPart = cumulativeFilled - entry.ReportedFilledQuantity; + if (newPart > 0m) + { + var fillPrice = orderState.FillPrice.Value; + if (leanOrders.Count == 1) + { + var leanOrder = leanOrders[0]; + if (!leanOrder.Status.IsClosed()) + { + orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero) + { + Status = cumulativeFilled >= leanOrder.AbsoluteQuantity ? OrderStatus.Filled : OrderStatus.PartiallyFilled, + FillQuantity = leanOrder.Direction == OrderDirection.Sell ? -newPart : newPart, + FillPrice = fillPrice + }); + } + entry.ReportedFilledQuantity = cumulativeFilled; + } + else if (leanOrders[0].GroupOrderManager == null || leanOrders[0].GroupOrderManager.Quantity == 0m) + { + Log.Error($"{GetType().Name}.{nameof(ProcessOrderState)}(): cannot split the fill of brokerage order '{brokerageId}' " + + $"across {leanOrders.Count} Lean orders without a group quantity, skipping the fill."); + } + else + { + // one brokerage id, many Lean leg orders: the state counts in strategy units, and + // each leg gets its share of the new part, sized by its own quantity + var groupQuantity = Math.Abs(leanOrders[0].GroupOrderManager.Quantity); + foreach (var leanOrder in leanOrders) + { + if (leanOrder.Status.IsClosed()) + { + continue; + } + orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero) + { + Status = cumulativeFilled >= groupQuantity ? OrderStatus.Filled : OrderStatus.PartiallyFilled, + FillQuantity = leanOrder.Quantity * newPart / groupQuantity, + FillPrice = fillPrice + }); + } + entry.ReportedFilledQuantity = cumulativeFilled; + } + } + } + + // the end of the order last, once. The id leaves the read list, but its state stays until a + // compare sees the Lean order closed - forgetting it here would re-report every fill if the + // next sweep lands before Lean applies this event. + if ((orderState.Status == OrderStatus.Canceled || orderState.Status == OrderStatus.Invalid) && !entry.TerminalReported) + { + foreach (var leanOrder in leanOrders) + { + if (!leanOrder.Status.IsClosed()) + { + orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero, orderState.Message) + { + Status = orderState.Status + }); + } + } + entry.TerminalReported = true; + } + + entry.LastSeen = orderState; + } + + if (orderEvents.Count > 0) + { + OrderEvents?.Invoke(this, orderEvents); + } + } + + /// + /// Starts the background polling task. Idempotent while running; after a later + /// call resumes polling. Does nothing once the service has been disposed. + /// + public void Start() + { + lock (_lock) + { + // A run is active while the source exists: Start creates it, Stop clears it. A stopped + // run's task exits on its own without handling more orders, so a new run does not wait for it. + if (_disposed || _cancellationTokenSource != null) + { + return; + } + + _isPolling = true; + _cancellationTokenSource = new CancellationTokenSource(); + var cancellationToken = _cancellationTokenSource.Token; + // Task.Run so the first read starts on a pool thread instead of blocking the caller. + _pollingTask = Task.Run(() => PollLoop(cancellationToken)); + } + } + + /// + /// Stops the polling loop but keeps the service usable, so a later resumes + /// polling. The registry survives a stop, so nothing already reported repeats after a restart. + /// + public void Stop() + { + Task pollingTask; + CancellationTokenSource cancellationTokenSource; + lock (_lock) + { + pollingTask = _pollingTask; + cancellationTokenSource = _cancellationTokenSource; + _cancellationTokenSource = null; + _pollingTask = null; + _isPolling = false; + } + + if (cancellationTokenSource == null) + { + return; + } + + cancellationTokenSource.Cancel(); + + // Dispose the source only once the loop has actually stopped, so the loop never waits on a + // disposed handle. If the loop is still blocked on a slow read, leave the source for the GC. + if (pollingTask == null || pollingTask.Wait(TimeSpan.FromSeconds(2))) + { + cancellationTokenSource.DisposeSafely(); + } + } + + /// + /// Stops the polling loop and marks the service as disposed, so it cannot be started again. + /// + public void Dispose() + { + lock (_lock) + { + if (_disposed) + { + return; + } + _disposed = true; + } + + Stop(); + } + + /// + /// Re-reads the broker on each sweep and routes every state, until cancelled. + /// + /// Cancelled to stop the loop. + private async Task PollLoop(CancellationToken cancellationToken) + { + Log.Trace($"{GetType().Name}.{nameof(PollLoop)}(): started, polling every {PollInterval.TotalMilliseconds}ms."); + + // per run, so a stopped loop still draining a slow read never shares them with the next run + var consecutiveFailureCount = 0; + var isPollingFailureReported = false; + + while (!cancellationToken.IsCancellationRequested) + { + try + { + foreach (var orderState in Sweep()) + { + if (cancellationToken.IsCancellationRequested) + { + break; + } + + // a per-id read returns null when the broker does not know the id yet + if (orderState != null) + { + _route(orderState); + } + } + + consecutiveFailureCount = 0; + isPollingFailureReported = false; + + // silence only means something after a read that succeeded: a failed sweep asked + // the broker nothing, so it must not count against a watched order + if (!cancellationToken.IsCancellationRequested) + { + CheckWatchTimeouts(); + } + } + catch (Exception ex) + { + // A transient read failure must not kill the loop: log and try again next sweep. + Log.Error($"{GetType().Name}.{nameof(PollLoop)}(): failed to poll orders: {ex.Message}"); + + // A failure that keeps coming back is not transient any more, and the run may have no + // order updates at all while it lasts, so say so once instead of only a log line. + if (++consecutiveFailureCount >= ConsecutiveFailuresBeforeReport && !isPollingFailureReported) + { + isPollingFailureReported = true; + Message?.Invoke(this, new BrokerageMessageEvent(BrokerageMessageType.Warning, "OrderPollingFailed", + $"Several order reads in a row failed, so no order update is reported until a read succeeds: {ex.Message}")); + } + } + + try + { + await Task.Delay(PollInterval, cancellationToken).ConfigureAwait(false); + } + catch (OperationCanceledException) + { + break; + } + } + + Log.Trace($"{GetType().Name}.{nameof(PollLoop)}(): stopped."); + } + + /// + /// Counts one interval of silence for every watched order the broker never reported, and raises + /// once for each one that reached the watch timeout. Called only + /// after a successful sweep, because only a read that succeeded proves the silence is real. + /// + private void CheckWatchTimeouts() + { + List expired = null; + lock (_lock) + { + foreach (var (brokerageId, entry) in _orderStates) + { + if (!entry.Watched || entry.Acknowledged) + { + continue; + } + + entry.UnacknowledgedFor += PollInterval; + if (entry.UnacknowledgedFor >= WatchTimeout) + { + (expired ??= new()).Add(new OrderNotAcknowledgedEventArgs(brokerageId, entry.UnacknowledgedFor)); + } + } + + if (expired != null) + { + foreach (var eventArgs in expired) + { + _orderStates.Remove(eventArgs.BrokerageOrderId); + } + } + } + + if (expired != null) + { + foreach (var eventArgs in expired) + { + OrderNotAcknowledged?.Invoke(this, eventArgs); + } + } + } + + /// + /// What the registry keeps per brokerage order id. + /// + private class OrderStateEntry + { + /// + /// The last state seen for the order, from any path. Null when nothing was seen yet, so the + /// submit is still due. + /// + public BrokerOrderState LastSeen; + + /// + /// The cumulative filled quantity already reported to Lean, by any path. Never shrinks. + /// + public decimal ReportedFilledQuantity; + + /// + /// Set once the submit was reported for the order, by any path, so it goes out exactly once. + /// + public bool SubmitReported; + + /// + /// Set once the order's end was reported, so the id leaves the read list and a later state + /// for it reports nothing new. + /// + public bool TerminalReported; + + /// + /// Set by : the watch timeout only applies to explicitly watched orders. + /// + public bool Watched; + + /// + /// Set once anything carried the id: a polled state, a stream write, or a seed. Stops the + /// watch timeout. + /// + public bool Acknowledged; + + /// + /// How long the order has been watched with nothing reporting it, in polling time. + /// + public TimeSpan UnacknowledgedFor; + } + } +} diff --git a/Brokerages/Services/OrderNotAcknowledgedEventArgs.cs b/Brokerages/Services/OrderNotAcknowledgedEventArgs.cs new file mode 100644 index 000000000000..d6bd0b33649f --- /dev/null +++ b/Brokerages/Services/OrderNotAcknowledgedEventArgs.cs @@ -0,0 +1,48 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; + +namespace QuantConnect.Brokerages.Services +{ + /// + /// Raised when a watched brokerage order id went unreported for the whole watch timeout of polling. + /// This is a question, not a verdict: the service does not know whether the order never reached the + /// broker or closed before the first sweep saw it. The brokerage decides what to do next. + /// + public class OrderNotAcknowledgedEventArgs : EventArgs + { + /// + /// The brokerage order id nothing ever reported. + /// + public string BrokerageOrderId { get; } + + /// + /// How long the id was watched, in polling time, before the timeout fired. + /// + public TimeSpan WatchedFor { get; } + + /// + /// Creates a new . + /// + /// The brokerage order id nothing ever reported. + /// How long the id was watched, in polling time, before the timeout fired. + public OrderNotAcknowledgedEventArgs(string brokerageOrderId, TimeSpan watchedFor) + { + BrokerageOrderId = brokerageOrderId; + WatchedFor = watchedFor; + } + } +} diff --git a/Brokerages/Services/PerOrderIdPollingService.cs b/Brokerages/Services/PerOrderIdPollingService.cs new file mode 100644 index 000000000000..eea946e3809b --- /dev/null +++ b/Brokerages/Services/PerOrderIdPollingService.cs @@ -0,0 +1,86 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using QuantConnect.Logging; +using QuantConnect.Securities; +using System.Collections.Generic; + +namespace QuantConnect.Brokerages.Services +{ + /// + /// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id - + /// no request when nothing is watched. A null return means the broker does not know the id, so the + /// watch timeout keeps counting. + /// + public class PerOrderIdPollingService : BrokerageOrderPollingService + { + /// + /// Reads the current state of one order by its brokerage id. + /// + private readonly Func _readOrder; + + /// + /// Creates a new . + /// + /// Reads the current state of one order by its brokerage id. A null + /// return means the broker does not know the id. + /// Where each state a sweep returns goes, normally the brokerage's message handler. + /// Resolves brokerage order ids to Lean orders. + /// How long the loop sleeps between sweeps. Null falls back to the + /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. + /// How long a watched order may stay unreported before + /// is raised. Null falls back to one minute. + public PerOrderIdPollingService(Func readOrder, Action route, + IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + : base(route, orderProvider, pollInterval, watchTimeout) + { + _readOrder = readOrder; + } + + /// + /// Calls the read once per watched brokerage id. One id whose read throws is logged and skipped, + /// so it cannot starve the other watched orders; the sweep only counts as failed when every read + /// of the sweep failed. + /// + protected override IEnumerable Sweep() + { + var brokerageIds = GetWatchedBrokerageIds(); + var orderStates = new List(brokerageIds.Count); + var failedReadCount = 0; + var lastError = default(Exception); + foreach (var brokerageId in brokerageIds) + { + try + { + orderStates.Add(_readOrder(brokerageId)); + } + catch (Exception ex) + { + failedReadCount++; + lastError = ex; + Log.Error($"{nameof(PerOrderIdPollingService)}.{nameof(Sweep)}(): failed to read order '{brokerageId}': {ex.Message}"); + } + } + + if (failedReadCount > 0 && failedReadCount == brokerageIds.Count) + { + throw lastError; + } + + return orderStates; + } + } +} diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index a08d26cad02d..835774f7c781 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -20,7 +20,7 @@ and it can stay up while quietly missing an update. In all four shapes the broke and nobody asks. This document proposes one helper class in Lean core, `BrokerageOrderPollingService`, that any brokerage can create -and use. The brokerage picks one of two constructors — read one order by its brokerage id, or read all orders — and +and use. The brokerage picks one of two classes — read one order by its brokerage id, or read all orders — and hands the service a read callback that converts each order from the broker's own model into one shared snapshot shape. Every N seconds the service runs the read, and the snapshots travel through the brokerage's message handler back into the service, which compares each one with the last state it has seen for that order and raises an event @@ -180,14 +180,15 @@ Two rules shape what the service does with a snapshot: One more fact, this one about the modes: IB **cannot ask the broker about one order id**. Its API only returns all open orders (`reqOpenOrders` / `reqAllOpenOrders`, with a 15 second wait — `InteractiveBrokersBrokerage.cs:626`). So the service cannot require per-order reads from every brokerage. Which orders one read covers is the brokerage's -choice, made when it picks the constructor. +choice, made when it picks the class. ## Design ### Where it lives -`Lean/Brokerages/Services/BrokerageOrderPollingService.cs`, namespace `QuantConnect.Brokerages.Services`, with -`BrokerOrderState.cs` next to it. +`Lean/Brokerages/Services/BrokerageOrderPollingService.cs` — the base class — with +`PerOrderIdPollingService.cs`, `AllOrdersPollingService.cs` and `BrokerOrderState.cs` next to it, all in +namespace `QuantConnect.Brokerages.Services`. `Services` is a new folder under `Brokerages`, and it follows the convention the other subfolders there already use: `Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. It also matches where @@ -200,7 +201,7 @@ Tests go to `Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs`. ``` poll thread — one sweep every PollInterval - service calls the read: once per watched id, or once for all orders (by constructor) + service calls the read: once per watched id, or once for all orders (by class) brokerage the read asks the broker and converts each order to a BrokerOrderState service sends each state to the brokerage's message handler (waits in the queue while an order request holds the lock) @@ -208,7 +209,7 @@ poll thread — one sweep every PollInterval service checks the watched orders: one the broker never reported is flagged after a timeout handler thread — when the message handler takes a state from the queue - brokerage hands it back: service.ProcessOrderState(orderState) + handler dispatches it to what the brokerage registered: service.ProcessOrderState(orderState) service compares it with the last state seen and keeps only what is new service raises OrderEvents: the submit first, then fills, then a close brokerage forwards them: OnOrderEvents(events) -> Lean applies them @@ -363,31 +364,25 @@ namespace QuantConnect.Brokerages.Services; /// /// Reads orders from the brokerage on an interval and turns the returned snapshots into order events. /// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order -/// the broker never replied about. +/// the broker never replied about. The base class owns everything both modes share — the loop, the +/// watch registry, the compare and the events. What one sweep reads is the subclass: +/// or . /// -public class BrokerageOrderPollingService : IDisposable +public abstract class BrokerageOrderPollingService : IDisposable { - /// - /// For a broker with a get-order endpoint. Every the service calls - /// once per watched brokerage id — no request when nothing is watched. - /// A null return means the broker does not know the id, so the watchdog keeps counting. - /// Each snapshot is handed to , normally the brokerage's message handler. - /// - public BrokerageOrderPollingService(Func readOrder, Action route, - IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + /// Initializes what both modes share: the route, the order provider, and the two time + /// settings with their defaults. Each snapshot the sweep returns is handed to + /// , normally the brokerage's message handler. + protected BrokerageOrderPollingService(Action route, IOrderProvider orderProvider, + TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); - /// - /// For a broker with only a bulk endpoint. Every the service calls - /// once, whatever is watched. - /// Each snapshot is handed to , normally the brokerage's message handler. - /// - public BrokerageOrderPollingService(Func> readAllOrders, Action route, - IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + /// One read of the broker, giving the states the sweep saw. The loop calls it every + /// poll interval, hands each state to the route, and counts a throw as one failed sweep. + protected abstract IEnumerable Sweep(); - /// Initializes what both modes share: the route, the order provider, and the two time - /// settings with their defaults. The public constructors chain here and only set their read. - private BrokerageOrderPollingService(Action route, IOrderProvider orderProvider, - TimeSpan? pollInterval, TimeSpan? watchTimeout); + /// A copy of the ids a sweep still has to read: everything tracked whose end was not + /// reported yet. + protected List GetWatchedBrokerageIds(); /// The order events one snapshot produced. Raised inside , never empty. public event EventHandler> OrderEvents; @@ -402,6 +397,12 @@ public class BrokerageOrderPollingService : IDisposable /// True while the polling task is running. public bool IsPolling { get; } + /// How long the loop sleeps between sweeps. + public TimeSpan PollInterval { get; } + + /// How long a watched order may stay completely unreported, in polling time. + public TimeSpan WatchTimeout { get; } + /// Watches a brokerage order id, with nothing seen for it yet. Idempotent: watching an /// already-watched id never overwrites its state. public void Watch(string brokerageId); @@ -424,8 +425,9 @@ public class BrokerageOrderPollingService : IDisposable /// /// Compares a snapshot with the last state seen for the same order and raises - /// with what is new. Call it from the message handler, so polled orders - /// queue behind an order request that holds the stream lock. + /// with what is new. Register it on the message handler for + /// , so polled orders queue behind an order request that holds + /// the stream lock. /// public void ProcessOrderState(BrokerOrderState orderState); @@ -433,35 +435,61 @@ public class BrokerageOrderPollingService : IDisposable public void Stop(); public void Dispose(); } + +/// +/// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id — +/// no request when nothing is watched. A null return means the broker does not know the id, so the +/// watch timeout keeps counting. A read that throws is logged and skipped, so one bad id cannot +/// starve the others; the sweep only counts as failed when every read of the sweep failed. +/// +public class PerOrderIdPollingService : BrokerageOrderPollingService +{ + public PerOrderIdPollingService(Func readOrder, Action route, + IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); +} + +/// +/// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched. +/// +public class AllOrdersPollingService : BrokerageOrderPollingService +{ + public AllOrdersPollingService(Func> readAllOrders, Action route, + IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); +} ``` -The loop, `Start`, `Stop`, `Dispose` and the failure counter are lifted from Schwab's service, which is the more -complete of the two. The watch registry, the snapshot compare and the seed-on-adopt come from Public's. +The loop, `Start`, `Stop`, `Dispose` and the failure counter come from Schwab's service, the more complete one. +The watch registry, the compare with the last state seen and the seeding of orders already open at startup come from Public's. ### The two modes -The mode is the constructor. Both run the same diff; they differ only in what one sweep reads: +The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `Sweep`: -- **Per order id** — `Func`: the service loops the watched ids and calls the read once per - id. Nothing watched, nothing requested. Public.com's own service has exactly this constructor today — - `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)`. -- **All orders** — `Func>`: the service calls the read once per sweep, and the read - returns everything the broker lists. +- **`PerOrderIdPollingService`** — `Func`: the sweep loops the watched ids and calls the + read once per id. Nothing watched, nothing requested. Public.com's own service has exactly this constructor + today — `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)`. +- **`AllOrdersPollingService`** — `Func>`: the sweep calls the read once, and the + read returns everything the broker lists. -Two constructors instead of one `Func` for both, because the bulk case does not fit a per-id -signature: called once per watched id, a bulk read would repeat the full-account request for every id in the same -sweep, and with nothing watched (Schwab's fallback watches nothing — it is the whole order path) it would never run -at all. +Two classes instead of one class with both reads, because a bulk read does not fit a per-id shape: called once +per watched id, it would repeat the full-account request for every id in the same sweep, and with nothing watched +it would never run at all (Schwab's fallback watches nothing — it is the whole order path). The split also keeps +each class clean: one class with both reads would hold a null field for the unused mode and a branch in the loop +to pick the right read; a subclass holds only its own read. -The watch registry serves both modes. In per-id mode it is also the read list. In all-orders mode it is only the -watchdog: the service still checks that every watched id eventually shows up in the snapshots. +The watch registry serves both modes. In per-id mode it is also the read list. In all-orders mode it only feeds +the watch timeout: the service still checks that every watched id shows up in the snapshots sooner or later. -| Plugin | The action reads | Scope | Why | +| Plugin | The sweep reads | Scope | Why | | --- | --- | --- | --- | | CharlesSchwab | bulk | all | one request returns the whole account, and Schwab is the order path for the run | | Public.com | per id | watched | Public.com has a get-order endpoint and only cares about its own orders | | InteractiveBrokers | bulk | watched | IB has **no** per-order request — `reqOpenOrders` always returns everything | -| Tradier | bulk | watched | same as IB: the unknown ids are re-checked against a full read | +| Tradier | bulk | watched | same as IB: the unknown ids are re-checked against a full read (`GetIntradayAndPendingOrders`, `TradierBrokerage.cs:1266`) | +| Webull | bulk | watched | its only order read is `GetOpenOrders` (`Api/ApiClient.cs:666`); order updates come over a gRPC stream, and the poll covers its drops | +| TradeStation | bulk | watched | one `GetOrders` request returns the account's orders (`Api/TradeStationApiClient.cs:185`); the stream stays the first path | +| Alpaca | per id | watched | the SDK has `GetOrderAsync` and the plugin already calls it (`AlpacaBrokerage.cs:546`), so a watch asks only about its own orders | +| Binance | bulk | watched | a single-order request needs the symbol next to the id and the plugin never built one; `GetOpenOrders` sits on the shared REST client base, so every market variant has it | ### The service never raises Lean order events itself @@ -473,22 +501,71 @@ still reporting `Submitted` for the same order. If the fill goes out first, the back to open, and Lean then accepts a cancel or update on it that the broker rejects. Schwab hit exactly this and fixed it by pushing polled orders through the same message handler as stream messages, so they wait in the queue while an order request holds the stream lock (`WithLockedStream`, -`Brokerages/BrokerageConcurrentMessageHandler.cs:99`). That is why `ProcessOrderState` is a separate public method +`Brokerages/BrokerageMessageQueue.cs:98`). That is why `ProcessOrderState` is a separate public method instead of something the loop calls itself: the brokerage puts its message handler between the read and the diff, -and the queue keeps the order right. The handler's message type just has to cover both the stream's model and the -snapshot — Schwab uses a small marker interface for exactly this. +and the queue keeps the order right. The handler's message type also has to cover both the stream's model and the +snapshot — the next section is how it does. One requirement travels with the queue: the place request, the `BrokerId` assignment and the `Submitted` report must all happen inside the same `WithLockedStream` block that the snapshots queue behind. That is what guarantees a queued snapshot always resolves the id and finds the status the request already reported. Schwab's fallback path does all three inside the lock today. +### One lock for two message types + +`BrokerageConcurrentMessageHandler` knows exactly one message type, so a plugin whose stream model and snapshot +differ had no way to push both through one lock. Schwab works around it with a marker interface: its stream model +and its polled model both implement `IOrderUpdateMessage`, and the handler is typed to that. This works inside one +plugin that owns both models, and it cannot be the shared answer — every plugin that adopts the service would +have to add the interface to its own wire models, and the core `BrokerOrderState` cannot implement a per-plugin +interface. + +So the handler is split instead, and the split ships with the service. The lock, the buffer and the drain loop +move unchanged into a new class, `BrokerageMessageQueue` (`Brokerages/BrokerageMessageQueue.cs`): the buffer holds +`object`, and every dequeued message is raised through a `MessageReceived` event, in arrival order. +`BrokerageConcurrentMessageHandler` stays as a thin wrapper over one queue, with the same public surface every +plugin compiles against today, plus one method: + +```csharp +// the stream type, exactly as today +_messageHandler = new BrokerageConcurrentMessageHandler(OnAccountContent, concurrencyEnabled); + +// one more line, and snapshots share the same lock — no marker interface, no second handler +_messageHandler.RegisterMessageType(_orderPollingService.ProcessOrderState); + +// one method for both; the compiler picks the type from the argument +_messageHandler.HandleNewMessage(accountContent); +_messageHandler.HandleNewMessage(orderState); +``` + +`RegisterMessageType` subscribes a filter on the queue's event: a dequeued message runs one `is` check per +registered type and lands in the action that matches. `HandleNewMessage` becomes generic, with the type inferred +from the argument, so every existing call in every plugin compiles unchanged. + +The stream's hot path pays nothing for this. The filter is created once at registration, never per message; the +buffer stored references before and stores references now (`T` was already constrained to `class`); the lock code +moved without an edit. Per message, the old direct delegate call becomes an event invoke plus one type check per +registered type — nanoseconds next to the lock the path already takes. The existing handler tests pass against +the split unchanged (`Tests/Brokerages/BrokerageConcurrentMessageHandlerTests.cs` — ordering, backpressure, single +drainer, exception recovery), and the shared lock across two types has its own +(`Tests/Brokerages/BrokerageMessageQueueTests.cs`). + +One rule of the split is absolute: **the generic `BrokerageConcurrentMessageHandler` stays.** Thirteen live +plugins hold a field of it today — CharlesSchwab, Public.com, Alpaca, Binance, TradeStation, Tastytrade, Webull, +ByBit, Eze, OANDA, IG, dYdX and TerminalLink — plus the Template scaffold. Removing or reshaping `` breaks all +of them at once. So the class keeps its name, both constructors, `HandleNewMessage(T message)`, +`WithLockedStream` and `Dispose`, and every plugin recompiles with zero edits. The riskiest call shape was +checked one repo at a time: Schwab, Public and Eze pass `_messageHandler.HandleNewMessage` as a delegate +(`CharlesSchwabBrokerage.OrderUpdatePolling.cs:65`, `PublicBrokerage.cs:182`, `EzeBrokerage.cs:259`), and all +three still compile; the one shape that could not — a bare `null` argument, which type inference cannot resolve — +appears in no repo. + ### The diff `ProcessOrderState` compares the snapshot with the last state it has seen for that brokerage id: ``` -record the id as seen, for the watchdog +record the id as seen, for the watch timeout find the Lean orders by brokerage id (IOrderProvider.GetOrdersByBrokerageId — a list, because combo legs can share one id and each leg is its own Lean order) none found -> skip and write nothing: not ours, or ours with the id not on the @@ -502,6 +579,8 @@ second gate matters in bulk mode beside a live stream: orders the stream already confirmed are not New in Lean, so a sweep seeing them for the first time stays quiet. then the fills, so a close can never outrun a fill of the same order: + a fill needs both numbers: without a FillPrice nothing is emitted and + alreadyReported does not move - the service never invents a price the new part is FilledQuantity - alreadyReported; nothing at zero or below one fill event, priced at the state's FillPrice alreadyReported never shrinks, and it moves only when a part was actually emitted @@ -575,9 +654,10 @@ diffs. With the snapshot the diff is shared, whole, and the plugins keep only th - `watchTimeout` of polling passes and nothing ever carried the id → `OrderNotAcknowledged` is raised once, with the id and how long it was watched, and the id is unwatched. -The timeout clock runs only while the service is polling. A watch set while the service is stopped does not count -down — otherwise every healthy order would trip the watchdog the moment polling starts. So a watch-mode plugin -calls `Start` together with `Watch` (`Start` is idempotent) and may `Stop` once nothing is watched. +The timeout only counts while the service is polling, and only on sweeps whose read succeeded — a failed read +asked the broker nothing, so it proves no silence. A watch set while the service is stopped does not count — +otherwise every healthy order would hit the timeout the moment polling starts. So a plugin in watch mode calls +`Start` together with `Watch` (calling `Start` twice is fine) and may `Stop` once nothing is watched. `OrderNotAcknowledged` is a question, not a verdict — this is rule 2 again, a missing order proves nothing. The service does not know whether the order never arrived or filled instantly, so it does not decide. The brokerage handles it: @@ -592,20 +672,54 @@ what the other already reported. The registry is that shared memory, in both dir - **The stream writes what it reports.** After reporting a stream fill, the plugin calls `UpdateOrderState` with the new cumulative state. The next poll compare starts from it and repeats nothing. -- **The stream reads before it reports.** Schwab's stream handler already keeps a cumulative-quantity dictionary to - drop duplicate stream events (`CharlesSchwabBrokerage.DataQueueHandler.cs:419-425`); `TryGetLastOrderState` is that - dictionary, shared. A stream update at or below the registry's quantity was already reported — by either path — - and is dropped. +- **The stream reads before it reports.** The stream's own duplicate check is not enough in watch mode, because it + does not know what the poll already reported. `TryGetLastOrderState` fills that gap: a stream update at or below + the registry's quantity was already reported — by either path — and is dropped. + +Without that write, watch mode reports a fill twice: the poll reports it, the stream reports the same fill a +moment later, and Lean applies both. This is the one wiring rule a plugin must follow when it polls while its +stream is alive. -Without the write half, watch mode double-counts: the poll reports a fill, the stream reports the same fill a -moment later, and Lean applies both. This is the one wiring rule that is not optional for a plugin that polls -while its stream lives. +A fallback-mode plugin needs none of that wiring, because its stream never comes back once polling starts. Its +stream handler makes no registry calls — Schwab's keeps only its own cumulative-quantity dictionary, untouched. +Everything it owes the service is one seed at the moment the stream dies (see "Seed before Start"). The registry +complements the plugin's existing logic, it never replaces it — and in fallback mode the stream and the service +never even touch while the stream lives. A replace moves the state, because the registry is keyed by brokerage id and a replace gives the order a new one. Inside the same locked block that reports `UpdateSubmitted`, the plugin moves it across: -`TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. The same seed -rule covers a plugin that reports `Submitted` from the request path, as Schwab does without the stream: watch the -new id with a `Submitted` snapshot in that block, so the next sweep does not repeat the submit. +`TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. A broker +whose replacement counts its executions from zero seeds the new id with a fresh `Submitted` snapshot instead of +moving the old state — Schwab's replace path does exactly that. The same seed rule covers a plugin that reports +`Submitted` from the request path, as Schwab does without the stream: watch the new id with a `Submitted` snapshot +in that block, so the next sweep does not repeat the submit. + +### Seed before Start + +Polling never starts first: the stream reported orders before it, and the registry must know what was already +reported before the first sweep runs. So every `Start` that follows stream time begins with a handover: + +1. Drain the message buffer with an empty `WithLockedStream` block, so every fill the stream already delivered is + counted. Nothing slips in after the drain, because the switch runs on the stream's own thread — the only thread + that delivers stream messages. +2. Seed the registry with one `Watch(id, lastSeen)` per open Lean order: the status comes from the Lean order, the + cumulative filled quantity from the plugin's own bookkeeping. Orders the stream already closed need no seed — + the diff skips every order Lean has closed. +3. `Start`. + +The first sweep then continues from what the stream reported instead of repeating it. A fallback-mode plugin does +this once, because its stream never comes back — Schwab's `SeedRegistryFromOpenOrders` is the working example. A +gap-mode plugin repeats it on every drop: seed, `Start`, and `Stop` when the stream returns. + +The seed source already exists in most streaming plugins, because they keep the same bookkeeping Schwab does — the +cumulative quantity already reported, per Lean order: Webull's `_orderIdToPreviousCumulativeQuantity` +(`WebullBrokerage.cs:78`), ByBit's `_cumulativeFillQuantity` (`BybitBrokerage.Messaging.cs:42`), Alpaca's and +TradeStation's `_orderIdToFillQuantity` (`AlpacaBrokerage.cs:59`, `TradeStationBrokerage.cs:132` — both signed, so +their seed takes the absolute value). Webull is the clearest next adopter: a dropped stream is a blind gap today, +nothing replays the missed events, and its `StreamDisconnected`/`StreamReconnected` events are ready-made +`Start`/`Stop` triggers. TradeStation needs the seed only for the outage window itself, because its server replays +an order snapshot on every reconnect. Two would add the dictionary first: Tastytrade tracks processed fill ids +instead of quantities, and Binance keeps nothing — its stream reports the per-fill delta the wire sends. ### Start and Stop follow the stream @@ -614,17 +728,19 @@ down and `Stop` when it comes back, so polling covers exactly the window in whic idempotent, and `Stop` does not dispose the service, so a run can switch back and forth as many times as the socket does. `Dispose` is the one-way door. -Two rules for adopters: +Three rules for a plugin that uses this: +- **Seed before every `Start`.** While the socket was up the stream was reporting and the registry was not + listening. The seed hands over what was reported — see "Seed before Start". - **Sweep once after the stream returns, before stopping.** The socket coming back does not replay what it missed, and no plugin re-reads orders on reconnect today. One last sweep closes the gap. - **Coming back is not always allowed.** Schwab must stay on polling for the rest of the run, because reconnecting takes the single streaming slot back from the other algorithm. That decision belongs to the plugin, not to this service, which is why the service only offers `Start` and `Stop` and never reconnects anything itself. -What a gap sweep recovers is decided by what the plugin's read carries. A read that only lists open orders recovers -missed submissions. A read that carries fill numbers recovers the missed fills too — Schwab's does. The service -reports whatever the states can prove, and nothing more. +What that sweep recovers depends on what the plugin's read returns. A read that only lists open orders brings back +missed submissions. A read with fill numbers brings back the missed fills too — Schwab's does. The service reports +what the states can prove, nothing more. ### Repeated read failures @@ -635,7 +751,7 @@ alone leaves the algorithm looking idle for no visible reason. Never an `Error`. An `Error` ends the run, which is the outcome this service exists to avoid. -### State the service owns, and what it does not +### What the service keeps, and what it does not The service owns the watch registry, the last snapshot seen and the already-reported quantity per order, the failure counter, and the polling task with its cancellation source. The streaming path shares the snapshot registry through @@ -644,17 +760,18 @@ does **not** own the Lean order state: that is read from `IOrderProvider` on eve drifts from what Lean actually knows. Four kinds of thread touch that state: the poll loop, the handler thread inside `ProcessOrderState`, the order -threads through `Watch`/`Unwatch`/`UpdateOrderState`, and the watchdog. One internal lock guards the registry against -all of them, and a per-id sweep copies the watched ids under that lock before reading, so `PlaceOrder` can watch a -new id mid-sweep. `ProcessOrderState` itself is not reentrant and does not need to be — the message handler already -serializes it, and an adopter without a handler must serialize the calls itself. +threads through `Watch`/`Unwatch`/`UpdateOrderState`, and the watch-timeout check. One internal lock protects the +registry from all of them. A per-id sweep copies the watched ids under that lock before it reads the broker, so +`PlaceOrder` can watch a new id while a sweep runs. `ProcessOrderState` must not run twice at the same time, and +the service does not guard that itself: the message handler already runs it one call at a time, and a plugin +without a handler must make its calls run one at a time too. ### Configuration Both time settings are optional constructor arguments. A plugin that has its own configuration key — Schwab keeps `charles-schwab-order-poll-interval-ms` — reads it and passes the value in, so no existing deployment changes. A -plugin that passes nothing gets the shared defaults, resolved once in the shared private constructor both public -ones chain to: +plugin that passes nothing gets the shared defaults, resolved once in the base class constructor both subclasses +chain to: ```csharp PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3000)); @@ -662,36 +779,37 @@ WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); ``` `brokerage-order-poll-interval-ms` is one new generic config entry, so the interval can be tuned once for every -brokerage that uses the default. Core helpers already work this way: `BrokerageConcurrentMessageHandler` reads +brokerage that uses the default. Core helpers already work this way: the message queue reads `brokerage-concurrent-message-handler-buffer-size` in its constructor -(`Brokerages/BrokerageConcurrentMessageHandler.cs:56`). The 3000 ms default is the value Schwab runs today. +(`Brokerages/BrokerageMessageQueue.cs:55`). The 3000 ms default is the value Schwab runs today. ## Wiring, per plugin -The general shape, for a streaming brokerage with a bulk endpoint — the constructor is the mode: +The general shape, for a streaming brokerage with a bulk endpoint — the class is the mode: ```csharp // bulk broker: one request per sweep reads the whole account -_orderPollingService = new BrokerageOrderPollingService( +_orderPollingService = new AllOrdersPollingService( () => _apiClient.GetAllOrders().Select(ToOrderState), // read: model -> snapshot _messageHandler.HandleNewMessage, // route: through the message handler _orderProvider, pollInterval: TimeSpan.FromSeconds(3), watchTimeout: TimeSpan.FromMinutes(1)); _orderPollingService.OrderEvents += (_, orderEvents) => OnOrderEvents(orderEvents); -// in the message handler callback, next to the stream messages -_orderPollingService.ProcessOrderState(orderState); +// the handler dequeues a snapshot and hands it to the diff +_messageHandler.RegisterMessageType(_orderPollingService.ProcessOrderState); ``` -A per-id broker only changes the read — this is Public.com's own constructor today, snapshot instead of its DTO: +A per-id broker only changes the class and the read — this is Public.com's own constructor today, snapshot +instead of its DTO: ```csharp -_orderPollingService = new BrokerageOrderPollingService( +_orderPollingService = new PerOrderIdPollingService( brokerageId => ToOrderState(_apiClient.GetOrderById(brokerageId)), _messageHandler.HandleNewMessage, _orderProvider); // nothing passed: brokerage-order-poll-interval-ms decides, default 3000 ms ``` -InteractiveBrokers, the plugin with no answer today, adopts the bulk constructor (it has no per-id request) with +InteractiveBrokers, the plugin with no answer today, adopts `AllOrdersPollingService` (it has no per-id request) with the thinnest possible mapping — id and status from the orders `reqAllOpenOrders` returns, fill fields left null. The fill numbers are not out of reach: the captured `IBApi.Order` already carries a `FilledQuantity` field, and the paired `orderStatus` callback adds the average fill price, so the mapping can grow later without a new endpoint. @@ -712,10 +830,11 @@ in the background. The `_noSubmissionOrderTypes` guess (`MarketOnOpen`, `ComboLe becomes a real check: the broker either lists the order, and `Submitted` is true, or it does not, and the brokerage is asked instead of Lean inventing an event. -The same two lines cover a dropped socket, in any streaming plugin: +The same three lines cover a dropped socket, in any streaming plugin: ```csharp // where the brokerage already handles the connection going down +SeedRegistryFromOpenOrders(); // one Watch(id, lastSeen) per open Lean order, see "Seed before Start" _orderPollingService.Start(); // on reconnect: one last sweep for the gap, then hand the job back to the stream @@ -728,7 +847,7 @@ inline `Task.Delay` block with `Watch` on the unknown ids. ## What stays in the plugin -- Reading the broker: the read callback and the constructor choice between per-id and all orders, plus how far +- Reading the broker: the read callback and the class choice between per-id and all orders, plus how far back a bulk read reaches (Schwab reads from its oldest open Lean order). - Converting model to state: the status mapping, combo leg ids (Schwab's `mainId + legId - 1`), reducing an execution history to the two numbers (Schwab sums its execution legs and takes the newest leg's price). A plugin @@ -757,6 +876,17 @@ inline `Task.Delay` block with `Watch` on the unknown ids. broker-to-Lean status mapping inside the wire DTOs as computed properties, and the service registry would hold whole wire objects alive as stored state (Schwab's `OrderResponse` carries the full execution history). A plain class the plugin fills is one pattern that works for all eight. +- **A marker interface as the message handler's type** — Schwab's current answer to two message types in one + handler (`IOrderUpdateMessage`). As the shared answer it fails the same way the core interface does: every + plugin edits its wire models, and the core `BrokerOrderState` cannot implement a per-plugin interface. + Replaced by the `BrokerageMessageQueue` split (see "One lock for two message types"). +- **A dual-generic handler**, `BrokerageConcurrentMessageHandler` with the stream type and the polled type. + Rejected: every existing plugin migrates to the new shape even with no polling, a plugin with no stream (IB) + has no honest `T`, and a third source would need ``. The queue split adds a type with a registration + call instead of a type parameter. +- **A handler base class that queues work items (`Action`) instead of messages.** The typed wrapper would then + wrap every stream message in a new closure — one allocation per message on the hottest path a brokerage has. + The queue split keeps the message itself in the buffer and allocates only at registration. - **Put it on the `Brokerage` base class, driven by the engine, like the cash sync.** `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:480`, called from `Engine/TransactionHandlers/BrokerageTransactionHandler.cs:731`) is the existing shape for core-driven periodic @@ -765,14 +895,14 @@ inline `Task.Delay` block with `Watch` on the unknown ids. has to be off by default — most plugins must not poll — and a base-class hook that nearly everyone turns off is worse than a class you create when you need it. - **Let the service call `OnOrderEvents` directly.** Simpler to wire and reintroduces the fill-before-submit bug in - every adopter. See "The service never raises Lean order events itself". + every plugin that uses it. See "The service never raises Lean order events itself". - **Emit `Canceled` when an order leaves the open list.** This is the tempting one, and it is wrong: an order that filled also leaves the open list, so this drops fills and desynchronizes holdings. It is why rule 2 exists — the service acts on what a snapshot says, never on an order going missing. - **Carry the execution history in the state** — a per-execution list of quantity, price and time, so every execution becomes its own event at its own price. The survey killed it: exactly one of the eight brokers (Schwab) returns such a list, and the cumulative compare already keeps quantities exact. The list would buy per-execution - price precision for one broker at the cost of a second diff branch every adopter has to reason about. If it is + price precision for one broker at the cost of a second diff branch every plugin has to reason about. If it is ever wanted, it comes back as one additive nullable field without breaking anyone — the same goes for a `GetOrderExecutions` API on `IBrokerage`. - **Leave it in the plugins.** It is already written three times, and the fourth copy would be IB's. @@ -782,19 +912,20 @@ inline `Task.Delay` block with `Watch` on the unknown ids. | Risk | What we do about it | | --- | --- | | A plugin maps a broker status to the wrong Lean status | The mapping is the same one its streaming path already needs, written once per plugin and covered by its own tests. The service only emits transitions, so a wrong mapping surfaces once, not as a flood. | -| A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the watchdog still fires. | +| A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the watch timeout still fires. | | Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier ships this trade-off today (`TradierBrokerage.cs:1469-1472`); a shorter poll interval narrows it. | | Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order is unacknowledged, and the interval is a constructor argument the plugin picks. | | A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | -| An adopter passes `ProcessOrderState` as the route and gets fill-before-submit | The route is documented as "your message handler". Schwab and Public have one; IB and Tradier do not, and adding one is named in their rollout steps. | +| A plugin passes `ProcessOrderState` as the route and gets fill-before-submit | The route is documented as "your message handler", and wiring it right is two lines: `RegisterMessageType(ProcessOrderState)` once, `HandleNewMessage` as the route. Schwab and Public have a handler; IB and Tradier do not, and adding one is named in their rollout steps. | | The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | -| The watchdog cannot tell "never arrived" from "filled instantly" | It does not try. `OrderNotAcknowledged` hands the question to the brokerage, which has the endpoints to answer it. | +| The watch timeout cannot tell "never arrived" from "filled instantly" | It does not try. `OrderNotAcknowledged` hands the question to the brokerage, which has the endpoints to answer it. | | Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | | A plugin starts polling on disconnect and forgets to stop on reconnect | Both paths are one line and sit next to the connection handling the plugin already has. The poll side repeats nothing the registry already holds, so the cost of forgetting is extra requests, not extra events. | ## Rollout -1. This PR: the service, the snapshot, and their unit tests in `Tests/Brokerages/Services/`, no brokerage changes. +1. This PR: the `BrokerageMessageQueue` split of the message handler, the service, the snapshot, and their unit + tests, no brokerage changes. The split keeps the handler's public surface, so every plugin compiles as before. 2. InteractiveBrokers: add the message handler it does not have, then replace the `NoBrokerageResponse` error and the invented `Submitted` with a watch. This is the proof that the abstraction holds for a plugin that did not write it. diff --git a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs new file mode 100644 index 000000000000..575a8d44d879 --- /dev/null +++ b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs @@ -0,0 +1,581 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using System.Linq; +using System.Threading; +using NUnit.Framework; +using QuantConnect.Orders; +using System.Collections.Generic; +using QuantConnect.Brokerages; +using QuantConnect.Brokerages.Services; + +namespace QuantConnect.Tests.Brokerages.Services +{ + [TestFixture] + public class BrokerageOrderPollingServiceTests + { + private OrderProvider _orderProvider; + private AllOrdersPollingService _service; + private List _orderEvents; + + [SetUp] + public void SetUp() + { + _orderProvider = new OrderProvider(); + _orderEvents = new List(); + // the read is unused: these tests drive the diff directly through ProcessOrderState + _service = new AllOrdersPollingService(() => Array.Empty(), route: null, _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(50), watchTimeout: TimeSpan.FromMilliseconds(120)); + _service.OrderEvents += (_, orderEvents) => _orderEvents.AddRange(orderEvents); + } + + [TearDown] + public void TearDown() + { + _service.Dispose(); + } + + private Order AddOrder(decimal quantity, string brokerageId, OrderStatus status = OrderStatus.New) + { + var order = new MarketOrder(Symbols.AAPL, quantity, new DateTime(2026, 8, 12, 14, 0, 0, DateTimeKind.Utc)); + order.Status = status; + order.BrokerId.Add(brokerageId); + _orderProvider.Add(order); + return order; + } + + private static BrokerOrderState State(string brokerageId, OrderStatus status, decimal? filled = null, decimal? price = null, string message = null) + { + return new BrokerOrderState + { + BrokerageOrderId = brokerageId, + Status = status, + FilledQuantity = filled, + FillPrice = price, + TimeUtc = new DateTime(2026, 8, 12, 14, 30, 0, DateTimeKind.Utc), + Message = message + }; + } + + [Test] + public void SubmitIsEmittedOnce() + { + AddOrder(100m, "42"); + + _service.ProcessOrderState(State("42", OrderStatus.Submitted)); + + Assert.AreEqual(1, _orderEvents.Count); + Assert.AreEqual(OrderStatus.Submitted, _orderEvents[0].Status); + + // the same state again reports nothing new + _service.ProcessOrderState(State("42", OrderStatus.Submitted)); + Assert.AreEqual(1, _orderEvents.Count); + } + + [Test] + public void FirstStateAlreadyFilledEmitsSubmitBeforeFill() + { + AddOrder(100m, "42"); + + // a market order can already be filled the first time a poll sees it + _service.ProcessOrderState(State("42", OrderStatus.Filled, filled: 100m, price: 310m)); + + Assert.AreEqual(2, _orderEvents.Count); + Assert.AreEqual(OrderStatus.Submitted, _orderEvents[0].Status); + Assert.AreEqual(OrderStatus.Filled, _orderEvents[1].Status); + Assert.AreEqual(100m, _orderEvents[1].FillQuantity); + Assert.AreEqual(310m, _orderEvents[1].FillPrice); + } + + [Test] + public void CumulativeFillsNeverRepeat() + { + // the ADR's worked example: long 1000, two 100-share fills at the same price + var order = AddOrder(1000m, "42", OrderStatus.Submitted); + order.Status = OrderStatus.Submitted; + + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 200m, price: 310m)); + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 200m, price: 310m)); + + var fills = _orderEvents.Where(orderEvent => orderEvent.FillQuantity != 0).ToList(); + Assert.AreEqual(2, fills.Count); + Assert.IsTrue(fills.All(orderEvent => orderEvent.FillQuantity == 100m && orderEvent.FillPrice == 310m)); + Assert.IsTrue(fills.All(orderEvent => orderEvent.Status == OrderStatus.PartiallyFilled)); + } + + [Test] + public void ShrinkingFilledQuantityEmitsNothing() + { + // a broker glitch: the cumulative total drops and comes back. Nothing below or at what was + // already reported may produce an event. + AddOrder(1000m, "42", OrderStatus.Submitted); + + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 50m, price: 310m)); + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + + var fills = _orderEvents.Where(orderEvent => orderEvent.FillQuantity != 0).ToList(); + Assert.AreEqual(1, fills.Count); + Assert.AreEqual(100m, fills[0].FillQuantity); + } + + [Test] + public void StreamWriteBelowTheReportedTotalNeverShrinksIt() + { + AddOrder(1000m, "42", OrderStatus.Submitted); + + // the poll reported 100, then the stream writes an older state with 50 + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + _service.UpdateOrderState("42", State("42", OrderStatus.PartiallyFilled, filled: 50m, price: 310m)); + + // the next sweep at 100 repeats nothing: the reported total never moved backwards + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + + var fills = _orderEvents.Where(orderEvent => orderEvent.FillQuantity != 0).ToList(); + Assert.AreEqual(1, fills.Count); + } + + [Test] + public void FilledQuantityWithoutPriceEmitsNoFill() + { + // a read that carries the quantity but no price cannot produce a fill event - the service + // never invents a number. The part stays unreported, so it goes out once the price arrives. + AddOrder(100m, "42", OrderStatus.Submitted); + + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 40m)); + Assert.IsEmpty(_orderEvents); + + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 40m, price: 310m)); + var fill = _orderEvents.Single(); + Assert.AreEqual(40m, fill.FillQuantity); + Assert.AreEqual(310m, fill.FillPrice); + } + + [Test] + public void FilledStatusWithoutNumbersEmitsSubmitOnly() + { + // the thinnest read: only the id and a status. The service emits what those two can prove + // and never closes an order without its fill numbers. + AddOrder(100m, "42"); + + _service.ProcessOrderState(State("42", OrderStatus.Filled)); + + Assert.AreEqual(1, _orderEvents.Count); + Assert.AreEqual(OrderStatus.Submitted, _orderEvents[0].Status); + } + + [Test] + public void SellOrderFillsAreSignedByDirection() + { + AddOrder(-100m, "42", OrderStatus.Submitted); + + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 40m, price: 310m)); + + var fill = _orderEvents.Single(orderEvent => orderEvent.FillQuantity != 0); + Assert.AreEqual(-40m, fill.FillQuantity); + } + + [Test] + public void SharedIdComboSplitsFillsByGroupQuantity() + { + // one brokerage id, two Lean leg orders: 5 strangles, put leg ratio 1 (quantity 5), + // call leg ratio 2 (quantity 10) + var groupOrderManager = new GroupOrderManager(1, legCount: 2, quantity: 5m); + var time = new DateTime(2026, 8, 12, 14, 0, 0, DateTimeKind.Utc); + var putLeg = new ComboMarketOrder(Symbols.SPY_P_192_Feb19_2016, 5m, time, groupOrderManager); + var callLeg = new ComboMarketOrder(Symbols.SPY_C_192_Feb19_2016, 10m, time, groupOrderManager); + foreach (var leg in new[] { putLeg, callLeg }) + { + leg.Status = OrderStatus.Submitted; + leg.BrokerId.Add("900"); + _orderProvider.Add(leg); + } + + // the broker reports 2 of 5 strangles filled, one number for the whole combo + _service.ProcessOrderState(State("900", OrderStatus.PartiallyFilled, filled: 2m, price: 3.5m)); + + Assert.AreEqual(2, _orderEvents.Count); + Assert.AreEqual(2m, _orderEvents.Single(orderEvent => orderEvent.Symbol == putLeg.Symbol).FillQuantity); + Assert.AreEqual(4m, _orderEvents.Single(orderEvent => orderEvent.Symbol == callLeg.Symbol).FillQuantity); + Assert.IsTrue(_orderEvents.All(orderEvent => orderEvent.Status == OrderStatus.PartiallyFilled)); + + // the rest fills: 3 more strangles complete the order + _orderEvents.Clear(); + _service.ProcessOrderState(State("900", OrderStatus.Filled, filled: 5m, price: 3.6m)); + + Assert.AreEqual(2, _orderEvents.Count); + Assert.AreEqual(3m, _orderEvents.Single(orderEvent => orderEvent.Symbol == putLeg.Symbol).FillQuantity); + Assert.AreEqual(6m, _orderEvents.Single(orderEvent => orderEvent.Symbol == callLeg.Symbol).FillQuantity); + Assert.IsTrue(_orderEvents.All(orderEvent => orderEvent.Status == OrderStatus.Filled)); + } + + [Test] + public void ComboWithOneLegClosedInLeanSplitsOnlyToTheOpenLeg() + { + var groupOrderManager = new GroupOrderManager(1, legCount: 2, quantity: 5m); + var time = new DateTime(2026, 8, 12, 14, 0, 0, DateTimeKind.Utc); + var putLeg = new ComboMarketOrder(Symbols.SPY_P_192_Feb19_2016, 5m, time, groupOrderManager); + var callLeg = new ComboMarketOrder(Symbols.SPY_C_192_Feb19_2016, 10m, time, groupOrderManager); + foreach (var leg in new[] { putLeg, callLeg }) + { + leg.BrokerId.Add("900"); + _orderProvider.Add(leg); + } + // Lean already applied the put leg's fill events, the call leg is still working + putLeg.Status = OrderStatus.Filled; + callLeg.Status = OrderStatus.Submitted; + + _service.ProcessOrderState(State("900", OrderStatus.PartiallyFilled, filled: 3m, price: 3.5m)); + + var fill = _orderEvents.Single(); + Assert.AreEqual(callLeg.Symbol, fill.Symbol); + Assert.AreEqual(6m, fill.FillQuantity); + + // the same state again reports nothing - the entry survived, the totals moved + _orderEvents.Clear(); + _service.ProcessOrderState(State("900", OrderStatus.PartiallyFilled, filled: 3m, price: 3.5m)); + Assert.IsEmpty(_orderEvents); + } + + [Test] + public void MultipleOrdersOnOneIdWithoutAGroupQuantityEmitNoFillAndDoNotThrow() + { + // two Lean orders behind one brokerage id but no group manager: the service cannot split the + // fill, so it skips it instead of guessing or dividing by zero + AddOrder(100m, "900", OrderStatus.Submitted); + AddOrder(200m, "900", OrderStatus.Submitted); + + Assert.DoesNotThrow(() => _service.ProcessOrderState(State("900", OrderStatus.PartiallyFilled, filled: 10m, price: 5m))); + Assert.IsEmpty(_orderEvents); + } + + [Test] + public void TerminalSeedIsNotRepeated() + { + AddOrder(100m, "42", OrderStatus.Submitted); + + // another path already reported the cancel; the watch moves the state, e.g. across a replace + _service.Watch("42", State("42", OrderStatus.Canceled, message: "canceled by the broker")); + + _service.ProcessOrderState(State("42", OrderStatus.Canceled, message: "canceled by the broker")); + Assert.IsEmpty(_orderEvents); + } + + [Test] + public void TerminalIsEmittedOnceWithMessageAndAfterFills() + { + AddOrder(100m, "42", OrderStatus.Submitted); + + // one state carries a last fill and the cancel: the fill must go out first + _service.ProcessOrderState(State("42", OrderStatus.Canceled, filled: 30m, price: 310m, message: "canceled by the broker")); + + Assert.AreEqual(2, _orderEvents.Count); + Assert.AreEqual(30m, _orderEvents[0].FillQuantity); + Assert.AreEqual(OrderStatus.Canceled, _orderEvents[1].Status); + Assert.AreEqual("canceled by the broker", _orderEvents[1].Message); + + // the same state on the next sweep reports nothing: the fill is at the reported total and + // the end already went out + _service.ProcessOrderState(State("42", OrderStatus.Canceled, filled: 30m, price: 310m, message: "canceled by the broker")); + Assert.AreEqual(2, _orderEvents.Count); + } + + [Test] + public void StateIsDroppedOnlyOnceLeanClosedTheOrder() + { + var order = AddOrder(100m, "42", OrderStatus.Submitted); + + _service.ProcessOrderState(State("42", OrderStatus.Canceled, filled: 30m, price: 310m)); + Assert.IsTrue(_service.TryGetLastOrderState("42", out _)); + + // Lean applied the cancel: the next compare sees the order closed and drops the state + order.Status = OrderStatus.Canceled; + _service.ProcessOrderState(State("42", OrderStatus.Canceled, filled: 30m, price: 310m)); + Assert.IsFalse(_service.TryGetLastOrderState("42", out _)); + } + + [Test] + public void UnknownBrokerageIdWritesNothing() + { + _service.ProcessOrderState(State("77", OrderStatus.Filled, filled: 100m, price: 310m)); + + Assert.IsEmpty(_orderEvents); + Assert.IsFalse(_service.TryGetLastOrderState("77", out _)); + } + + [Test] + public void RejectDoesNotEmitSubmit() + { + AddOrder(100m, "42"); + + _service.ProcessOrderState(State("42", OrderStatus.Invalid, message: "rejected")); + + Assert.AreEqual(1, _orderEvents.Count); + Assert.AreEqual(OrderStatus.Invalid, _orderEvents[0].Status); + Assert.AreEqual("rejected", _orderEvents[0].Message); + } + + [Test] + public void SeededWatchDoesNotRepeatWhatWasAlreadyReported() + { + AddOrder(200m, "42", OrderStatus.PartiallyFilled); + + // another path already reported the submit and 100 shares + _service.Watch("42", State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + Assert.IsEmpty(_orderEvents); + + // only the part above the seed is new + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 150m, price: 311m)); + var fill = _orderEvents.Single(); + Assert.AreEqual(50m, fill.FillQuantity); + Assert.AreEqual(311m, fill.FillPrice); + } + + [Test] + public void StreamWriteThroughUpdateOrderStateIsNotRepeatedByThePoll() + { + AddOrder(200m, "42", OrderStatus.PartiallyFilled); + + // the stream reported a fill and wrote it into the shared registry + _service.UpdateOrderState("42", State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + + // the next sweep sees the same broker numbers and repeats nothing + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + Assert.IsEmpty(_orderEvents); + + Assert.IsTrue(_service.TryGetLastOrderState("42", out var lastSeen)); + Assert.AreEqual(100m, lastSeen.FilledQuantity); + } + + [Test] + public void StreamReportedTerminalIsNotRepeatedByThePoll() + { + AddOrder(100m, "42", OrderStatus.Submitted); + + // the stream reported the cancel and wrote it into the shared registry, but Lean has not + // applied it yet when the next sweep lands + _service.UpdateOrderState("42", State("42", OrderStatus.Canceled, message: "canceled by the broker")); + + _service.ProcessOrderState(State("42", OrderStatus.Canceled, message: "canceled by the broker")); + Assert.IsEmpty(_orderEvents); + } + + [Test] + public void WatchNeverOverwritesExistingState() + { + AddOrder(200m, "42", OrderStatus.PartiallyFilled); + _service.UpdateOrderState("42", State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + + // a later plain watch keeps the recorded state + _service.Watch("42"); + + Assert.IsTrue(_service.TryGetLastOrderState("42", out var lastSeen)); + Assert.AreEqual(100m, lastSeen.FilledQuantity); + + // and a later seeded watch ignores its seed too: the recorded state wins, so the next poll + // cannot re-report fills the stream already delivered + _service.Watch("42", State("42", OrderStatus.Submitted, filled: 0m)); + + Assert.IsTrue(_service.TryGetLastOrderState("42", out lastSeen)); + Assert.AreEqual(100m, lastSeen.FilledQuantity); + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + Assert.IsEmpty(_orderEvents); + } + + [Test] + public void WatchTimeoutFiresOnceAndUnwatchesTheId() + { + using var notAcknowledged = new ManualResetEventSlim(false); + var raised = new List(); + using var service = new PerOrderIdPollingService( + _ => null, // the broker never knows the id + route: null, + _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(25), + watchTimeout: TimeSpan.FromMilliseconds(75)); + service.OrderNotAcknowledged += (_, eventArgs) => + { + lock (raised) + { + raised.Add(eventArgs); + } + notAcknowledged.Set(); + }; + + service.Watch("77"); + service.Start(); + + Assert.IsTrue(notAcknowledged.Wait(TimeSpan.FromSeconds(5)), "the watch timeout never fired"); + + // let a few more sweeps run: the id was unwatched with the event, so it fires exactly once + Thread.Sleep(200); + service.Stop(); + + lock (raised) + { + Assert.AreEqual(1, raised.Count); + Assert.AreEqual("77", raised[0].BrokerageOrderId); + Assert.GreaterOrEqual(raised[0].WatchedFor, TimeSpan.FromMilliseconds(75)); + } + Assert.IsFalse(service.TryGetLastOrderState("77", out _)); + } + + [Test] + public void AcknowledgedWatchNeverTimesOut() + { + var fired = 0; + using var service = new PerOrderIdPollingService( + _ => null, + route: null, + _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(25), + watchTimeout: TimeSpan.FromMilliseconds(75)); + service.OrderNotAcknowledged += (_, _) => Interlocked.Increment(ref fired); + + // the stream acknowledged the order right after it was watched + service.Watch("77"); + service.UpdateOrderState("77", State("77", OrderStatus.Submitted)); + service.Start(); + + Thread.Sleep(300); + service.Stop(); + + Assert.AreEqual(0, fired); + } + + [Test] + public void RepeatedReadFailuresRaiseOneWarningPerOutage() + { + var fail = true; + var warnings = new List(); + using var warned = new AutoResetEvent(false); + using var service = new AllOrdersPollingService( + () => fail ? throw new Exception("read failed") : Array.Empty(), + route: null, + _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(25)); + service.Message += (_, message) => + { + lock (warnings) + { + warnings.Add(message); + } + warned.Set(); + }; + + service.Start(); + Assert.IsTrue(warned.WaitOne(TimeSpan.FromSeconds(5)), "the failure warning never fired"); + + // more failing sweeps do not warn again inside the same outage + Thread.Sleep(200); + lock (warnings) + { + Assert.AreEqual(1, warnings.Count); + Assert.AreEqual(BrokerageMessageType.Warning, warnings[0].Type); + Assert.AreEqual("OrderPollingFailed", warnings[0].Code); + } + + // one successful read arms the warning again, so the next outage warns once more + fail = false; + Thread.Sleep(200); + fail = true; + Assert.IsTrue(warned.WaitOne(TimeSpan.FromSeconds(5)), "the second outage never warned"); + + service.Stop(); + lock (warnings) + { + Assert.AreEqual(2, warnings.Count); + } + } + + [Test] + public void PerOrderIdSweepReadsOnlyWatchedIdsAndRoutesTheStates() + { + AddOrder(100m, "42"); + var readIds = new List(); + using var routed = new ManualResetEventSlim(false); + // the route loops each state back into the diff, standing in for the message handler + PerOrderIdPollingService service = null; + using var serviceReference = service = new PerOrderIdPollingService( + brokerageId => + { + lock (readIds) + { + readIds.Add(brokerageId); + } + return State(brokerageId, OrderStatus.Submitted); + }, + route: orderState => service.ProcessOrderState(orderState), + _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(25)); + var events = new List(); + service.OrderEvents += (_, orderEvents) => + { + lock (events) + { + events.AddRange(orderEvents); + } + routed.Set(); + }; + + // nothing watched: sweeps read nothing + service.Start(); + Thread.Sleep(100); + lock (readIds) + { + Assert.IsEmpty(readIds); + } + + service.Watch("42"); + Assert.IsTrue(routed.Wait(TimeSpan.FromSeconds(5)), "the watched id never produced an event"); + service.Stop(); + + lock (readIds) + { + Assert.IsTrue(readIds.All(id => id == "42")); + } + lock (events) + { + Assert.AreEqual(OrderStatus.Submitted, events[0].Status); + } + } + + [Test] + public void StartAndStopAreIdempotentAndDisposeIsFinal() + { + Assert.IsFalse(_service.IsPolling); + + _service.Start(); + _service.Start(); + Assert.IsTrue(_service.IsPolling); + + _service.Stop(); + _service.Stop(); + Assert.IsFalse(_service.IsPolling); + + _service.Start(); + Assert.IsTrue(_service.IsPolling); + _service.Stop(); + + _service.Dispose(); + _service.Start(); + Assert.IsFalse(_service.IsPolling); + } + } +} From 4a6382778424f144d62c8917efd83c73182b4d6b Mon Sep 17 00:00:00 2001 From: Romazes Date: Thu, 13 Aug 2026 17:52:32 +0300 Subject: [PATCH 04/25] feature: SeedAndStart, the stream-to-polling handover in one call - guard, buffered-message processing, one seed per open Lean order, Start - in the only safe order - both callbacks optional and nullable --- .../Services/BrokerageOrderPollingService.cs | 37 +++++++++++ .../0001-brokerage-order-polling-service.md | 41 +++++++----- .../BrokerageOrderPollingServiceTests.cs | 62 +++++++++++++++++++ 3 files changed, 124 insertions(+), 16 deletions(-) diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/BrokerageOrderPollingService.cs index cf1d37c5a707..22513d7c20dd 100644 --- a/Brokerages/Services/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/BrokerageOrderPollingService.cs @@ -436,6 +436,43 @@ public void ProcessOrderState(BrokerOrderState orderState) } } + /// + /// The whole handover from a stream to polling, in the only safe order: process what the stream + /// already delivered, seed the registry with one per + /// open Lean order, then . Does nothing while polling already runs. + /// + /// + /// The stream reported 100 of 233 shares, the seed carries 100, so the first sweep reports only the other 133. + /// + /// Processes the stream messages still waiting in the plugin's + /// message handler, so the seeds count every fill the stream delivered. Null skips the step. + /// Builds the seed for one open Lean order: the brokerage id, the order's status + /// and the cumulative filled quantity already reported. A null return skips the order, and a null + /// callback skips seeding. + public void SeedAndStart(Action drainBufferedMessages = null, Func seed = null) + { + if (IsPolling) + { + return; + } + + drainBufferedMessages?.Invoke(); + + if (seed != null) + { + foreach (var openLeanOrder in _orderProvider?.GetOpenOrders() ?? []) + { + var lastSeen = seed(openLeanOrder); + if (lastSeen != null && !string.IsNullOrEmpty(lastSeen.BrokerageOrderId)) + { + Watch(lastSeen.BrokerageOrderId, lastSeen); + } + } + } + + Start(); + } + /// /// Starts the background polling task. Idempotent while running; after a later /// call resumes polling. Does nothing once the service has been disposed. diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 835774f7c781..63722d2f2c86 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -431,6 +431,11 @@ public abstract class BrokerageOrderPollingService : IDisposable /// public void ProcessOrderState(BrokerOrderState orderState); + /// The whole handover from a stream to polling, in the only safe order: process what the + /// stream already delivered, seed one watch per open Lean order, then Start. Both callbacks are + /// optional. See "Seed before Start". + public void SeedAndStart(Action drainBufferedMessages = null, Func seed = null); + public void Start(); public void Stop(); public void Dispose(); @@ -697,19 +702,24 @@ in that block, so the next sweep does not repeat the submit. ### Seed before Start Polling never starts first: the stream reported orders before it, and the registry must know what was already -reported before the first sweep runs. So every `Start` that follows stream time begins with a handover: - -1. Drain the message buffer with an empty `WithLockedStream` block, so every fill the stream already delivered is - counted. Nothing slips in after the drain, because the switch runs on the stream's own thread — the only thread - that delivers stream messages. -2. Seed the registry with one `Watch(id, lastSeen)` per open Lean order: the status comes from the Lean order, the - cumulative filled quantity from the plugin's own bookkeeping. Orders the stream already closed need no seed — - the diff skips every order Lean has closed. +reported before the first sweep runs. So every `Start` that follows stream time begins with a handover, and the +service owns its order — `SeedAndStart(drainBufferedMessages, seed)` does nothing while polling already runs, and +otherwise runs three steps: + +1. `drainBufferedMessages` — the plugin passes `() => _messageHandler.WithLockedStream(() => { })`, so every fill + the stream already delivered is counted. Nothing slips in after it, because the switch runs on the stream's own + thread — the only thread that delivers stream messages. +2. One `seed(openLeanOrder)` call per open Lean order, each becoming a `Watch(id, lastSeen)`: the plugin returns + the brokerage id, the order's status and the cumulative filled quantity from its own bookkeeping. Orders the + stream already closed need no seed — the diff skips every order Lean has closed, and a null return skips the + order. 3. `Start`. -The first sweep then continues from what the stream reported instead of repeating it. A fallback-mode plugin does -this once, because its stream never comes back — Schwab's `SeedRegistryFromOpenOrders` is the working example. A -gap-mode plugin repeats it on every drop: seed, `Start`, and `Stop` when the stream returns. +Both callbacks are optional: a plugin with no message handler passes no drain, a plugin whose stream reported +nothing passes no seed. The first sweep then continues from what the stream reported instead of repeating it. A +fallback-mode plugin makes this one call, because its stream never comes back — Schwab's `ToSeedState` is the +working example of the seed callback. A gap-mode plugin repeats the call on every drop, and `Stop`s when the +stream returns. The seed source already exists in most streaming plugins, because they keep the same bookkeeping Schwab does — the cumulative quantity already reported, per Lean order: Webull's `_orderIdToPreviousCumulativeQuantity` @@ -731,7 +741,7 @@ does. `Dispose` is the one-way door. Three rules for a plugin that uses this: - **Seed before every `Start`.** While the socket was up the stream was reporting and the registry was not - listening. The seed hands over what was reported — see "Seed before Start". + listening. `SeedAndStart` hands over what was reported, in the right order — see "Seed before Start". - **Sweep once after the stream returns, before stopping.** The socket coming back does not replay what it missed, and no plugin re-reads orders on reconnect today. One last sweep closes the gap. - **Coming back is not always allowed.** Schwab must stay on polling for the rest of the run, because reconnecting @@ -830,12 +840,11 @@ in the background. The `_noSubmissionOrderTypes` guess (`MarketOnOpen`, `ComboLe becomes a real check: the broker either lists the order, and `Submitted` is true, or it does not, and the brokerage is asked instead of Lean inventing an event. -The same three lines cover a dropped socket, in any streaming plugin: +The same two lines cover a dropped socket, in any streaming plugin: ```csharp -// where the brokerage already handles the connection going down -SeedRegistryFromOpenOrders(); // one Watch(id, lastSeen) per open Lean order, see "Seed before Start" -_orderPollingService.Start(); +// where the brokerage already handles the connection going down; see "Seed before Start" +_orderPollingService.SeedAndStart(() => _messageHandler.WithLockedStream(() => { }), ToSeedState); // on reconnect: one last sweep for the gap, then hand the job back to the stream _orderPollingService.Stop(); diff --git a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs index 575a8d44d879..cc5fb2e3dbba 100644 --- a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs @@ -577,5 +577,67 @@ public void StartAndStopAreIdempotentAndDisposeIsFinal() _service.Start(); Assert.IsFalse(_service.IsPolling); } + + [Test] + public void SeedAndStartDrainsThenSeedsThenStarts() + { + AddOrder(233m, "42", OrderStatus.Submitted); + var steps = new List(); + + _service.SeedAndStart( + drainBufferedMessages: () => steps.Add("drain"), + seed: order => + { + steps.Add($"seed:{order.BrokerId[0]}"); + // the stream already reported 100 of the 233 shares + return State(order.BrokerId[0], order.Status, filled: 100m); + }); + + Assert.IsTrue(_service.IsPolling); + CollectionAssert.AreEqual(new[] { "drain", "seed:42" }, steps); + + // the first sweep continues from the seed: only the other 133 shares are reported + _service.ProcessOrderState(State("42", OrderStatus.Filled, filled: 233m, price: 310m)); + Assert.AreEqual(1, _orderEvents.Count); + Assert.AreEqual(OrderStatus.Filled, _orderEvents[0].Status); + Assert.AreEqual(133m, _orderEvents[0].FillQuantity); + } + + [Test] + public void SeedAndStartWithoutCallbacksJustStarts() + { + _service.SeedAndStart(); + + Assert.IsTrue(_service.IsPolling); + Assert.IsEmpty(_orderEvents); + } + + [Test] + public void SeedAndStartWhilePollingDoesNothing() + { + AddOrder(100m, "42"); + _service.Start(); + + var callCount = 0; + _service.SeedAndStart(drainBufferedMessages: () => callCount++, seed: _ => { callCount++; return null; }); + + Assert.IsTrue(_service.IsPolling); + Assert.AreEqual(0, callCount); + } + + [Test] + public void SeedWithoutBrokerageIdIsSkipped() + { + AddOrder(100m, "42"); + AddOrder(100m, "43"); + + _service.SeedAndStart(seed: order => order.BrokerId[0] == "42" + ? null + : new BrokerOrderState { Status = OrderStatus.Submitted }); + + Assert.IsTrue(_service.IsPolling); + Assert.IsFalse(_service.TryGetLastOrderState("42", out _)); + Assert.IsFalse(_service.TryGetLastOrderState("43", out _)); + } } } From b8b75325531e4031765d784b52299f6031303e64 Mon Sep 17 00:00:00 2001 From: Romazes Date: Thu, 13 Aug 2026 21:12:07 +0300 Subject: [PATCH 05/25] refactor: polling service wires the message handler itself - add non-generic BrokerageConcurrentMessageHandler with per-type Register, reusing the generic handler by composition - restore BrokerageConcurrentMessageHandler to master and drop BrokerageMessageQueue - service constructor takes the handler, registers ProcessOrderState and routes the states; SeedAndStart drains the handler internally - update the ADR wiring, seed-before-start and risk sections, and note the CharlesSchwab pilot --- .../BrokerageConcurrentMessageHandler.cs | 279 +++++++++++++++--- Brokerages/BrokerageMessageQueue.cs | 249 ---------------- .../Services/AllOrdersPollingService.cs | 7 +- .../Services/BrokerageOrderPollingService.cs | 51 +++- .../Services/PerOrderIdPollingService.cs | 7 +- .../0001-brokerage-order-polling-service.md | 134 ++++----- ...oncurrentMessageHandlerMultiSourceTests.cs | 107 +++++++ .../Brokerages/BrokerageMessageQueueTests.cs | 151 ---------- .../BrokerageOrderPollingServiceTests.cs | 100 +++++-- 9 files changed, 533 insertions(+), 552 deletions(-) delete mode 100644 Brokerages/BrokerageMessageQueue.cs create mode 100644 Tests/Brokerages/BrokerageConcurrentMessageHandlerMultiSourceTests.cs delete mode 100644 Tests/Brokerages/BrokerageMessageQueueTests.cs diff --git a/Brokerages/BrokerageConcurrentMessageHandler.cs b/Brokerages/BrokerageConcurrentMessageHandler.cs index aa770f5ed1d5..6cc2f6514203 100644 --- a/Brokerages/BrokerageConcurrentMessageHandler.cs +++ b/Brokerages/BrokerageConcurrentMessageHandler.cs @@ -14,19 +14,24 @@ */ using System; +using System.Threading; +using QuantConnect.Logging; +using System.Collections.Generic; +using QuantConnect.Configuration; namespace QuantConnect.Brokerages { /// /// Brokerage helper class to lock message stream while executing an action, for example placing an order /// - /// A thin wrapper around : the lock, buffer and dispatch all - /// live there now, so a second source with its own message type can be handled through the exact same - /// lock this handler uses - see . public class BrokerageConcurrentMessageHandler : IDisposable where T : class { - private readonly BrokerageMessageQueue _queue; + private readonly Action _processMessages; + private readonly Queue _messageBuffer; + private readonly ILock _lock; + private readonly ManualResetEventSlim _messagesProcessedEvent; + private readonly int _maxMessageBufferSize; /// /// Creates a new instance @@ -44,71 +49,275 @@ public BrokerageConcurrentMessageHandler(Action processMessages) /// Whether to enable concurrent order submission public BrokerageConcurrentMessageHandler(Action processMessages, bool concurrencyEnabled) { - _queue = new BrokerageMessageQueue(concurrencyEnabled); - RegisterMessageType(processMessages); + _processMessages = processMessages; + _messageBuffer = new Queue(); + _lock = concurrencyEnabled ? new ReaderWriterLockWrapper() : new MonitorWrapper(); + _messagesProcessedEvent = new ManualResetEventSlim(false); + _maxMessageBufferSize = Config.GetInt("brokerage-concurrent-message-handler-buffer-size", 20); } /// - /// Registers another message type this handler also processes, alongside . - /// A source with its own message type - for example the order polling service - registers here once, - /// then calls like anything else, and its messages queue - /// behind the exact same lock as instead of getting a second, independent - /// lock that would not actually serialize against this one. + /// Disposes of the resources used by this instance /// - /// The action to call for each new message of this type - public void RegisterMessageType(Action processMessages) - where TMessage : class + public void Dispose() { - _queue.MessageReceived += message => + _lock.Dispose(); + _messagesProcessedEvent.Dispose(); + } + + /// + /// Will process or enqueue a message for later processing it + /// + /// The new message + public void HandleNewMessage(T message) + { + lock (_messageBuffer) { - if (message is TMessage typed) + if (_lock.TryEnterReadLockImmediately()) { - processMessages(typed); + try + { + ProcessMessages(message); + } + finally + { + _lock.ExitReadLock(); + } } - }; + else if (message != default) + { + // if someone has the lock just enqueue the new message they will process any remaining messages + // if by chance they are about to free the lock, no worries, we will always process first any remaining message first see 'ProcessMessages' + _messageBuffer.Enqueue(message); + } + } } /// - /// Disposes of the resources used by this instance + /// Lock the streaming processing while we're sending orders as sometimes they fill before the call returns. /// - public void Dispose() + public void WithLockedStream(Action code) { - _queue.Dispose(); + // Let's limit the amount of messages we can buffer, so we wait until + // consumers process a full queue of messages before we potentially add more + var queueIsFull = false; + lock (_messageBuffer) + { + queueIsFull = _messageBuffer.Count >= _maxMessageBufferSize; + } + if (queueIsFull) + { + _messagesProcessedEvent.Wait(); + _messagesProcessedEvent.Reset(); + } + + _lock.EnterWriteLock(); + try + { + code(); + } + finally + { + // once we finish our 'code' we will process any message that come through, + // to make sure no message get's left behind (race condition between us finishing 'ProcessMessages' + // and some message being enqueued to it, we just take a lock on the buffer + lock (_messageBuffer) + { + var lockedStreams = _lock.CurrentWriteCount; + + // we release the semaphore first so by the time we release '_messageBuffer' any new message is processed immediately and not enqueued + _lock.ExitWriteLock(); + // only process if no other threads will process them after us + if (lockedStreams == 1) + { + ProcessMessages(); + } + } + } } /// - /// Will process or enqueue a message for later processing it + /// Process any pending message and the provided one if any /// - /// The new message - public void HandleNewMessage(T message) + /// To be called owing the stream lock + private void ProcessMessages(T message = null) { - if (message != null) + try { - _queue.Enqueue(message); + if (message != null) + { + _messageBuffer.Enqueue(message); + } + + // double check there isn't any pending message + while (_messageBuffer.TryDequeue(out var e)) + { + try + { + _processMessages(e); + } + catch (Exception ex) + { + Log.Error(ex); + } + } + } + finally + { + _messagesProcessedEvent.Set(); } } + private interface ILock : IDisposable + { + int CurrentWriteCount { get; } + + void ExitReadLock(); + + bool TryEnterReadLockImmediately(); + + void EnterWriteLock(); + + void ExitWriteLock(); + } + /// - /// Same as , for any other type registered through - /// - the type is inferred from the argument, - /// so the call looks the same either way. + /// A simple reader/writer lock implementation that allows us to switch the meaning of read and write locks + /// so that it can be used for single reader and multiple writers scenario. + /// + /// We want to allow multiple producers so, for example, a brokerage can be placing multiple orders concurrently, + /// since the transaction handler can have multiple threads processing orders. + /// But, on the other side, we need to ensure that messages are processed only when no producers are writing + /// to the stream (hence only one reader). For example, a brokerage needs the to lock the stream and + /// only handle incoming order event messages after it releases the lock, but we now support multiple streams + /// (so multiple orders) so we wait for all the current producers to release the lock before processing any messages. /// - /// The new message - public void HandleNewMessage(TMessage message) - where TMessage : class + private class ReaderWriterLockWrapper : ILock { - if (message != null) + private readonly ReaderWriterLockSlim _lock; + + public int CurrentWriteCount => _lock.CurrentReadCount; + + public ReaderWriterLockWrapper() + { + _lock = new ReaderWriterLockSlim(LockRecursionPolicy.NoRecursion); + } + public void ExitReadLock() => _lock.ExitWriteLock(); + public bool TryEnterReadLockImmediately() => _lock.TryEnterWriteLock(0); + public void EnterWriteLock() => _lock.EnterReadLock(); + public void ExitWriteLock() => _lock.ExitReadLock(); + + public void Dispose() { - _queue.Enqueue(message); + _lock.Dispose(); } } + private class MonitorWrapper : ILock + { + private readonly object _lockObject; + + private long _currentWriteCount; + + public int CurrentWriteCount => (int)Interlocked.Read(ref _currentWriteCount); + + public MonitorWrapper() + { + _lockObject = new object(); + } + + public void ExitReadLock() => Monitor.Exit(_lockObject); + + public bool TryEnterReadLockImmediately() => Monitor.TryEnter(_lockObject); + + public void EnterWriteLock() + { + Monitor.Enter(_lockObject); + Interlocked.Exchange(ref _currentWriteCount, 1); + } + + public void ExitWriteLock() + { + Interlocked.Exchange(ref _currentWriteCount, 0); + Monitor.Exit(_lockObject); + } + + public void Dispose() + { + } + } + } + + /// + /// The multi-source version of : one lock and one buffer, + /// any number of listeners. Each source registers its own message type through + /// and enqueues through the same - for example a websocket stream and the + /// order polling service - so no two listeners ever run at the same time, and none runs while an order + /// request holds . + /// + public class BrokerageConcurrentMessageHandler : IDisposable + { + /// + /// The single-type handler, with object as the message type, owns the lock, the buffer and the + /// dispatch, so both classes share one synchronization implementation. + /// + private readonly BrokerageConcurrentMessageHandler _handler; + + /// + /// One filter per registered listener. A field-like event, so registering a new listener while + /// messages flow is safe. + /// + private event Action ProcessMessage; + + /// + /// Creates a new instance + /// + /// Whether to enable concurrent order submission + public BrokerageConcurrentMessageHandler(bool concurrencyEnabled = false) + { + _handler = new BrokerageConcurrentMessageHandler(message => ProcessMessage?.Invoke(message), concurrencyEnabled); + } + + /// + /// Registers a listener for one message type. A dequeued message runs every listener whose type + /// matches, in registration order; a message no listener matches is dropped. + /// + /// The action to call for each new message of this type + public void Register(Action processMessages) + where TMessage : class + { + ProcessMessage += message => + { + if (message is TMessage typedMessage) + { + processMessages(typedMessage); + } + }; + } + + /// + /// Will process or enqueue a message for later processing it + /// + /// The new message + public void HandleNewMessage(object message) + { + _handler.HandleNewMessage(message); + } + /// /// Lock the streaming processing while we're sending orders as sometimes they fill before the call returns. /// public void WithLockedStream(Action code) { - _queue.WithLockedStream(code); + _handler.WithLockedStream(code); + } + + /// + /// Disposes of the resources used by this instance + /// + public void Dispose() + { + _handler.Dispose(); } } } diff --git a/Brokerages/BrokerageMessageQueue.cs b/Brokerages/BrokerageMessageQueue.cs deleted file mode 100644 index bfb487715ebe..000000000000 --- a/Brokerages/BrokerageMessageQueue.cs +++ /dev/null @@ -1,249 +0,0 @@ -/* - * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. - * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. -*/ - -using System; -using System.Threading; -using QuantConnect.Logging; -using System.Collections.Generic; -using QuantConnect.Configuration; - -namespace QuantConnect.Brokerages -{ - /// - /// Owns the lock and buffer that used to keep to itself. - /// A message of any type can be enqueued here; raises it once it is this - /// message's turn, and each subscriber picks out the type it cares about. This is what lets more than one - /// source share a single lock: a brokerage's own stream messages and, separately, the order polling - /// service's snapshots, so a stream message and a polled order update can never run at the same time, and - /// neither can run while an order request holds . - /// - public class BrokerageMessageQueue : IDisposable - { - private readonly Queue _messageBuffer; - private readonly ILock _lock; - private readonly ManualResetEventSlim _messagesProcessedEvent; - private readonly int _maxMessageBufferSize; - - /// - /// Raised for every dequeued message, in arrival order, one at a time. A brokerage's stream handler - /// and the order polling service each subscribe and filter for their own message type, e.g. - /// MessageReceived += message => { if (message is OrderResponse r) OnPolledOrder(r); }; - /// - public event Action MessageReceived; - - /// - /// Creates a new instance - /// - /// Whether to enable concurrent order submission - public BrokerageMessageQueue(bool concurrencyEnabled = false) - { - _messageBuffer = new Queue(); - _lock = concurrencyEnabled ? new ReaderWriterLockWrapper() : new MonitorWrapper(); - _messagesProcessedEvent = new ManualResetEventSlim(false); - _maxMessageBufferSize = Config.GetInt("brokerage-concurrent-message-handler-buffer-size", 20); - } - - /// - /// Disposes of the resources used by this instance - /// - public void Dispose() - { - _lock.Dispose(); - _messagesProcessedEvent.Dispose(); - } - - /// - /// Will process or enqueue a message for later processing it - /// - /// The new message - public void Enqueue(object message) - { - lock (_messageBuffer) - { - if (_lock.TryEnterReadLockImmediately()) - { - try - { - ProcessMessages(message); - } - finally - { - _lock.ExitReadLock(); - } - } - else if (message != null) - { - // if someone has the lock just enqueue the new message they will process any remaining messages - // if by chance they are about to free the lock, no worries, we will always process first any remaining message first see 'ProcessMessages' - _messageBuffer.Enqueue(message); - } - } - } - - /// - /// Lock the streaming processing while we're sending orders as sometimes they fill before the call returns. - /// - public void WithLockedStream(Action code) - { - // Let's limit the amount of messages we can buffer, so we wait until - // consumers process a full queue of messages before we potentially add more - var queueIsFull = false; - lock (_messageBuffer) - { - queueIsFull = _messageBuffer.Count >= _maxMessageBufferSize; - } - if (queueIsFull) - { - _messagesProcessedEvent.Wait(); - _messagesProcessedEvent.Reset(); - } - - _lock.EnterWriteLock(); - try - { - code(); - } - finally - { - // once we finish our 'code' we will process any message that come through, - // to make sure no message get's left behind (race condition between us finishing 'ProcessMessages' - // and some message being enqueued to it, we just take a lock on the buffer - lock (_messageBuffer) - { - var lockedStreams = _lock.CurrentWriteCount; - - // we release the semaphore first so by the time we release '_messageBuffer' any new message is processed immediately and not enqueued - _lock.ExitWriteLock(); - // only process if no other threads will process them after us - if (lockedStreams == 1) - { - ProcessMessages(); - } - } - } - } - - /// - /// Process any pending message and the provided one if any - /// - /// To be called owing the stream lock - private void ProcessMessages(object message = null) - { - try - { - if (message != null) - { - _messageBuffer.Enqueue(message); - } - - // double check there isn't any pending message - while (_messageBuffer.TryDequeue(out var e)) - { - try - { - MessageReceived?.Invoke(e); - } - catch (Exception ex) - { - Log.Error(ex); - } - } - } - finally - { - _messagesProcessedEvent.Set(); - } - } - - private interface ILock : IDisposable - { - int CurrentWriteCount { get; } - - void ExitReadLock(); - - bool TryEnterReadLockImmediately(); - - void EnterWriteLock(); - - void ExitWriteLock(); - } - - /// - /// A simple reader/writer lock implementation that allows us to switch the meaning of read and write locks - /// so that it can be used for single reader and multiple writers scenario. - /// - /// We want to allow multiple producers so, for example, a brokerage can be placing multiple orders concurrently, - /// since the transaction handler can have multiple threads processing orders. - /// But, on the other side, we need to ensure that messages are processed only when no producers are writing - /// to the stream (hence only one reader). For example, a brokerage needs the to lock the stream and - /// only handle incoming order event messages after it releases the lock, but we now support multiple streams - /// (so multiple orders) so we wait for all the current producers to release the lock before processing any messages. - /// - private class ReaderWriterLockWrapper : ILock - { - private readonly ReaderWriterLockSlim _lock; - - public int CurrentWriteCount => _lock.CurrentReadCount; - - public ReaderWriterLockWrapper() - { - _lock = new ReaderWriterLockSlim(LockRecursionPolicy.NoRecursion); - } - public void ExitReadLock() => _lock.ExitWriteLock(); - public bool TryEnterReadLockImmediately() => _lock.TryEnterWriteLock(0); - public void EnterWriteLock() => _lock.EnterReadLock(); - public void ExitWriteLock() => _lock.ExitReadLock(); - - public void Dispose() - { - _lock.Dispose(); - } - } - - private class MonitorWrapper : ILock - { - private readonly object _lockObject; - - private long _currentWriteCount; - - public int CurrentWriteCount => (int)Interlocked.Read(ref _currentWriteCount); - - public MonitorWrapper() - { - _lockObject = new object(); - } - - public void ExitReadLock() => Monitor.Exit(_lockObject); - - public bool TryEnterReadLockImmediately() => Monitor.TryEnter(_lockObject); - - public void EnterWriteLock() - { - Monitor.Enter(_lockObject); - Interlocked.Exchange(ref _currentWriteCount, 1); - } - - public void ExitWriteLock() - { - Interlocked.Exchange(ref _currentWriteCount, 0); - Monitor.Exit(_lockObject); - } - - public void Dispose() - { - } - } - } -} diff --git a/Brokerages/Services/AllOrdersPollingService.cs b/Brokerages/Services/AllOrdersPollingService.cs index aea112ed20e2..6c28a7429608 100644 --- a/Brokerages/Services/AllOrdersPollingService.cs +++ b/Brokerages/Services/AllOrdersPollingService.cs @@ -34,15 +34,16 @@ public class AllOrdersPollingService : BrokerageOrderPollingService /// Creates a new . /// /// Reads every order the broker lists, one state per brokerage order id. - /// Where each state a sweep returns goes, normally the brokerage's message handler. + /// The brokerage's message handler; the service registers itself and + /// enqueues every polled state through it. Null processes each state directly. /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is raised. Null falls back to one minute. - public AllOrdersPollingService(Func> readAllOrders, Action route, + public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) - : base(route, orderProvider, pollInterval, watchTimeout) + : base(messageHandler, orderProvider, pollInterval, watchTimeout) { _readAllOrders = readAllOrders; } diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/BrokerageOrderPollingService.cs index 22513d7c20dd..77870a2caa22 100644 --- a/Brokerages/Services/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/BrokerageOrderPollingService.cs @@ -53,9 +53,15 @@ public abstract class BrokerageOrderPollingService : IDisposable private readonly Dictionary _orderStates = new(); /// - /// Where each state a sweep returns goes, normally the brokerage's message handler. The handler - /// dispatches it back into , so polled states queue behind an order - /// request that holds the stream lock. + /// The brokerage's message handler, when it has one. The constructor wires it both ways: polled + /// states enqueue here, and is registered as their listener, so + /// polled states queue behind an order request that holds the stream lock. + /// + private readonly BrokerageConcurrentMessageHandler _messageHandler; + + /// + /// Where each state a sweep returns goes: the message handler, or without one straight into + /// . /// private readonly Action _route; @@ -123,19 +129,32 @@ public abstract class BrokerageOrderPollingService : IDisposable public TimeSpan WatchTimeout { get; } /// - /// Initializes what both modes share: the route, the order provider, and the two time settings - /// with their defaults. + /// Initializes what both modes share: the message handler wiring, the order provider, and the two + /// time settings with their defaults. /// - /// Where each state a sweep returns goes, normally the brokerage's message handler. + /// The brokerage's message handler. The service wires it both ways + /// itself: it registers and enqueues every polled state, so one + /// handler serializes polled states with everything else the brokerage processes. Null routes each + /// state straight into - only the poll loop calls it then, so the + /// calls are still one at a time. /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is raised. Null falls back to one minute. - protected BrokerageOrderPollingService(Action route, IOrderProvider orderProvider, + protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) { - _route = route; + if (messageHandler != null) + { + _messageHandler = messageHandler; + _route = messageHandler.HandleNewMessage; + messageHandler.Register(ProcessOrderState); + } + else + { + _route = ProcessOrderState; + } _orderProvider = orderProvider; PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3000)); WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); @@ -283,10 +302,10 @@ public bool TryGetLastOrderState(string brokerageId, out BrokerOrderState lastSe /// /// Compares a state with the last one seen for the same order and raises - /// with what is new: the submit first, then fills, then a close. Register it on the message handler - /// for , so polled orders queue behind an order request that holds the - /// stream lock. Not safe to run twice at the same time - the handler already runs it one call at a - /// time, and a caller without a handler must do the same. + /// with what is new: the submit first, then fills, then a close. The constructor registers it on the + /// message handler, so polled orders queue behind an order request that holds the stream lock. Not + /// safe to run twice at the same time - the handler runs it one call at a time, and without a + /// handler only the poll loop calls it. /// /// The state a sweep read from the broker. public void ProcessOrderState(BrokerOrderState orderState) @@ -444,19 +463,19 @@ public void ProcessOrderState(BrokerOrderState orderState) /// /// The stream reported 100 of 233 shares, the seed carries 100, so the first sweep reports only the other 133. /// - /// Processes the stream messages still waiting in the plugin's - /// message handler, so the seeds count every fill the stream delivered. Null skips the step. /// Builds the seed for one open Lean order: the brokerage id, the order's status /// and the cumulative filled quantity already reported. A null return skips the order, and a null /// callback skips seeding. - public void SeedAndStart(Action drainBufferedMessages = null, Func seed = null) + public void SeedAndStart(Func seed = null) { if (IsPolling) { return; } - drainBufferedMessages?.Invoke(); + // an empty locked block waits for any order request in flight and processes the stream messages + // it buffered, so the seeds below count every fill the stream delivered + _messageHandler?.WithLockedStream(() => { }); if (seed != null) { diff --git a/Brokerages/Services/PerOrderIdPollingService.cs b/Brokerages/Services/PerOrderIdPollingService.cs index eea946e3809b..9489a29ef179 100644 --- a/Brokerages/Services/PerOrderIdPollingService.cs +++ b/Brokerages/Services/PerOrderIdPollingService.cs @@ -37,15 +37,16 @@ public class PerOrderIdPollingService : BrokerageOrderPollingService /// /// Reads the current state of one order by its brokerage id. A null /// return means the broker does not know the id. - /// Where each state a sweep returns goes, normally the brokerage's message handler. + /// The brokerage's message handler; the service registers itself and + /// enqueues every polled state through it. Null processes each state directly. /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is raised. Null falls back to one minute. - public PerOrderIdPollingService(Func readOrder, Action route, + public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) - : base(route, orderProvider, pollInterval, watchTimeout) + : base(messageHandler, orderProvider, pollInterval, watchTimeout) { _readOrder = readOrder; } diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 63722d2f2c86..5ba75e60fe14 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -4,6 +4,11 @@ Proposed - 2026-08-07 +Pilot implemented - 2026-08-13: CharlesSchwab is the first plugin on the service +(`Lean.Brokerages.CharlesSchwab`, draft PR #107). Its own `OrderUpdatePollingService` and diff are deleted; +what remains in the plugin is what this document says remains - the read with its sweep window, the +model-to-state mapping, and the seeded stream-to-polling handover through `SeedAndStart`. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -370,14 +375,16 @@ namespace QuantConnect.Brokerages.Services; /// public abstract class BrokerageOrderPollingService : IDisposable { - /// Initializes what both modes share: the route, the order provider, and the two time - /// settings with their defaults. Each snapshot the sweep returns is handed to - /// , normally the brokerage's message handler. - protected BrokerageOrderPollingService(Action route, IOrderProvider orderProvider, + /// Initializes what both modes share: the message handler wiring, the order provider, and + /// the two time settings with their defaults. The service wires the handler both ways itself: it + /// registers and enqueues every polled snapshot, so one handler + /// serializes polled snapshots with everything else the brokerage processes. A null handler routes + /// each snapshot straight into . + protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); /// One read of the broker, giving the states the sweep saw. The loop calls it every - /// poll interval, hands each state to the route, and counts a throw as one failed sweep. + /// poll interval, hands each state to the message handler, and counts a throw as one failed sweep. protected abstract IEnumerable Sweep(); /// A copy of the ids a sweep still has to read: everything tracked whose end was not @@ -425,16 +432,15 @@ public abstract class BrokerageOrderPollingService : IDisposable /// /// Compares a snapshot with the last state seen for the same order and raises - /// with what is new. Register it on the message handler for - /// , so polled orders queue behind an order request that holds - /// the stream lock. + /// with what is new. The constructor registers it on the message + /// handler, so polled orders queue behind an order request that holds the stream lock. /// public void ProcessOrderState(BrokerOrderState orderState); /// The whole handover from a stream to polling, in the only safe order: process what the - /// stream already delivered, seed one watch per open Lean order, then Start. Both callbacks are + /// stream already delivered, seed one watch per open Lean order, then Start. The seed callback is /// optional. See "Seed before Start". - public void SeedAndStart(Action drainBufferedMessages = null, Func seed = null); + public void SeedAndStart(Func seed = null); public void Start(); public void Stop(); @@ -449,7 +455,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// public class PerOrderIdPollingService : BrokerageOrderPollingService { - public PerOrderIdPollingService(Func readOrder, Action route, + public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); } @@ -458,7 +464,7 @@ public class PerOrderIdPollingService : BrokerageOrderPollingService /// public class AllOrdersPollingService : BrokerageOrderPollingService { - public AllOrdersPollingService(Func> readAllOrders, Action route, + public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); } ``` @@ -506,8 +512,8 @@ still reporting `Submitted` for the same order. If the fill goes out first, the back to open, and Lean then accepts a cancel or update on it that the broker rejects. Schwab hit exactly this and fixed it by pushing polled orders through the same message handler as stream messages, so they wait in the queue while an order request holds the stream lock (`WithLockedStream`, -`Brokerages/BrokerageMessageQueue.cs:98`). That is why `ProcessOrderState` is a separate public method -instead of something the loop calls itself: the brokerage puts its message handler between the read and the diff, +`Brokerages/BrokerageConcurrentMessageHandler.cs:99`). That is why `ProcessOrderState` is a separate public method +instead of something the loop calls itself: the message handler sits between the read and the diff, and the queue keeps the order right. The handler's message type also has to cover both the stream's model and the snapshot — the next section is how it does. @@ -525,45 +531,34 @@ plugin that owns both models, and it cannot be the shared answer — every plugi have to add the interface to its own wire models, and the core `BrokerOrderState` cannot implement a per-plugin interface. -So the handler is split instead, and the split ships with the service. The lock, the buffer and the drain loop -move unchanged into a new class, `BrokerageMessageQueue` (`Brokerages/BrokerageMessageQueue.cs`): the buffer holds -`object`, and every dequeued message is raised through a `MessageReceived` event, in arrival order. -`BrokerageConcurrentMessageHandler` stays as a thin wrapper over one queue, with the same public surface every -plugin compiles against today, plus one method: +So a second, non-generic `BrokerageConcurrentMessageHandler` ships with the service, in the same file +(`Brokerages/BrokerageConcurrentMessageHandler.cs`). It wraps a `BrokerageConcurrentMessageHandler` +inside, so the lock, the buffer and the drain loop are the exact same code, not a copy. Any number of listeners +register, one per message type, and every source enqueues through one `HandleNewMessage(object)`: ```csharp -// the stream type, exactly as today -_messageHandler = new BrokerageConcurrentMessageHandler(OnAccountContent, concurrencyEnabled); - -// one more line, and snapshots share the same lock — no marker interface, no second handler -_messageHandler.RegisterMessageType(_orderPollingService.ProcessOrderState); +_messageHandler = new BrokerageConcurrentMessageHandler(concurrencyEnabled); +_messageHandler.Register(OnAccountContent); // the stream's own type +// the polling service registers its own BrokerOrderState listener itself, in its constructor -// one method for both; the compiler picks the type from the argument +// one method for both sources _messageHandler.HandleNewMessage(accountContent); _messageHandler.HandleNewMessage(orderState); ``` -`RegisterMessageType` subscribes a filter on the queue's event: a dequeued message runs one `is` check per -registered type and lands in the action that matches. `HandleNewMessage` becomes generic, with the type inferred -from the argument, so every existing call in every plugin compiles unchanged. - -The stream's hot path pays nothing for this. The filter is created once at registration, never per message; the -buffer stored references before and stores references now (`T` was already constrained to `class`); the lock code -moved without an edit. Per message, the old direct delegate call becomes an event invoke plus one type check per -registered type — nanoseconds next to the lock the path already takes. The existing handler tests pass against -the split unchanged (`Tests/Brokerages/BrokerageConcurrentMessageHandlerTests.cs` — ordering, backpressure, single -drainer, exception recovery), and the shared lock across two types has its own -(`Tests/Brokerages/BrokerageMessageQueueTests.cs`). - -One rule of the split is absolute: **the generic `BrokerageConcurrentMessageHandler` stays.** Thirteen live -plugins hold a field of it today — CharlesSchwab, Public.com, Alpaca, Binance, TradeStation, Tastytrade, Webull, -ByBit, Eze, OANDA, IG, dYdX and TerminalLink — plus the Template scaffold. Removing or reshaping `` breaks all -of them at once. So the class keeps its name, both constructors, `HandleNewMessage(T message)`, -`WithLockedStream` and `Dispose`, and every plugin recompiles with zero edits. The riskiest call shape was -checked one repo at a time: Schwab, Public and Eze pass `_messageHandler.HandleNewMessage` as a delegate -(`CharlesSchwabBrokerage.OrderUpdatePolling.cs:65`, `PublicBrokerage.cs:182`, `EzeBrokerage.cs:259`), and all -three still compile; the one shape that could not — a bare `null` argument, which type inference cannot resolve — -appears in no repo. +`Register` subscribes a filter: a dequeued message runs one `is` check per registered type and lands in every +listener that matches, in registration order; a message no listener matches is dropped. A plugin that adopts the +service switches its handler field to the non-generic class and turns the constructor callback into one +`Register` call — the stream's hot path pays one type check per registered type, nanoseconds next to the lock the +path already takes. The shared lock across two types has its own tests +(`Tests/Brokerages/BrokerageConcurrentMessageHandlerMultiSourceTests.cs`). + +One rule is absolute: **the generic `BrokerageConcurrentMessageHandler` stays byte-identical to master.** +Thirteen live plugins hold a field of it today — CharlesSchwab, Public.com, Alpaca, Binance, TradeStation, +Tastytrade, Webull, ByBit, Eze, OANDA, IG, dYdX and TerminalLink — plus the Template scaffold. It is not edited, +not even additively: the non-generic class reuses it by composition, and its tests +(`Tests/Brokerages/BrokerageConcurrentMessageHandlerTests.cs`) stay untouched with it. Only a plugin that adopts +the polling service moves to the non-generic class, one plugin at a time. ### The diff @@ -703,20 +698,20 @@ in that block, so the next sweep does not repeat the submit. Polling never starts first: the stream reported orders before it, and the registry must know what was already reported before the first sweep runs. So every `Start` that follows stream time begins with a handover, and the -service owns its order — `SeedAndStart(drainBufferedMessages, seed)` does nothing while polling already runs, and -otherwise runs three steps: +service owns its order — `SeedAndStart(seed)` does nothing while polling already runs, and otherwise runs three +steps: -1. `drainBufferedMessages` — the plugin passes `() => _messageHandler.WithLockedStream(() => { })`, so every fill - the stream already delivered is counted. Nothing slips in after it, because the switch runs on the stream's own - thread — the only thread that delivers stream messages. +1. Drain the message handler — the service runs an empty `WithLockedStream` block on the handler it was built + with, so every fill the stream already delivered is counted. Nothing slips in after it, because the switch + runs on the stream's own thread — the only thread that delivers stream messages. 2. One `seed(openLeanOrder)` call per open Lean order, each becoming a `Watch(id, lastSeen)`: the plugin returns the brokerage id, the order's status and the cumulative filled quantity from its own bookkeeping. Orders the stream already closed need no seed — the diff skips every order Lean has closed, and a null return skips the order. 3. `Start`. -Both callbacks are optional: a plugin with no message handler passes no drain, a plugin whose stream reported -nothing passes no seed. The first sweep then continues from what the stream reported instead of repeating it. A +The seed callback is optional: a plugin whose stream reported nothing passes no seed, and without a message +handler there is nothing to drain. The first sweep then continues from what the stream reported instead of repeating it. A fallback-mode plugin makes this one call, because its stream never comes back — Schwab's `ToSeedState` is the working example of the seed callback. A gap-mode plugin repeats the call on every drop, and `Stop`s when the stream returns. @@ -789,24 +784,23 @@ WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); ``` `brokerage-order-poll-interval-ms` is one new generic config entry, so the interval can be tuned once for every -brokerage that uses the default. Core helpers already work this way: the message queue reads +brokerage that uses the default. Core helpers already work this way: the message handler reads `brokerage-concurrent-message-handler-buffer-size` in its constructor -(`Brokerages/BrokerageMessageQueue.cs:55`). The 3000 ms default is the value Schwab runs today. +(`Brokerages/BrokerageConcurrentMessageHandler.cs:56`). The 3000 ms default is the value Schwab runs today. ## Wiring, per plugin The general shape, for a streaming brokerage with a bulk endpoint — the class is the mode: ```csharp -// bulk broker: one request per sweep reads the whole account +// bulk broker: one request per sweep reads the whole account. The service wires itself onto the +// handler: it registers its diff as a listener and enqueues every snapshot, so the plugin never +// touches that relationship again. _orderPollingService = new AllOrdersPollingService( () => _apiClient.GetAllOrders().Select(ToOrderState), // read: model -> snapshot - _messageHandler.HandleNewMessage, // route: through the message handler + _messageHandler, _orderProvider, pollInterval: TimeSpan.FromSeconds(3), watchTimeout: TimeSpan.FromMinutes(1)); _orderPollingService.OrderEvents += (_, orderEvents) => OnOrderEvents(orderEvents); - -// the handler dequeues a snapshot and hands it to the diff -_messageHandler.RegisterMessageType(_orderPollingService.ProcessOrderState); ``` A per-id broker only changes the class and the read — this is Public.com's own constructor today, snapshot @@ -815,7 +809,7 @@ instead of its DTO: ```csharp _orderPollingService = new PerOrderIdPollingService( brokerageId => ToOrderState(_apiClient.GetOrderById(brokerageId)), - _messageHandler.HandleNewMessage, + _messageHandler, _orderProvider); // nothing passed: brokerage-order-poll-interval-ms decides, default 3000 ms ``` @@ -844,7 +838,7 @@ The same two lines cover a dropped socket, in any streaming plugin: ```csharp // where the brokerage already handles the connection going down; see "Seed before Start" -_orderPollingService.SeedAndStart(() => _messageHandler.WithLockedStream(() => { }), ToSeedState); +_orderPollingService.SeedAndStart(ToSeedState); // on reconnect: one last sweep for the gap, then hand the job back to the stream _orderPollingService.Stop(); @@ -862,7 +856,7 @@ inline `Task.Delay` block with `Watch` on the unknown ids. execution history to the two numbers (Schwab sums its execution legs and takes the newest leg's price). A plugin whose leg ids are derived rather than returned by the broker should verify they resolve and warn when they do not — the service skips silently. -- Routing: passing snapshots through its message handler and forwarding `OrderEvents` to `OnOrderEvents`. +- Routing: handing the service its message handler at construction and forwarding `OrderEvents` to `OnOrderEvents`. - Reporting without the stream: Schwab's submit and replace events from the REST reply stay in the plugin; the service only asks that they seed the registry (see "One registry for the stream and the poll"). - Deciding what an unacknowledged order means. @@ -888,14 +882,14 @@ inline `Task.Delay` block with `Watch` on the unknown ids. - **A marker interface as the message handler's type** — Schwab's current answer to two message types in one handler (`IOrderUpdateMessage`). As the shared answer it fails the same way the core interface does: every plugin edits its wire models, and the core `BrokerOrderState` cannot implement a per-plugin interface. - Replaced by the `BrokerageMessageQueue` split (see "One lock for two message types"). + Replaced by the non-generic multi-source handler (see "One lock for two message types"). - **A dual-generic handler**, `BrokerageConcurrentMessageHandler` with the stream type and the polled type. Rejected: every existing plugin migrates to the new shape even with no polling, a plugin with no stream (IB) - has no honest `T`, and a third source would need ``. The queue split adds a type with a registration - call instead of a type parameter. + has no honest `T`, and a third source would need ``. The non-generic handler adds a `Register` call + instead of a type parameter. - **A handler base class that queues work items (`Action`) instead of messages.** The typed wrapper would then wrap every stream message in a new closure — one allocation per message on the hottest path a brokerage has. - The queue split keeps the message itself in the buffer and allocates only at registration. + The non-generic handler keeps the message itself in the buffer and allocates only at registration. - **Put it on the `Brokerage` base class, driven by the engine, like the cash sync.** `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:480`, called from `Engine/TransactionHandlers/BrokerageTransactionHandler.cs:731`) is the existing shape for core-driven periodic @@ -925,7 +919,7 @@ inline `Task.Delay` block with `Watch` on the unknown ids. | Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier ships this trade-off today (`TradierBrokerage.cs:1469-1472`); a shorter poll interval narrows it. | | Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order is unacknowledged, and the interval is a constructor argument the plugin picks. | | A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | -| A plugin passes `ProcessOrderState` as the route and gets fill-before-submit | The route is documented as "your message handler", and wiring it right is two lines: `RegisterMessageType(ProcessOrderState)` once, `HandleNewMessage` as the route. Schwab and Public have a handler; IB and Tradier do not, and adding one is named in their rollout steps. | +| A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; IB and Tradier do not, and adding one is named in their rollout steps. | | The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | | The watch timeout cannot tell "never arrived" from "filled instantly" | It does not try. `OrderNotAcknowledged` hands the question to the brokerage, which has the endpoints to answer it. | | Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | @@ -933,8 +927,8 @@ inline `Task.Delay` block with `Watch` on the unknown ids. ## Rollout -1. This PR: the `BrokerageMessageQueue` split of the message handler, the service, the snapshot, and their unit - tests, no brokerage changes. The split keeps the handler's public surface, so every plugin compiles as before. +1. This PR: the non-generic multi-source message handler, the service, the snapshot, and their unit + tests, no brokerage changes. The generic handler is untouched, so every plugin compiles as before. 2. InteractiveBrokers: add the message handler it does not have, then replace the `NoBrokerageResponse` error and the invented `Submitted` with a watch. This is the proof that the abstraction holds for a plugin that did not write it. diff --git a/Tests/Brokerages/BrokerageConcurrentMessageHandlerMultiSourceTests.cs b/Tests/Brokerages/BrokerageConcurrentMessageHandlerMultiSourceTests.cs new file mode 100644 index 000000000000..8c56a497f98e --- /dev/null +++ b/Tests/Brokerages/BrokerageConcurrentMessageHandlerMultiSourceTests.cs @@ -0,0 +1,107 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using NUnit.Framework; +using System.Threading; +using System.Threading.Tasks; +using QuantConnect.Brokerages; +using System.Collections.Generic; + +namespace QuantConnect.Tests.Brokerages +{ + /// + /// Tests for the non-generic, multi-source . + /// + [TestFixture] + public class BrokerageConcurrentMessageHandlerMultiSourceTests + { + [Test] + public void TwoRegisteredMessageTypesShareOneLock() + { + var processed = new List(); + using var handler = new BrokerageConcurrentMessageHandler(concurrencyEnabled: true); + handler.Register(stream => processed.Add($"stream:{stream.Value}")); + handler.Register(polled => processed.Add($"poll:{polled.Value}")); + + using var lockedEvent = new ManualResetEventSlim(false); + using var releaseEvent = new ManualResetEventSlim(false); + + // Simulates an order request holding WithLockedStream while a place/update/cancel call is in flight. + var writer = Task.Run(() => + { + handler.WithLockedStream(() => + { + lockedEvent.Set(); + releaseEvent.Wait(); + }); + }); + + lockedEvent.Wait(); + + // A "stream" message and a "poll" message both queue behind the write lock instead of running concurrently with it. + handler.HandleNewMessage(new StreamMessage("s1")); + handler.HandleNewMessage(new PolledMessage("p1")); + + Assert.AreEqual(0, processed.Count); + + releaseEvent.Set(); + writer.Wait(); + + CollectionAssert.AreEqual(new[] { "stream:s1", "poll:p1" }, processed); + } + + [Test] + public void MessageWithNoRegisteredListenerIsSkippedWithoutStoppingLaterMessages() + { + var processed = new List(); + using var handler = new BrokerageConcurrentMessageHandler(); + handler.Register(stream => processed.Add(stream.Value)); + + // Nobody listens for PolledMessage: must not throw, and must not block s1 behind it. + Assert.DoesNotThrow(() => handler.HandleNewMessage(new PolledMessage("ignored"))); + handler.HandleNewMessage(new StreamMessage("s1")); + + CollectionAssert.AreEqual(new[] { "s1" }, processed); + } + + [Test] + public void ListenerRegisteredWhileMessagesFlowReceivesLaterMessages() + { + var processed = new List(); + using var handler = new BrokerageConcurrentMessageHandler(); + handler.Register(stream => processed.Add($"stream:{stream.Value}")); + + handler.HandleNewMessage(new StreamMessage("s1")); + + // The order polling service registers its own type mid-run, when a stream dies. + handler.Register(polled => processed.Add($"poll:{polled.Value}")); + handler.HandleNewMessage(new PolledMessage("p1")); + + CollectionAssert.AreEqual(new[] { "stream:s1", "poll:p1" }, processed); + } + + private sealed class StreamMessage + { + public string Value { get; } + public StreamMessage(string value) => Value = value; + } + + private sealed class PolledMessage + { + public string Value { get; } + public PolledMessage(string value) => Value = value; + } + } +} diff --git a/Tests/Brokerages/BrokerageMessageQueueTests.cs b/Tests/Brokerages/BrokerageMessageQueueTests.cs deleted file mode 100644 index df06a457b966..000000000000 --- a/Tests/Brokerages/BrokerageMessageQueueTests.cs +++ /dev/null @@ -1,151 +0,0 @@ -/* - * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. - * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. -*/ - -using System; -using NUnit.Framework; -using System.Threading; -using System.Threading.Tasks; -using QuantConnect.Brokerages; -using System.Collections.Generic; - -namespace QuantConnect.Tests.Brokerages -{ - [TestFixture] - public class BrokerageMessageQueueTests - { - [Test] - public void TwoSubscribedMessageTypesShareOneLock() - { - var processed = new List(); - var queue = new BrokerageMessageQueue(concurrencyEnabled: true); - queue.MessageReceived += message => - { - switch (message) - { - case StreamMessage stream: - processed.Add($"stream:{stream.Value}"); - break; - case PolledMessage polled: - processed.Add($"poll:{polled.Value}"); - break; - } - }; - - using var lockedEvent = new ManualResetEventSlim(false); - using var releaseEvent = new ManualResetEventSlim(false); - - // Simulates an order request holding WithLockedStream while a place/update/cancel call is in flight. - var writer = Task.Run(() => - { - queue.WithLockedStream(() => - { - lockedEvent.Set(); - releaseEvent.Wait(); - }); - }); - - lockedEvent.Wait(); - - // A "stream" message and a "poll" message both queue behind the write lock instead of running concurrently with it. - queue.Enqueue(new StreamMessage("s1")); - queue.Enqueue(new PolledMessage("p1")); - - Assert.AreEqual(0, processed.Count); - - releaseEvent.Set(); - writer.Wait(); - - CollectionAssert.AreEqual(new[] { "stream:s1", "poll:p1" }, processed); - } - - [Test] - public void MessageWithNoInterestedSubscriberIsSkippedWithoutStoppingLaterMessages() - { - var processed = new List(); - var queue = new BrokerageMessageQueue(); - queue.MessageReceived += message => - { - if (message is StreamMessage stream) - { - processed.Add(stream.Value); - } - }; - - // Nobody filters for PolledMessage: must not throw, and must not block s1 behind it. - Assert.DoesNotThrow(() => queue.Enqueue(new PolledMessage("ignored"))); - queue.Enqueue(new StreamMessage("s1")); - - CollectionAssert.AreEqual(new[] { "s1" }, processed); - } - - [Test] - public void ConcurrentMessageHandlerHandlesARegisteredSecondMessageType() - { - var processed = new List(); - var handler = new BrokerageConcurrentMessageHandler(m => processed.Add($"stream:{m}")); - - // A second, unrelated message type registered on the same handler - standing in for the order - // polling service registering BrokerOrderState alongside the brokerage's own stream type. - handler.RegisterMessageType(m => processed.Add($"poll:{m.Value}")); - - // Same method name, either type - the generic argument is inferred from what is passed in. - handler.HandleNewMessage("s1"); - handler.HandleNewMessage(new PolledMessage("p1")); - - CollectionAssert.AreEqual(new[] { "stream:s1", "poll:p1" }, processed); - } - - [Test] - public void ConcurrentMessageHandlerStillBlocksOnWithLockedStream() - { - var processed = new List(); - var handler = new BrokerageConcurrentMessageHandler(processed.Add, concurrencyEnabled: true); - - using var lockedEvent = new ManualResetEventSlim(false); - using var releaseEvent = new ManualResetEventSlim(false); - - var writer = Task.Run(() => - { - handler.WithLockedStream(() => - { - lockedEvent.Set(); - releaseEvent.Wait(); - }); - }); - - lockedEvent.Wait(); - - handler.HandleNewMessage("s1"); - Assert.AreEqual(0, processed.Count); - - releaseEvent.Set(); - writer.Wait(); - - CollectionAssert.AreEqual(new[] { "s1" }, processed); - } - - private sealed class StreamMessage - { - public string Value { get; } - public StreamMessage(string value) => Value = value; - } - - private sealed class PolledMessage - { - public string Value { get; } - public PolledMessage(string value) => Value = value; - } - } -} diff --git a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs index cc5fb2e3dbba..ef2c2de40307 100644 --- a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs @@ -16,6 +16,7 @@ using System; using System.Linq; using System.Threading; +using System.Threading.Tasks; using NUnit.Framework; using QuantConnect.Orders; using System.Collections.Generic; @@ -37,7 +38,7 @@ public void SetUp() _orderProvider = new OrderProvider(); _orderEvents = new List(); // the read is unused: these tests drive the diff directly through ProcessOrderState - _service = new AllOrdersPollingService(() => Array.Empty(), route: null, _orderProvider, + _service = new AllOrdersPollingService(() => Array.Empty(), messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(50), watchTimeout: TimeSpan.FromMilliseconds(120)); _service.OrderEvents += (_, orderEvents) => _orderEvents.AddRange(orderEvents); } @@ -405,7 +406,7 @@ public void WatchTimeoutFiresOnceAndUnwatchesTheId() var raised = new List(); using var service = new PerOrderIdPollingService( _ => null, // the broker never knows the id - route: null, + messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25), watchTimeout: TimeSpan.FromMilliseconds(75)); @@ -442,7 +443,7 @@ public void AcknowledgedWatchNeverTimesOut() var fired = 0; using var service = new PerOrderIdPollingService( _ => null, - route: null, + messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25), watchTimeout: TimeSpan.FromMilliseconds(75)); @@ -467,7 +468,7 @@ public void RepeatedReadFailuresRaiseOneWarningPerOutage() using var warned = new AutoResetEvent(false); using var service = new AllOrdersPollingService( () => fail ? throw new Exception("read failed") : Array.Empty(), - route: null, + messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25)); service.Message += (_, message) => @@ -505,14 +506,13 @@ public void RepeatedReadFailuresRaiseOneWarningPerOutage() } [Test] - public void PerOrderIdSweepReadsOnlyWatchedIdsAndRoutesTheStates() + public void PerOrderIdSweepReadsOnlyWatchedIdsAndProcessesTheStates() { AddOrder(100m, "42"); var readIds = new List(); - using var routed = new ManualResetEventSlim(false); - // the route loops each state back into the diff, standing in for the message handler - PerOrderIdPollingService service = null; - using var serviceReference = service = new PerOrderIdPollingService( + using var processed = new ManualResetEventSlim(false); + // no message handler: the loop hands each state straight into the diff + using var service = new PerOrderIdPollingService( brokerageId => { lock (readIds) @@ -521,7 +521,7 @@ public void PerOrderIdSweepReadsOnlyWatchedIdsAndRoutesTheStates() } return State(brokerageId, OrderStatus.Submitted); }, - route: orderState => service.ProcessOrderState(orderState), + messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25)); var events = new List(); @@ -531,7 +531,7 @@ public void PerOrderIdSweepReadsOnlyWatchedIdsAndRoutesTheStates() { events.AddRange(orderEvents); } - routed.Set(); + processed.Set(); }; // nothing watched: sweeps read nothing @@ -543,7 +543,7 @@ public void PerOrderIdSweepReadsOnlyWatchedIdsAndRoutesTheStates() } service.Watch("42"); - Assert.IsTrue(routed.Wait(TimeSpan.FromSeconds(5)), "the watched id never produced an event"); + Assert.IsTrue(processed.Wait(TimeSpan.FromSeconds(5)), "the watched id never produced an event"); service.Stop(); lock (readIds) @@ -579,22 +579,20 @@ public void StartAndStopAreIdempotentAndDisposeIsFinal() } [Test] - public void SeedAndStartDrainsThenSeedsThenStarts() + public void SeedAndStartSeedsThenStarts() { AddOrder(233m, "42", OrderStatus.Submitted); - var steps = new List(); + var seeded = new List(); - _service.SeedAndStart( - drainBufferedMessages: () => steps.Add("drain"), - seed: order => - { - steps.Add($"seed:{order.BrokerId[0]}"); - // the stream already reported 100 of the 233 shares - return State(order.BrokerId[0], order.Status, filled: 100m); - }); + _service.SeedAndStart(order => + { + seeded.Add(order.BrokerId[0]); + // the stream already reported 100 of the 233 shares + return State(order.BrokerId[0], order.Status, filled: 100m); + }); Assert.IsTrue(_service.IsPolling); - CollectionAssert.AreEqual(new[] { "drain", "seed:42" }, steps); + CollectionAssert.AreEqual(new[] { "42" }, seeded); // the first sweep continues from the seed: only the other 133 shares are reported _service.ProcessOrderState(State("42", OrderStatus.Filled, filled: 233m, price: 310m)); @@ -604,7 +602,59 @@ public void SeedAndStartDrainsThenSeedsThenStarts() } [Test] - public void SeedAndStartWithoutCallbacksJustStarts() + public void SeedAndStartDrainsTheMessageHandlerBeforeSeeding() + { + AddOrder(233m, "42", OrderStatus.Submitted); + var steps = new List(); + using var handler = new BrokerageConcurrentMessageHandler(); + using var service = new AllOrdersPollingService(() => Array.Empty(), handler, _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(50)); + service.OrderEvents += (_, orderEvents) => + { + lock (steps) + { + steps.AddRange(orderEvents.Select(orderEvent => $"event:{orderEvent.FillQuantity}")); + } + }; + + using var lockedEvent = new ManualResetEventSlim(false); + using var releaseEvent = new ManualResetEventSlim(false); + // an order request is in flight while the stream dies + var orderRequest = Task.Run(() => handler.WithLockedStream(() => + { + lockedEvent.Set(); + releaseEvent.Wait(); + })); + lockedEvent.Wait(); + + // the stream delivered a fill that is still buffered behind the order request + handler.HandleNewMessage(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + + var seedAndStart = Task.Run(() => service.SeedAndStart(openOrder => + { + lock (steps) + { + steps.Add("seed"); + } + return State(openOrder.BrokerId[0], openOrder.Status, filled: 100m); + })); + + // the handover waits for the order request instead of seeding early + Assert.IsFalse(seedAndStart.Wait(TimeSpan.FromMilliseconds(100))); + + releaseEvent.Set(); + Task.WaitAll(orderRequest, seedAndStart); + + Assert.IsTrue(service.IsPolling); + lock (steps) + { + // the buffered fill was processed before the seeds were taken + CollectionAssert.AreEqual(new[] { "event:100", "seed" }, steps); + } + } + + [Test] + public void SeedAndStartWithoutSeedJustStarts() { _service.SeedAndStart(); @@ -619,7 +669,7 @@ public void SeedAndStartWhilePollingDoesNothing() _service.Start(); var callCount = 0; - _service.SeedAndStart(drainBufferedMessages: () => callCount++, seed: _ => { callCount++; return null; }); + _service.SeedAndStart(_ => { callCount++; return null; }); Assert.IsTrue(_service.IsPolling); Assert.AreEqual(0, callCount); From 26ff1187e3d41e81fdc11451bb5998a356941b43 Mon Sep 17 00:00:00 2001 From: Romazes Date: Fri, 14 Aug 2026 00:58:35 +0300 Subject: [PATCH 06/25] chore: shorten the diff's submit message to submitted by polling --- Brokerages/Services/BrokerageOrderPollingService.cs | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/BrokerageOrderPollingService.cs index 77870a2caa22..2a32c271b474 100644 --- a/Brokerages/Services/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/BrokerageOrderPollingService.cs @@ -367,7 +367,7 @@ public void ProcessOrderState(BrokerOrderState orderState) { if (leanOrder.Status == OrderStatus.New) { - orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero, "Submitted by order polling") + orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero, "Submitted by polling") { Status = OrderStatus.Submitted }); @@ -652,7 +652,7 @@ private void CheckWatchTimeouts() entry.UnacknowledgedFor += PollInterval; if (entry.UnacknowledgedFor >= WatchTimeout) { - (expired ??= new()).Add(new OrderNotAcknowledgedEventArgs(brokerageId, entry.UnacknowledgedFor)); + (expired ??= []).Add(new OrderNotAcknowledgedEventArgs(brokerageId, entry.UnacknowledgedFor)); } } From e5c0f62ccd798f90001b50d5a5ed1c4a8b1a1f7f Mon Sep 17 00:00:00 2001 From: Romazes Date: Fri, 14 Aug 2026 00:58:35 +0300 Subject: [PATCH 07/25] docs: align the ADR with the Schwab pilot wiring - a plain place watches the main id; the first sweep assigns the leg ids by symbol and reports the submit through the diff - the assignment is one shared method with the stream's OrderAccepted and releases the three minute place wait in both modes - the pilot note lists the leg id assignment and the replace reporting as what stays in the plugin --- .../0001-brokerage-order-polling-service.md | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 5ba75e60fe14..58ed8b031572 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -7,7 +7,8 @@ Proposed - 2026-08-07 Pilot implemented - 2026-08-13: CharlesSchwab is the first plugin on the service (`Lean.Brokerages.CharlesSchwab`, draft PR #107). Its own `OrderUpdatePollingService` and diff are deleted; what remains in the plugin is what this document says remains - the read with its sweep window, the -model-to-state mapping, and the seeded stream-to-polling handover through `SeedAndStart`. +model-to-state mapping, the leg id assignment shared with the stream's `OrderAccepted`, the replace +reporting, and the seeded stream-to-polling handover through `SeedAndStart`. ## Purpose @@ -517,10 +518,12 @@ instead of something the loop calls itself: the message handler sits between the and the queue keeps the order right. The handler's message type also has to cover both the stream's model and the snapshot — the next section is how it does. -One requirement travels with the queue: the place request, the `BrokerId` assignment and the `Submitted` report -must all happen inside the same `WithLockedStream` block that the snapshots queue behind. That is what guarantees -a queued snapshot always resolves the id and finds the status the request already reported. Schwab's fallback path -does all three inside the lock today. +One requirement travels with the queue: the place request, the record the sweep needs to assign the ids, and +the watch must all happen inside the same `WithLockedStream` block that the snapshots queue behind. The leg ids +themselves are assigned by the first sweep that sees the order, from the snapshot: only the broker says which +leg id belongs to which symbol, so an id derived from the request's leg order would be a guess. The assignment +is the same code the stream's `OrderAccepted` runs, and it is what releases the plugin's place wait. A report the +plugin makes itself — a replace — sits in the same block, with its seed. Schwab's fallback path works this way today. ### One lock for two message types @@ -690,9 +693,14 @@ A replace moves the state, because the registry is keyed by brokerage id and a r Inside the same locked block that reports `UpdateSubmitted`, the plugin moves it across: `TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. A broker whose replacement counts its executions from zero seeds the new id with a fresh `Submitted` snapshot instead of -moving the old state — Schwab's replace path does exactly that. The same seed rule covers a plugin that reports -`Submitted` from the request path, as Schwab does without the stream: watch the new id with a `Submitted` snapshot -in that block, so the next sweep does not repeat the submit. +moving the old state — Schwab's replace path does exactly that. The seed rule is simply: what the plugin reports +itself, it seeds in the same locked block, so the next sweep does not repeat it. A plain place reports nothing — +Schwab watches the main id and keeps the placement's Lean orders by symbol; the first sweep to see the order +assigns each leg id from the snapshot — one shared method with the stream's `OrderAccepted`, because only the +broker says which leg id belongs to which symbol — and reports the submit through the diff, one poll interval +later at most. The assignment also releases the three minute place wait, so the same `MissingWebSocketResponse` +error guards a placement nothing ever confirms, on the stream and on the poll alike. The watch doubles as the +alarm: an id the broker never lists raises `OrderNotAcknowledged` instead of staying silent. ### Seed before Start @@ -857,8 +865,10 @@ inline `Task.Delay` block with `Watch` on the unknown ids. whose leg ids are derived rather than returned by the broker should verify they resolve and warn when they do not — the service skips silently. - Routing: handing the service its message handler at construction and forwarding `OrderEvents` to `OnOrderEvents`. -- Reporting without the stream: Schwab's submit and replace events from the REST reply stay in the plugin; the - service only asks that they seed the registry (see "One registry for the stream and the poll"). +- Reporting without the stream: Schwab's replace events from the REST reply stay in the plugin; the + service only asks that they seed the registry (see "One registry for the stream and the poll"). A plain place + is not reported by the plugin at all — it watches the main id, and the first sweep assigns the leg ids by + symbol and reports the submit. - Deciding what an unacknowledged order means. ## Alternatives not taken @@ -934,7 +944,7 @@ inline `Task.Delay` block with `Watch` on the unknown ids. write it. 3. CharlesSchwab and Public.com: delete their service class and their fill/close diff. What stays is real and named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and its - without-stream submit and replace reporting. Two behavior changes are intentional: a Public poll that shows a + without-stream replace reporting. Two behavior changes are intentional: a Public poll that shows a new fill and the cancel together now emits both — today's code drops the cancel (`PublicBrokerage.Brokerage.cs:629-638`) — and polled fills are priced at the broker's reported price of the sweep, so Public's change-of-average recovery and Schwab's per-execution prices become per-sweep prices while From d9f62ce33608dda9661d8d4605839a363bc9434d Mon Sep 17 00:00:00 2001 From: Romazes Date: Fri, 14 Aug 2026 21:19:50 +0300 Subject: [PATCH 08/25] feature: report a replace's update submit through the polling diff - watch replacement watches the new id of a replace and drops the replaced one - the diff's first state for a marked id reports the update submit - the new id starts with no fill state; carry-over brokers seed instead --- .../Services/BrokerageOrderPollingService.cs | 46 ++++++++++++++- .../BrokerageOrderPollingServiceTests.cs | 59 +++++++++++++++++++ 2 files changed, 104 insertions(+), 1 deletion(-) diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/BrokerageOrderPollingService.cs index 2a32c271b474..0709a8b78f1c 100644 --- a/Brokerages/Services/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/BrokerageOrderPollingService.cs @@ -231,6 +231,34 @@ public void Watch(string brokerageId, BrokerOrderState lastSeen) } } + /// + /// Watches the new brokerage order id of a replace and drops the replaced id in the same step. + /// The first state to carry the new id reports the order as update submitted, which a stream + /// would otherwise do. The new id starts with no fill state, because a replacement that counts + /// its executions from zero must not inherit the old order's numbers; a broker that carries the + /// fills across a replace seeds with instead. + /// + /// The brokerage order id the replacement runs under. + /// The replaced brokerage order id, or null when it is unknown. + public void WatchReplacement(string brokerageId, string previousBrokerageId) + { + lock (_lock) + { + if (previousBrokerageId != null) + { + _orderStates.Remove(previousBrokerageId); + } + + if (!_orderStates.TryGetValue(brokerageId, out var entry)) + { + entry = new OrderStateEntry(); + _orderStates[brokerageId] = entry; + } + entry.Watched = true; + entry.IsReplacement = true; + } + } + /// /// Stops watching an order and drops its state. /// @@ -358,7 +386,9 @@ public void ProcessOrderState(BrokerOrderState orderState) // the submit first, once: when nothing was emitted for the id yet, the Lean order is still New, // and the state is not a reject. Lean requires it before any fill, and a market order can - // already be Filled the first time a poll sees it. + // already be Filled the first time a poll sees it. The new id of a replace is the one case + // where the Lean order is already past New: there the state proves the replacement is live, + // so the update submit goes out instead. if (!entry.SubmitReported && (entry.LastSeen == null || entry.LastSeen.Status == OrderStatus.New) && orderState.Status != OrderStatus.Invalid) @@ -373,6 +403,14 @@ public void ProcessOrderState(BrokerOrderState orderState) }); entry.SubmitReported = true; } + else if (entry.IsReplacement && !leanOrder.Status.IsClosed()) + { + orderEvents.Add(new OrderEvent(leanOrder, timeUtc, OrderFee.Zero, "Update submitted by polling") + { + Status = OrderStatus.UpdateSubmitted + }); + entry.SubmitReported = true; + } } } @@ -706,6 +744,12 @@ private class OrderStateEntry /// public bool Watched; + /// + /// Set by : the id is the new id of a replace, so the first + /// state to carry it reports the update submit instead of a plain submit. + /// + public bool IsReplacement; + /// /// Set once anything carried the id: a polled state, a stream write, or a seed. Stops the /// watch timeout. diff --git a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs index ef2c2de40307..b143fc932ef5 100644 --- a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs @@ -101,6 +101,65 @@ public void FirstStateAlreadyFilledEmitsSubmitBeforeFill() Assert.AreEqual(310m, _orderEvents[1].FillPrice); } + [Test] + public void ReplacementWatchReportsUpdateSubmittedOnce() + { + // a replace moved the order onto a new brokerage id, and the plugin marked the new id + var order = AddOrder(100m, "42", OrderStatus.Submitted); + order.BrokerId.Clear(); + order.BrokerId.Add("43"); + _service.WatchReplacement("43", "42"); + + _service.ProcessOrderState(State("43", OrderStatus.Submitted)); + + var updateSubmit = _orderEvents.Single(); + Assert.AreEqual(OrderStatus.UpdateSubmitted, updateSubmit.Status); + Assert.AreEqual("Update submitted by polling", updateSubmit.Message); + + // the same state again reports nothing new + _service.ProcessOrderState(State("43", OrderStatus.Submitted)); + Assert.AreEqual(1, _orderEvents.Count); + } + + [Test] + public void ReplacementWatchDropsThePreviousIdAndCountsFillsFromZero() + { + // the old id already reported a fill, then the replace re-keys the order. The replacement + // counts its executions from zero, so its first fill must not be shrunk by the old total. + var order = AddOrder(200m, "42", OrderStatus.Submitted); + _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + Assert.AreEqual(100m, _orderEvents.Single().FillQuantity); + + order.BrokerId.Clear(); + order.BrokerId.Add("43"); + _service.WatchReplacement("43", "42"); + _orderEvents.Clear(); + + // the first state of the new id can already carry a fill: the update submit still goes first + _service.ProcessOrderState(State("43", OrderStatus.PartiallyFilled, filled: 40m, price: 311m)); + + Assert.AreEqual(2, _orderEvents.Count); + Assert.AreEqual(OrderStatus.UpdateSubmitted, _orderEvents[0].Status); + Assert.AreEqual(OrderStatus.PartiallyFilled, _orderEvents[1].Status); + Assert.AreEqual(40m, _orderEvents[1].FillQuantity); + + // the old order's own end no longer resolves to the Lean order, so it reports nothing + _service.ProcessOrderState(State("42", OrderStatus.Canceled)); + Assert.AreEqual(2, _orderEvents.Count); + } + + [Test] + public void LiveOrderWithoutReplacementMarkStaysQuiet() + { + // an order the stream already confirmed, or one adopted at startup: it is past New in Lean + // and its id carries no replacement mark, so a sweep seeing it for the first time stays quiet + AddOrder(100m, "42", OrderStatus.Submitted); + + _service.ProcessOrderState(State("42", OrderStatus.Submitted)); + + Assert.IsEmpty(_orderEvents); + } + [Test] public void CumulativeFillsNeverRepeat() { From 90b9579c34ff30f5d4606c3c3b3473ba9d5ea3f0 Mon Sep 17 00:00:00 2001 From: Romazes Date: Fri, 14 Aug 2026 21:19:51 +0300 Subject: [PATCH 09/25] docs: add the replace survey and the replacement watch to the adr - survey of nine sibling plugins' update paths, checked line by line - replacement watch design, the no-wait rule, and the rejected event reuse --- .../0001-brokerage-order-polling-service.md | 157 +++++++++++++++--- 1 file changed, 134 insertions(+), 23 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 58ed8b031572..e4cf255e48bd 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -7,8 +7,12 @@ Proposed - 2026-08-07 Pilot implemented - 2026-08-13: CharlesSchwab is the first plugin on the service (`Lean.Brokerages.CharlesSchwab`, draft PR #107). Its own `OrderUpdatePollingService` and diff are deleted; what remains in the plugin is what this document says remains - the read with its sweep window, the -model-to-state mapping, the leg id assignment shared with the stream's `OrderAccepted`, the replace -reporting, and the seeded stream-to-polling handover through `SeedAndStart`. +model-to-state mapping, the leg id assignment shared with the stream's `OrderAccepted`, and the seeded +stream-to-polling handover through `SeedAndStart`. + +Replacement watch implemented - 2026-08-14: a polled replace goes through `WatchReplacement` and the diff +reports the update submit. The plugin's own replace reporting and its by-position leg id derivation are +deleted. Backed by the replace survey (see "A replace, across the brokers"). ## Purpose @@ -420,6 +424,11 @@ public abstract class BrokerageOrderPollingService : IDisposable /// the request path, and to move state onto the new id of a replace. public void Watch(string brokerageId, BrokerOrderState lastSeen); + /// Watches the new brokerage order id of a replace and drops the replaced id in the same + /// step, so the first state to carry the new id reports the update submit. The new id starts with + /// no fill state; a broker that carries fills across a replace seeds with Watch instead. + public void WatchReplacement(string brokerageId, string previousBrokerageId); + /// Stops watching an order and drops its state. public void Unwatch(string brokerageId); @@ -518,12 +527,16 @@ instead of something the loop calls itself: the message handler sits between the and the queue keeps the order right. The handler's message type also has to cover both the stream's model and the snapshot — the next section is how it does. -One requirement travels with the queue: the place request, the record the sweep needs to assign the ids, and +One requirement travels with the queue: the order request, the record the sweep needs to assign the ids, and the watch must all happen inside the same `WithLockedStream` block that the snapshots queue behind. The leg ids themselves are assigned by the first sweep that sees the order, from the snapshot: only the broker says which -leg id belongs to which symbol, so an id derived from the request's leg order would be a guess. The assignment -is the same code the stream's `OrderAccepted` runs, and it is what releases the plugin's place wait. A report the -plugin makes itself — a replace — sits in the same block, with its seed. Schwab's fallback path works this way today. +leg id belongs to which symbol, so an id derived from the request's leg order would be a guess. The +placement's assignment is the same code the stream's `OrderAccepted` runs, and it is what releases the +plugin's place wait. A replace has a mirror of it, fed from its own pending record: its assignment moves +each Lean order onto its new id and marks it through `WatchReplacement`, so the diff reports the update +submit instead of a plain submit. Nothing waits on a replacement - the replace reply already confirmed it, +and the watch raises `OrderNotAcknowledged` if the broker never lists it. Schwab's fallback path works this +way today. ### One lock for two message types @@ -580,6 +593,8 @@ the Lean order is still New, and the snapshot is not a reject. Lean requires it any fill, and a market order can already be Filled the first time a poll sees it. The second gate matters in bulk mode beside a live stream: orders the stream already confirmed are not New in Lean, so a sweep seeing them for the first time stays quiet. +The marked exception is the new id of a replace: WatchReplacement flagged it, its Lean +order is already past New, and the first state reports UpdateSubmitted instead. then the fills, so a close can never outrun a fill of the same order: a fill needs both numbers: without a FillPrice nothing is emitted and @@ -689,18 +704,103 @@ Everything it owes the service is one seed at the moment the stream dies (see "S complements the plugin's existing logic, it never replaces it — and in fallback mode the stream and the service never even touch while the stream lives. -A replace moves the state, because the registry is keyed by brokerage id and a replace gives the order a new one. -Inside the same locked block that reports `UpdateSubmitted`, the plugin moves it across: -`TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. A broker -whose replacement counts its executions from zero seeds the new id with a fresh `Submitted` snapshot instead of -moving the old state — Schwab's replace path does exactly that. The seed rule is simply: what the plugin reports -itself, it seeds in the same locked block, so the next sweep does not repeat it. A plain place reports nothing — -Schwab watches the main id and keeps the placement's Lean orders by symbol; the first sweep to see the order -assigns each leg id from the snapshot — one shared method with the stream's `OrderAccepted`, because only the -broker says which leg id belongs to which symbol — and reports the submit through the diff, one poll interval -later at most. The assignment also releases the three minute place wait, so the same `MissingWebSocketResponse` -error guards a placement nothing ever confirms, on the stream and on the poll alike. The watch doubles as the -alarm: an id the broker never lists raises `OrderNotAcknowledged` instead of staying silent. +A place and a replace are reported the same way: not by the plugin. For a place, Schwab watches the main id +and keeps the placement's Lean orders by symbol; the first sweep to see the order assigns each leg id from +the snapshot — one shared method with the stream's `OrderAccepted`, because only the broker says which leg id +belongs to which symbol — and reports the submit through the diff, one poll interval later at most. For a +replace, its own pending record goes in under the new main id, and a mirror of the assignment handles it: it +moves each Lean order onto its new leg id and marks it with `WatchReplacement`, which drops the replaced id +in the same call — so the dead order's last snapshot reports nothing — and makes the diff report the update +submit instead of a plain submit. The new id starts with no fill state, because a Charles Schwab replacement counts +its executions from zero; a broker that carries fills across a replace moves the state instead: +`TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. + +The seed rule stays for what a plugin still reports itself: it seeds in the same locked block, so the next +sweep does not repeat it. The assignment also releases the three minute place wait, so the same +`MissingWebSocketResponse` error guards a placement nothing ever confirms, on the stream and on the poll +alike. A replacement has no wait of its own - the replace reply already confirmed it. The watch doubles as +the alarm: an id the broker never lists raises `OrderNotAcknowledged` instead of staying silent. + +### A replace, across the brokers + +A replace is the one order action that can change the key the registry lives on. Whether it does is the +broker's design, not the plugin's choice — so before deciding what the service should own here, the update +path of nine sibling plugins was read, twice (a survey pass and a line-by-line check pass), for the four +facts that matter to a poll: does the id change, where does the new id come from, who reports +`UpdateSubmitted` today, and what happens to the fill count. + +| Plugin | Update | Id after a replace | New id from | `UpdateSubmitted` reported by | Fills after | +| --- | --- | --- | --- | --- | --- | +| CharlesSchwab | yes, combos too | new — one per leg | REST reply, legs from snapshot/stream | plugin after REST (poll mode); stream `ChangeAccepted` | reset | +| Tastytrade | yes, combos too | new — one for all legs | REST reply | stream `Routed`/`Live`, waited 100 s | unknown | +| Alpaca | yes, no combos | new | REST reply | stream `Replaced` only | unknown | +| InteractiveBrokers | yes, combos too | same — modified in place | — | stream `orderStatus` + an updated flag | carry over | +| TradeStation | yes, combos too | same | — (the reply's `OrderID` is never read) | plugin after REST; the stream echo is swallowed | carry over | +| Tradier | price and type only | same | — | plugin after REST | carry over | +| Public.com | single-leg only | same | — (the reply echoes the id) | plugin after REST | carry over | +| Webull | single-leg only | same (`client_order_id` kept) | — | stream `MODIFY_SUCCESS` | carry over | +| ByBit | futures amend only | same | — | plugin after REST | carry over | +| Binance | no — cancel and re-create | — | — | never emitted | — | + +The rows in code: IB re-sends the same broker id (`InteractiveBrokersBrokerage.cs:1599,1626`), TradeStation +PUTs to the existing id and never reads the reply's `OrderID` (`Api/TradeStationApiClient.cs:329-335`), +Tradier's PUT has no quantity parameter at all (`TradierBrokerage.cs:486-507`), Public replaces in place +with the id echoed back (`Api/ApiClient.cs:351-362`), Webull keeps the `client_order_id` that is the Lean +`BrokerId` (`Api/ApiClient.cs:595-605`), ByBit amends futures under the same id +(`Api/BybitTradeApiEndpoint.cs:94-100`) and cannot amend spot (`BybitBrokerage.Brokerage.cs:200-204`), and +Binance's `UpdateOrder` throws (`BinanceBrokerage.cs:346-349`). + +**The same-id half needs nothing new from the service.** The watched id survives the replace, and the fill +count continues on it — all six carry-over cells are verified in code, e.g. TradeStation's cumulative +`ExecQuantity` delta is never reset by an update (`TradeStationBrokerage.cs:1187-1194`). And the report +itself can never come from a sweep: the state carries a status and two fill numbers, no price and no +quantity, so a modified order polls exactly like an unmodified one. The rule for a same-id plugin is +Tradier's and Public's, already shipping: report `UpdateSubmitted` right after the REST reply +(`TradierBrokerage.cs:970-971`, `PublicBrokerage.Brokerage.cs:687`) and leave the registry alone — Public +runs on this service today and its `UpdateOrder` makes no service call. The two that report from the stream +instead (IB `InteractiveBrokersBrokerage.cs:2359,2380`, Webull `WebullBrokerage.Brokerage.cs:448-453`) +simply have no update report while their channel is down; on adoption they move the report next to the REST +reply, like the other four. + +**The cancel-replace half is where the report can move into the service.** All three learn the new id +synchronously, from the REST reply itself: Schwab's `UpdateOrder` result, Alpaca's `PatchOrderAsync` +response (`AlpacaBrokerage.cs:782-793`), Tastytrade's `ReplaceOrderById` return +(`Api/TastytradeApiClient.cs:172-186`). So the plugin can always watch the new id inside the same locked +block as the request — the same rule the place already follows. What no REST reply gives is the +confirmation that the replacement is live: Tastytrade holds its `UpdateSubmitted` until the stream says the +new order is `Routed`/`Live` and waits 100 seconds for that (`TastytradeBrokerage.Brokerage.cs:431,530-533`), +and Alpaca reports it only from the stream's `Replaced` event (`AlpacaBrokerage.cs:677,1058-1059`), so with +the stream down it is never reported. The first sweep that lists the new id is exactly that confirmation. +So the general shape is one addition, the mirror of the place rule: a **replacement watch** — +`WatchReplacement(newBrokerageId, previousBrokerageId)` marks the new id and drops the old one in one +locked step, and the diff's first state for a marked id whose Lean order is open but past `New` reports +`UpdateSubmitted`, "Update submitted by polling". Combos fit both shapes: Tastytrade's whole combo takes +one new id (`TastytradeBrokerage.Brokerage.cs:416`), and Schwab's per-leg ids go through the same by-symbol +assignment its place already runs. + +**Dropping the old id is a correctness step, not housekeeping.** A cancel-replace ends the old order at the +broker, and the old order's last snapshot says so — Schwab lists it `Replaced`, Tastytrade sends +`Cancelled` for it, Alpaca `Replaced`. A registry still holding the old id could read that as the Lean +order's end. The streaming plugins prove the hazard is real: Tastytrade swallows exactly this `Cancelled` +today (`TastytradeBrokerage.Brokerage.cs:575-581`), and Schwab's status mapping keeps `Replaced` +non-terminal on purpose. The replacement watch removes the old entry in the same call, and once the plugin +re-keys the Lean order the old id no longer resolves — both guards, one step. + +**The fill count restarts with the id.** Schwab's replacement counts its executions from zero, so the new +entry starts at zero reported. Alpaca and Tastytrade leave no evidence either way in code or tests — +"unknown" is the honest cell. A broker that turns out to carry fills across a replace seeds instead of +starting fresh: `TryGetLastOrderState(oldId)` then `Watch(newId, lastSeen)`, two calls that already exist. + +**Telling Lean about the new id stays in the plugin.** Three plugins, three conventions: Schwab swaps the +whole `BrokerId` list through `OnOrderIdChangedEvent`, Tastytrade does the same right after the REST reply +(`TastytradeBrokerage.Brokerage.cs:416`), and Alpaca appends the new id and reads `BrokerId.Last()` from +then on (`AlpacaBrokerage.cs:789-793`). The service never touches `Order.BrokerId` — the plugin tells +Lean, the service only watches. + +So the seed rule above stands for the same-id half — what a plugin reports itself, it seeds — and the +cancel-replace half gets the replacement watch instead: one additive method and one diff branch. Schwab, +the pilot, runs on it: its polled replace goes through a mirror of its place's by-symbol assignment and the +diff reports the update submit. The by-position leg id derivation its first pilot shipped with is deleted. ### Seed before Start @@ -865,10 +965,9 @@ inline `Task.Delay` block with `Watch` on the unknown ids. whose leg ids are derived rather than returned by the broker should verify they resolve and warn when they do not — the service skips silently. - Routing: handing the service its message handler at construction and forwarding `OrderEvents` to `OnOrderEvents`. -- Reporting without the stream: Schwab's replace events from the REST reply stay in the plugin; the - service only asks that they seed the registry (see "One registry for the stream and the poll"). A plain place - is not reported by the plugin at all — it watches the main id, and the first sweep assigns the leg ids by - symbol and reports the submit. +- Reporting without the stream: the plugin reports neither the place nor the replace itself. It watches the + main id, the first sweep assigns the leg ids by symbol — a replacement's through `WatchReplacement` — and + the diff reports the submit or the update submit. - Deciding what an unacknowledged order means. ## Alternatives not taken @@ -918,6 +1017,18 @@ inline `Task.Delay` block with `Watch` on the unknown ids. price precision for one broker at the cost of a second diff branch every plugin has to reason about. If it is ever wanted, it comes back as one additive nullable field without breaking anyone — the same goes for a `GetOrderExecutions` API on `IBrokerage`. +- **Arm the replacement watch from the `OrderIdChanged` event instead of `WatchReplacement`.** The plugin + already raises `OnOrderIdChangedEvent` when a replace re-keys a Lean order, so the service could subscribe + to it instead of offering a method. Rejected three times over. The event does not mean "replace": Lean + core raises it on a plain place — the cross-zero flow assigns the second part's id through it + (`Brokerages/Brokerage.cs:842`) — and Binance assigns every initial id with it, so the service would + report an update submit for a placement. The event does not carry the previous id, and the service cannot + recover it: the transaction handler subscribes first and swaps `order.BrokerId` before a later subscriber + runs (`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:1497`), so the old registry entries could + not be dropped. And a stream-alive plugin raises the event while reporting `UpdateSubmitted` itself, so an + event-armed service would report it a second time. The reuse that works is locality, not wiring: the + plugin calls `WatchReplacement` on the same line where it builds the id-changed event, with the old and + new id it already holds. - **Leave it in the plugins.** It is already written three times, and the fourth copy would be IB's. ## Risks @@ -944,7 +1055,7 @@ inline `Task.Delay` block with `Watch` on the unknown ids. write it. 3. CharlesSchwab and Public.com: delete their service class and their fill/close diff. What stays is real and named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and its - without-stream replace reporting. Two behavior changes are intentional: a Public poll that shows a + by-symbol leg id assignment. Two behavior changes are intentional: a Public poll that shows a new fill and the cancel together now emits both — today's code drops the cancel (`PublicBrokerage.Brokerage.cs:629-638`) — and polled fills are priced at the broker's reported price of the sweep, so Public's change-of-average recovery and Schwab's per-execution prices become per-sweep prices while From af65e588985a205ad1b5a466cd9f112a79acf296 Mon Sep 17 00:00:00 2001 From: Romazes Date: Sat, 15 Aug 2026 01:54:17 +0300 Subject: [PATCH 10/25] docs: record the live pilot proof and the public.com adoption in the adr - status entries: pilot verified live on real accounts; public.com is the second plugin on the service - pricing text matches what shipped: public keeps its change-of-average recovery inside its mapping - rollout item 3 trimmed to the one intentional behavior change --- .../0001-brokerage-order-polling-service.md | 29 ++++++++++++++----- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index e4cf255e48bd..8f0c36f0328b 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -14,6 +14,18 @@ Replacement watch implemented - 2026-08-14: a polled replace goes through `Watch reports the update submit. The plugin's own replace reporting and its by-position leg id derivation are deleted. Backed by the replace survey (see "A replace, across the brokers"). +Pilot verified live - 2026-08-14: place, replace, cancel and fill were reported correctly by the polled +connection running next to a streaming one, on real CharlesSchwab accounts. The diff's documented edges +showed up as designed: a market order already filled on its first listing reports the submit and the fill +in one batch, and the executions of one sweep arrive as one event. + +Second plugin adopted - 2026-08-15: Public.com runs on `PerOrderIdPollingService` +(`Lean.Brokerages.Public`, draft PR #6). Its own service class, diff and snapshot model are deleted; the +plugin keeps the get-order read and the model-to-state mapping. Its same-id replace stays plugin-side, so +`WatchReplacement` is not wired, and a get-order 404 maps to a null state - the contract's "the broker +does not know the id". One rollout intention changed: Public kept its change-of-average price recovery, +moved into its mapping (see "The diff", pricing). + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -634,9 +646,11 @@ identical to the first and be lost. Pricing is deliberately simple: the new part takes the state's `FillPrice`, as the broker reported it. When several fills land inside one sweep, the quantity is still exact and the price is the broker's reported price at sweep time, not each fill's own — Tradier's poller ships exactly this trade-off today and documents it -(`TradierBrokerage.cs:1469-1472`). Public today recovers the exact increment price from the change of the average -(`PublicBrokerage.Brokerage.cs:697`); the service drops that arithmetic on purpose — it amplifies broker rounding -and can even go negative on a tiny part, while the simple price needs no guard at all. +(`TradierBrokerage.cs:1469-1472`). Public recovers the exact increment price from the change of the average; +the service refuses that arithmetic on purpose — it amplifies broker rounding and can even go negative on a +tiny part, while the simple price needs no guard at all. Public's adoption kept the recovery anyway, but inside +its own mapping: the state's `FillPrice` arrives already recovered, with a guard that falls back to the plain +average when the new part is not positive. The service still only copies the price it is given. One more detail is load-bearing. **State outlives the terminal event.** Forgetting an order the moment its `Canceled` goes out re-reports every fill if the next sweep lands before Lean applies the event — Schwab's own ADR @@ -1055,11 +1069,10 @@ inline `Task.Delay` block with `Watch` on the unknown ids. write it. 3. CharlesSchwab and Public.com: delete their service class and their fill/close diff. What stays is real and named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and its - by-symbol leg id assignment. Two behavior changes are intentional: a Public poll that shows a - new fill and the cancel together now emits both — today's code drops the cancel - (`PublicBrokerage.Brokerage.cs:629-638`) — and polled fills are priced at the broker's reported price of the - sweep, so Public's change-of-average recovery and Schwab's per-execution prices become per-sweep prices while - the quantities stay exact. + by-symbol leg id assignment. One behavior change is intentional: a Public poll that shows a + new fill and the cancel together now emits both — the old code dropped the cancel. Schwab's per-execution + prices become per-sweep prices while the quantities stay exact; Public kept its change-of-average price + recovery inside its mapping, so its part prices stay exact too. 4. Tradier: replace the inline re-check block with a watch. Tradier splits orders across zero, so one brokerage order can cover only part of the Lean quantity — its watch resolves submissions only, and fills stay on its existing path. From b8bf345c2dece4f6a13a473cdba893037fcdb191 Mon Sep 17 00:00:00 2001 From: Romazes Date: Mon, 17 Aug 2026 19:37:06 +0300 Subject: [PATCH 11/25] feature: create and wire the order polling service from the brokerage - protected create overloads pick the per-order-id or all-orders mode by the read callback - the service events forward onto the brokerage events; a virtual method owns the not-acknowledged warning - dispose covers the created service --- Brokerages/Brokerage.cs | 97 ++++++++++++++++++++++++++++++++++++++++- 1 file changed, 95 insertions(+), 2 deletions(-) diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index 94611485adc0..1f7cb08cb6a0 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -29,6 +29,7 @@ using QuantConnect.Api; using QuantConnect.Brokerages.Authentication; using QuantConnect.Brokerages.CrossZero; +using QuantConnect.Brokerages.Services; using QuantConnect.Util; namespace QuantConnect.Brokerages @@ -149,11 +150,11 @@ protected Brokerage(string name) public abstract void Disconnect(); /// - /// Dispose of the brokerage instance + /// Dispose of the brokerage instance, including the order polling service when one was created /// public virtual void Dispose() { - // NOP + OrderPollingService.DisposeSafely(); } /// @@ -347,6 +348,98 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC return handler; } + #region Order Polling + + /// + /// The order polling service one of the create overloads built and wired, null until then. The brokerage + /// calls Watch, WatchReplacement, Start, Stop and SeedAndStart on it at its own lifecycle points; + /// disposes it. + /// + protected BrokerageOrderPollingService OrderPollingService { get; private set; } + + /// + /// Returns true while the order polling service is running + /// + protected bool IsOrderPolling => OrderPollingService?.IsPolling == true; + + /// + /// Creates a , for a broker with a get-order endpoint, and wires it + /// into this brokerage: polled order events reach , polling outage warnings + /// reach , and an order the broker never reports goes to + /// . The created service is kept in . + /// + /// Reads the current state of one order by its brokerage id. A null + /// return means the broker does not know the id. + /// The brokerage's message handler; the service registers itself and + /// enqueues every polled state through it. Null processes each state directly. + /// Resolves brokerage order ids to Lean orders. + /// How long the loop sleeps between sweeps. Null falls back to the + /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. + /// How long a watched order may stay unreported before + /// is called. Null falls back to one minute. + /// The created and wired service. + protected PerOrderIdPollingService CreateOrderPollingService(Func readOrder, + BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, + TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + { + var service = new PerOrderIdPollingService(readOrder, messageHandler, orderProvider, pollInterval, watchTimeout); + WireOrderPollingService(service); + return service; + } + + /// + /// Creates an , for a broker with only a bulk orders endpoint, and + /// wires it into this brokerage the same way as the per-order overload. + /// + /// Reads every order the broker lists, one state per brokerage order id. + /// The brokerage's message handler; the service registers itself and + /// enqueues every polled state through it. Null processes each state directly. + /// Resolves brokerage order ids to Lean orders. + /// How long the loop sleeps between sweeps. Null falls back to the + /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. + /// How long a watched order may stay unreported before + /// is called. Null falls back to one minute. + /// The created and wired service. + protected AllOrdersPollingService CreateOrderPollingService(Func> readAllOrders, + BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, + TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + { + var service = new AllOrdersPollingService(readAllOrders, messageHandler, orderProvider, pollInterval, watchTimeout); + WireOrderPollingService(service); + return service; + } + + /// + /// Stores the created service and forwards its events onto the brokerage events + /// + /// The service one of the create overloads built. + private void WireOrderPollingService(BrokerageOrderPollingService service) + { + if (OrderPollingService != null) + { + throw new InvalidOperationException($"{Name}: an order polling service was already created."); + } + + service.OrderEvents += (_, orderEvents) => OnOrderEvents(orderEvents); + service.Message += (_, message) => OnMessage(message); + service.OrderNotAcknowledged += (_, notAcknowledged) => OnOrderPollingNotAcknowledged(notAcknowledged); + OrderPollingService = service; + } + + /// + /// Called when the broker never reported a watched order for the whole watch timeout. The default + /// sends one warning message; override it to decide what the silence means for this broker. + /// + /// The brokerage order id and how long it was watched. + protected virtual void OnOrderPollingNotAcknowledged(OrderNotAcknowledgedEventArgs notAcknowledged) + { + OnMessage(new BrokerageMessageEvent(BrokerageMessageType.Warning, "OrderNotAcknowledged", + $"{Name} never reported order '{notAcknowledged.BrokerageOrderId}' after " + + $"{notAcknowledged.WatchedFor.TotalSeconds:F0} seconds of polling. The order may not have been accepted, verify it manually.")); + } + + #endregion + /// /// Helper method that will try to get the live holdings from the provided brokerage data collection else will default to the algorithm state /// From 77d72ceb12aa1d8a220b815d22f5ea129c72f552 Mon Sep 17 00:00:00 2001 From: Romazes Date: Mon, 17 Aug 2026 19:37:06 +0300 Subject: [PATCH 12/25] docs: record the seam, tradier adoption and testing process in the adr - status entries for the wiring seam and the third plugin - cross-zero section: leg chaining from the read and the fill-offset seed - wiring examples moved onto the create seam - testing section: live capture first, recorded mock data, offline checklist --- .../0001-brokerage-order-polling-service.md | 278 ++++++++++++++++-- 1 file changed, 251 insertions(+), 27 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 8f0c36f0328b..98f94d1c55b4 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -26,6 +26,22 @@ plugin keeps the get-order read and the model-to-state mapping. Its same-id repl does not know the id". One rollout intention changed: Public kept its change-of-average price recovery, moved into its mapping (see "The diff", pricing). +Wiring moved into core - 2026-08-17: the root `Brokerage` class gained the seam the base-class section +describes - two protected `CreateOrderPollingService` overloads whose read-callback signature picks the +mode, the service as a protected `OrderPollingService` property with `IsOrderPolling`, a virtual +`OnOrderPollingNotAcknowledged` for the silence warning, and a `Dispose` that covers the service. +CharlesSchwab and Public.com were moved onto it: the creation, the three event forwards and the dispose +call left both plugins (see "Wiring, per plugin"). + +Third plugin adopted - 2026-08-17: Tradier runs on `PerOrderIdPollingService` +(`Lean.Brokerages.Tradier`, branch `feature-core-order-polling-service`). Its fill timer, `CheckForFills` +diff, order cache and unknown-id verification are deleted (~230 lines); the plugin keeps the get-order +read and the mapping. Tradier is the first adopter that splits orders across zero: the legs chain from +the read through the base cross-zero helpers, and the second leg's watch seed carries what the first leg +filled (see "A cross-zero order, two ids"). Two behavior changes are intentional: orders placed outside +Lean are ignored - a per-id read never sees them, where the old code raised a fatal "UnknownOrderId" +error - and the fee attaches once per Lean order instead of once per broker leg. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -667,7 +683,8 @@ handler whether to accept the order, and adopts it with `AddOpenOrder`. TradeSta stream (`TradeStationBrokerage.cs:1088`); a poll can feed the same door. Not in this PR, for one reason: telling "placed outside Lean" apart from "ours, id not assigned yet" needs care — -Tradier's 2-second recheck exists exactly because an unknown id can turn out to be ours a moment later. The safe +the 2-second recheck Tradier's replaced poll carried existed exactly because an unknown id can turn out to be +ours a moment later. The safe shape is: only an id that stays unknown across several sweeps, and is not inside any order request, gets raised as a new brokerage-side order. That rule can be added to the diff later without changing the snapshot or the API, so it is future work, not part of this design. @@ -816,6 +833,32 @@ cancel-replace half gets the replacement watch instead: one additive method and the pilot, runs on it: its polled replace goes through a mirror of its place's by-symbol assignment and the diff reports the update submit. The by-position leg id derivation its first pilot shipped with is deleted. +### A cross-zero order, two ids + +A broker that cannot cross a position through zero gets two brokerage orders for one Lean order: a closing +leg, then an opening leg the base class places when the first one fills +(`TryHandleRemainingCrossZeroOrder`, `Brokerages/Brokerage.cs:892`). That breaks the diff's frame twice. +The diff calls a fill final only when the cumulative reaches the Lean order's whole quantity, and neither +leg reaches it alone. And the trigger for the second leg is the first leg's *broker-side* fill — a state +the diff never surfaces, because at that point the Lean order is only half done. + +Tradier, the first adopter with this split, keeps both jobs in the read, on the base helpers that already +own the pending leg: + +- The read sees the closing leg filled at the broker and hands `TryHandleRemainingCrossZeroOrder` a + `Filled` event carrying the unreported part. The helper knows whether a remaining leg is pending — for + every other order it declines and nothing happens. When it takes the event, it reports the fill as + `PartiallyFilled` and places the second leg itself; the read then records the state through + `UpdateOrderState`, so the sweep's own diff stays silent. +- The second leg's watch seed carries what the first leg filled, and its reads add the same offset to the + leg's own cumulative. The diff's frame is whole again: the leg's last fill reaches the Lean quantity and + reports `Filled`. + +The service itself needs nothing new for this — the seed, `UpdateOrderState` and `TryGetLastOrderState` +were already there. What it costs the plugin is one map (the first leg's filled quantity, keyed by the +Lean order id between the second leg's request and its watch, then by the second leg's brokerage id for +the reads) and one closed-order hook in the read. + ### Seed before Start Polling never starts first: the stream reported orders before it, and the registry must know what was already @@ -912,29 +955,34 @@ brokerage that uses the default. Core helpers already work this way: the message ## Wiring, per plugin -The general shape, for a streaming brokerage with a bulk endpoint — the class is the mode: +The general shape, for a streaming brokerage with a bulk endpoint. The root `Brokerage` class owns the +wiring: one protected create call builds the service, forwards its events onto the brokerage events, +stores it in the protected `OrderPollingService` property and hands it to `Dispose`. The read callback's +signature picks the mode — a bulk read can only build an `AllOrdersPollingService`: ```csharp // bulk broker: one request per sweep reads the whole account. The service wires itself onto the // handler: it registers its diff as a listener and enqueues every snapshot, so the plugin never // touches that relationship again. -_orderPollingService = new AllOrdersPollingService( +CreateOrderPollingService( () => _apiClient.GetAllOrders().Select(ToOrderState), // read: model -> snapshot _messageHandler, _orderProvider, pollInterval: TimeSpan.FromSeconds(3), watchTimeout: TimeSpan.FromMinutes(1)); -_orderPollingService.OrderEvents += (_, orderEvents) => OnOrderEvents(orderEvents); ``` -A per-id broker only changes the class and the read — this is Public.com's own constructor today, snapshot -instead of its DTO: +A per-id broker only changes the read — one call replaces Public.com's construction, its three event +forwards and its dispose line: ```csharp -_orderPollingService = new PerOrderIdPollingService( - brokerageId => ToOrderState(_apiClient.GetOrderById(brokerageId)), - _messageHandler, - _orderProvider); // nothing passed: brokerage-order-poll-interval-ms decides, default 3000 ms +CreateOrderPollingService(ReadOrderState, _messageHandler, _orderProvider); +// BrokerOrderState ReadOrderState(string brokerageId): get-order by id, a 404 maps to null. +// Nothing else passed: brokerage-order-poll-interval-ms decides the interval, default 3000 ms. ``` +What silence means is one override per plugin: `OnOrderPollingNotAcknowledged` defaults to a single +warning built from the brokerage name, and Schwab overrides it to keep the wording its live pilot +verified. + InteractiveBrokers, the plugin with no answer today, adopts `AllOrdersPollingService` (it has no per-id request) with the thinnest possible mapping — id and status from the orders `reqAllOpenOrders` returns, fill fields left null. The fill numbers are not out of reach: the captured `IBApi.Order` already carries a `FilledQuantity` field, and the @@ -943,8 +991,8 @@ The first step stays thin: ```csharp // IBPlaceOrder, instead of blocking up to 5 minutes and then killing the run -_orderPollingService.Watch(ibOrderId.ToStringInvariant()); -_orderPollingService.Start(); // idempotent; Stop once nothing is watched +OrderPollingService.Watch(ibOrderId.ToStringInvariant()); +OrderPollingService.Start(); // idempotent; Stop once nothing is watched ``` One honest cost first: IB has no `BrokerageConcurrentMessageHandler` today — only Schwab and Public do — so its @@ -960,15 +1008,15 @@ The same two lines cover a dropped socket, in any streaming plugin: ```csharp // where the brokerage already handles the connection going down; see "Seed before Start" -_orderPollingService.SeedAndStart(ToSeedState); +OrderPollingService.SeedAndStart(ToSeedState); // on reconnect: one last sweep for the gap, then hand the job back to the stream -_orderPollingService.Stop(); +OrderPollingService.Stop(); ``` CharlesSchwab and Public.com delete their own service class **and their diff**, and keep only the mapping from -their models to the snapshot. Schwab keeps its own rule of never going back to the stream. Tradier replaces the -inline `Task.Delay` block with `Watch` on the unknown ids. +their models to the snapshot. Schwab keeps its own rule of never going back to the stream. Tradier deletes its +fill timer and `CheckForFills` diff the same way; its cross-zero split is "A cross-zero order, two ids". ## What stays in the plugin @@ -978,11 +1026,111 @@ inline `Task.Delay` block with `Watch` on the unknown ids. execution history to the two numbers (Schwab sums its execution legs and takes the newest leg's price). A plugin whose leg ids are derived rather than returned by the broker should verify they resolve and warn when they do not — the service skips silently. -- Routing: handing the service its message handler at construction and forwarding `OrderEvents` to `OnOrderEvents`. +- Routing: handing the create call its message handler, or null when the poll loop is the only caller of + the diff. The event forwards and the dispose are the base class's job. - Reporting without the stream: the plugin reports neither the place nor the replace itself. It watches the main id, the first sweep assigns the leg ids by symbol — a replacement's through `WatchReplacement` — and the diff reports the submit or the update submit. -- Deciding what an unacknowledged order means. +- Deciding what an unacknowledged order means — the `OnOrderPollingNotAcknowledged` override; the default + is one warning built from the brokerage name. + +## Testing the polling in a plugin + +The service's own behavior — the diff, the watch, the registry — is covered once, in Lean's +`Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs`. A plugin does not test the service +again: it tests its read, its mapping and its wiring **through** the service. The offline tests carry +the everyday coverage; the live tests prove the whole path against the real broker — and for a plugin +starting from nothing they come first, because their runs record the data the offline tests replay. The reference is the Schwab pilot's fixture, +`Lean.Brokerages.CharlesSchwab/QuantConnect.CharlesSchwabBrokerage.Tests/CharlesSchwabBrokerageOrderUpdatePollingTests.cs`; +its mock classes sit next to it in `Tests/Models`. + +### The test doubles + +Two subclasses, and no reflection: + +- A mock brokerage deriving from the real one. It runs the normal initialization and overrides exactly + the edges: the API client factory returns a mock, the socket is replaced with one a test can feed + captured messages into, and the protected polling trigger gets a public wrapper (Schwab's + `SwitchToOrderUpdatePolling` calls its stream-loss handler; a polling-primary plugin just calls + `Connect`). It also plays the transaction handler's part where a test has none: its + `OnOrderIdChangedEvent` override moves the new broker id onto the order, so a sweep can find a + replacement. +- A mock API client deriving from the real one. The read's source is a settable property (Schwab: the + order collection its bulk read returns; a per-id broker sets the get-order response), and an + `AutoResetEvent` fires when the read runs, so a test waits for the next sweep instead of sleeping. + +A test stages broker time by swapping that property between waits: set the working snapshot, wait for +the events, set the half-filled snapshot, wait, set the filled one. One config line per test shortens +the poll interval (Schwab sets `charles-schwab-order-poll-interval-ms` to 100 ms), and the fixture's +setup puts it back, so no test changes the pace of the next one. + +### The mock data is recorded, not invented + +Every snapshot and stream message in the offline tests is a copy of something the broker really sent. +That sets the build order for a new plugin: one with no `TestData/` folder and no captured payloads +cannot start with the offline tests, because there is nothing to feed them. It starts with one live +test — write it, and run it by hand against a real account. The debug log of that run holds the +broker's real JSON responses and raw socket messages; copying them out is what creates the mock data, +and the offline tests come second, replaying it. Getting a payload is a manual, one-time step: + +- REST order JSON comes from one live run: run the plugin (or one of the live tests below) with debug + logging and copy the orders-endpoint response from the log, or save it to a `TestData/*.json` file + (Schwab keeps `REST_GET_OPEN_ORDERS.json`). Paste it with its real ids, and write a comment naming + the recorded order and its story — "order 1002449136610: two option legs, ended EXPIRED". +- Raw socket messages are copied from the same live log (Schwab's sits at `bin/Debug/log.txt` after one + Explicit run) and fed through the mock socket, so the wrapper handles them exactly like live bytes. +- A staged sequence interpolates only what the stage changes into one recorded skeleton — Schwab's + `BuildRigMarketOrderJson(status, filledQuantity, executionActivities)` replays a real market order's + five executions one sweep at a time. + +A note for the AI model driving an adoption: the log only records what the code writes to it. Before +the capture run, go through every request path the polling touches — the order read, place, replace, +cancel, and the socket's message handling — and check that each one traces the broker's raw response +(`Log.Trace`). Where a path is silent, add the trace and warn the developer which paths were silent: +the capture run only records payloads on a build that has the logging. + +Short payloads stay inline in the test as verbatim strings; only large or shared ones go to +`TestData/*.json`. + +### Offline tests + +These run without credentials, through the real order methods of the mock brokerage, and no request +leaves the test. What they must show, per plugin — the list is the checklist for the next adopter: + +- **Place while polling**: the sweep reports the submit; a recorded terminal snapshot reports the close + once; a rejected snapshot reports `Invalid` carrying the broker's words; the fills of one sweep + arrive as one event; a staged sequence reports every partial fill; a working order without fill data + does not stop the sweep. A combo plugin adds the leg id assignment from the snapshot by symbol. +- **Stream-to-polling handover**, for a hybrid plugin: the stream reports part of a fill and the seeded + poll reports only the rest; the same fill split differently by the two paths stays consistent; a leg + the stream already closed is not repeated by the poll. +- **Event ordering**: a fill arriving within the first sweep still reports `Submitted` before `Filled`, + and `UpdateSubmitted` before `Filled` after a replace — the proof the message-handler lock holds. +- **The whole lifecycle**: place, update, cancel while polling, and the same on the stream, so the two + paths can be compared event for event. +- **The plugin's own edges**: Schwab adds its sweep window (a read reaches back to the oldest open + order) and its closed-socket subscription guards; Tradier adds its cross-zero legs. + +### Live tests + +The live half is `[Explicit]`, run by hand during market hours against a real account, with the cost +spelled out in the reason string — "takes the single streamer connection", "places real orders", "buys +real shares" — so nobody runs it by accident. Schwab's `WebSocketVsPolling` category runs two brokerage +connections side by side: the first loses the stream and falls back to polling, the second keeps +streaming, and every test asserts the same lifecycle on both. What proved worth copying: + +- Place-update-cancel first: the full lifecycle on a limit order resting far from the market costs + nothing and checks the submit, the update submit and the cancel on both connections. +- Fills on a cheap liquid stock next: market and limit orders sized to a few dollars, run until + `Filled`. +- Account hygiene inside the test: a flat-account pre-check that sells leftovers before asserting, a + sell-back after every real buy — seeding the holdings first, so the sell maps to the right + instruction — and `cleanup:` log lines separating the hygiene from the behavior under test. +- Partial-fill hunting is its own test and it logs instead of asserting: a bid resting inside the + spread for the whole balance may catch a partial fill, but nothing can force one, so a hard assert + would only make the test flaky. The developer adjusts the price to the live quote before running. +- One live run doubles as the recorder: its debug log is where the offline tests' REST and socket + payloads come from. ## Alternatives not taken @@ -1045,6 +1193,78 @@ inline `Task.Delay` block with `Watch` on the unknown ids. new id it already holds. - **Leave it in the plugins.** It is already written three times, and the fourth copy would be IB's. +### A base brokerage class that owns the wiring + +Every adopter wires the service the same way: create the mode class with the read callback, forward +`OrderEvents` to `OnOrderEvents` and `Message` to `OnMessage`, decide what `OrderNotAcknowledged` means, and +dispose the service with the brokerage. So the tempting next step is to move that wiring into an inheritance +layer — one abstract brokerage class that owns the service and asks the plugin only for overrides: + +```csharp +public abstract class BaseWebSocketsAndPollingServiceBrokerage : BaseWebsocketsBrokerage +{ + // owns the service, forwards its events, disposes it with the brokerage + protected abstract IEnumerable ReadOrderStates(); // or a per-id read + protected virtual BrokerOrderState ToSeedState(Order order) => null; +} +``` + +The attraction is real: a developer who picks the base class is told by the compiler what to implement, +instead of reading this document. Considered and not taken, on three facts. + +**The class cannot reach the plugins that need it.** A C# class inherits one class, so a layer under +`BaseWebsocketsBrokerage` serves only the plugins that already sit there — and most do not: + +| Plugin | Base class today | Order updates today | Could inherit it | +| --- | --- | --- | --- | +| CharlesSchwab | `BaseWebsocketsBrokerage` | account activity stream, this service as the fallback | yes | +| Tradier | `BaseWebsocketsBrokerage` | its own inline poll | yes | +| Binance (+ US and futures variants) | `BaseWebsocketsBrokerage` | user data stream | only through `BinanceBrokerage` — the variants subclass it | +| ByBit | `BaseWebsocketsBrokerage` | user data stream | yes, as a fallback only | +| dYdX | `BaseWebsocketsBrokerage` | subaccounts channel | yes, as a fallback only | +| Eze | `BaseWebsocketsBrokerage` | websocket protobuf push | yes, as a fallback only | +| Public.com | `Brokerage` | none — this service is the order path | **no** | +| InteractiveBrokers | `Brokerage`, `sealed` | vendor TCP callback SDK | **no** | +| TradeStation | `Brokerage` | HTTP stream, not a websocket | **no** | +| Alpaca | `Brokerage` | vendor SDK streaming client | **no** | +| Tastytrade | `Brokerage` | its own two-socket wrapper | no — a base class migration first | +| WeBull | `Brokerage` | its own websocket order events | no — a base class migration first | +| IG | `Brokerage` | Lightstreamer vendor SDK | no | +| OANDA | `Brokerage`, through its own `OandaRestApiBase` | HTTP transaction stream | no — its own hierarchy holds the slot | +| TerminalLink | `Brokerage` | Bloomberg session API | no | +| TradingTechnologies, Fix.Bloomberg, Fix.InteractiveBrokers | `Brokerage` / `FixBrokerage` | FIX execution reports | no | + +Only the first six rows sit on `BaseWebsocketsBrokerage`, and only two of those poll. The two plugins this +document opens with as the ones that need polling most — Public.com, with no order stream at all +(`PublicBrokerage.cs:42` extends plain `Brokerage`), and InteractiveBrokers, whose lost reply ends the run +(`InteractiveBrokersBrokerage.cs:68`, a `sealed` class on the vendor SDK) — are exactly the two the class +can never serve. + +**Serving both sides means writing the class twice.** The only way around the table is a second abstract +class with the same body under `Brokerage`, next to the websocket one. The bodies cannot be shared: a class +inherits one class, and a default interface method can neither hold the service field nor call the protected +`OnOrderEvents`. That is the same wiring twice in core, edited in pairs forever, and a fix applied to one +copy and not the other quietly makes the two halves behave differently. + +**The overrides are not where the adoption work is.** Counted on the two adopters: the wiring a base class +could absorb is about 30 lines in Schwab and about 45 in Public — the creation, three event forwards, one +dispose call. What it cannot absorb is everything else the plugin writes: the read and the mapping (~200 +lines in Schwab, ~45 in Public), the `Watch` calls inside the order methods, and the start trigger, which is +broker policy — Public starts polling in `Connect` because the service is its connection, Schwab starts it +when the stream is taken away. The class cannot even own the stop: `Disconnect` is abstract on `Brokerage` +(`Brokerages/Brokerage.cs:149`), so `Stop` stays a line the plugin writes either way. An abstract read would +guard the one step nobody gets wrong, and guard it for a third of the plugins. + +The place that reaches every row of the table is the root `Brokerage` class, the way the cross-zero +helpers and `CreateOAuthTokenHandler` already sit there as protected members only some plugins use +(`Brokerages/Brokerage.cs:604`, `:342`): one protected creation helper per mode, whose read-callback +signature picks the class, the service as a protected property, and a `Dispose` that covers it. Every +brokerage derives from `Brokerage`, the sealed IB class included, and constructing the service directly +stays possible — the helper is additive. Implemented that way on 2026-08-17: the seam is the two +`CreateOrderPollingService` overloads, `OrderPollingService`, `IsOrderPolling` and the virtual +`OnOrderPollingNotAcknowledged`, and Schwab, Public.com and Tradier all create their service through it +(see "Wiring, per plugin"). The abstract class this section declines stays declined. + ## Risks | Risk | What we do about it | @@ -1054,7 +1274,7 @@ inline `Task.Delay` block with `Watch` on the unknown ids. | Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier ships this trade-off today (`TradierBrokerage.cs:1469-1472`); a shorter poll interval narrows it. | | Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order is unacknowledged, and the interval is a constructor argument the plugin picks. | | A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | -| A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; IB and Tradier do not, and adding one is named in their rollout steps. | +| A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; Tradier adopted with null — its submit event goes out before the watch begins, so the poll cannot outrun it — and IB's rollout step still names adding one. | | The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | | The watch timeout cannot tell "never arrived" from "filled instantly" | It does not try. `OrderNotAcknowledged` hands the question to the brokerage, which has the endpoints to answer it. | | Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | @@ -1062,17 +1282,21 @@ inline `Task.Delay` block with `Watch` on the unknown ids. ## Rollout -1. This PR: the non-generic multi-source message handler, the service, the snapshot, and their unit - tests, no brokerage changes. The generic handler is untouched, so every plugin compiles as before. +1. This PR: the non-generic multi-source message handler, the service, the snapshot, their unit tests, + and the protected create seam on `Brokerage`. The generic handler is untouched, so every plugin + compiles as before, and no plugin is forced onto the seam. 2. InteractiveBrokers: add the message handler it does not have, then replace the `NoBrokerageResponse` error and the invented `Submitted` with a watch. This is the proof that the abstraction holds for a plugin that did not write it. -3. CharlesSchwab and Public.com: delete their service class and their fill/close diff. What stays is real and - named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and its - by-symbol leg id assignment. One behavior change is intentional: a Public poll that shows a +3. CharlesSchwab and Public.com (done): delete their service class and their fill/close diff. What stays is real + and named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and + its by-symbol leg id assignment. One behavior change is intentional: a Public poll that shows a new fill and the cancel together now emits both — the old code dropped the cancel. Schwab's per-execution prices become per-sweep prices while the quantities stay exact; Public kept its change-of-average price recovery inside its mapping, so its part prices stay exact too. -4. Tradier: replace the inline re-check block with a watch. Tradier splits orders across zero, so one brokerage - order can cover only part of the Lean quantity — its watch resolves submissions only, and fills stay on its - existing path. +4. Tradier (done, further than planned here): the step was a watch for submissions with fills staying on its own + path. The adoption moved the fill path itself onto `PerOrderIdPollingService` and handles the cross-zero split + as "A cross-zero order, two ids" describes. Two costs are accepted: a sweep is one gated request per watched + order instead of one bulk request — the one-order-per-symbol rule keeps that count small, and an idle account + now polls nothing at all — and orders placed outside Lean are ignored, where the old code raised a fatal + "UnknownOrderId" error. From 92b87137ef1314d035f25bc6ece4ade945eb25b3 Mon Sep 17 00:00:00 2001 From: Romazes Date: Mon, 17 Aug 2026 22:17:59 +0300 Subject: [PATCH 13/25] docs: add the status write-back rule to the adr testing section - fixtures play the transaction handler: write each reported status back onto the lean order and assert in live cycle order --- Documentation/ADR/0001-brokerage-order-polling-service.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 98f94d1c55b4..d861f6afd9f8 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -1063,6 +1063,14 @@ A test stages broker time by swapping that property between waits: set the worki the events, set the half-filled snapshot, wait, set the filled one. One config line per test shortens the poll interval (Schwab sets `charles-schwab-order-poll-interval-ms` to 100 ms), and the fixture's setup puts it back, so no test changes the pace of the next one. + +And the test plays the transaction handler: every fixture subscribes to `OrdersStatusChanged` and +writes each reported status back onto the Lean order before anything else (Schwab's `ApplyOrderStatus` +helper) — the same closing of the loop Lean does live. The diff decides from the Lean order: the +submit goes out only while the order is still `New`, and an id leaves the registry only once its order +closed. A fixture that skips the write-back gets duplicate submits and orders that never stop being +read. The asserts then follow the live trading cycle, on the collected events in the order it produces +them: the submit, the fills, the close. ### The mock data is recorded, not invented From f6d4cb506b67f1acb660e339cad29bfa139b0697 Mon Sep 17 00:00:00 2001 From: Romazes Date: Tue, 18 Aug 2026 16:52:56 +0300 Subject: [PATCH 14/25] refactor: fold the seeded start into Start and add state constructors - rename SeedAndStart(seed) to the Start(preLoadOpenOrders) overload - add BrokerOrderState constructors, one overload takes the message without fills - trace the wired service with its intervals and the pre-load counts --- Brokerages/Brokerage.cs | 6 ++- Brokerages/Services/BrokerOrderState.cs | 41 +++++++++++++++++++ .../Services/BrokerageOrderPollingService.cs | 28 ++++++++----- .../BrokerageOrderPollingServiceTests.cs | 33 ++++++--------- 4 files changed, 77 insertions(+), 31 deletions(-) diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index 1f7cb08cb6a0..87fb79f3401a 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -352,7 +352,8 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// /// The order polling service one of the create overloads built and wired, null until then. The brokerage - /// calls Watch, WatchReplacement, Start, Stop and SeedAndStart on it at its own lifecycle points; + /// calls Watch, WatchReplacement, Start - plain or with its pre-load callback - and Stop on it at + /// its own lifecycle points; /// disposes it. /// protected BrokerageOrderPollingService OrderPollingService { get; private set; } @@ -424,6 +425,9 @@ private void WireOrderPollingService(BrokerageOrderPollingService service) service.Message += (_, message) => OnMessage(message); service.OrderNotAcknowledged += (_, notAcknowledged) => OnOrderPollingNotAcknowledged(notAcknowledged); OrderPollingService = service; + + Log.Trace($"Brokerage.WireOrderPollingService(): {Name} created a {service.GetType().Name}: " + + $"poll interval {service.PollInterval.TotalMilliseconds}ms, watch timeout {service.WatchTimeout.TotalSeconds}s."); } /// diff --git a/Brokerages/Services/BrokerOrderState.cs b/Brokerages/Services/BrokerOrderState.cs index de3136515d8b..cf54beb16672 100644 --- a/Brokerages/Services/BrokerOrderState.cs +++ b/Brokerages/Services/BrokerOrderState.cs @@ -60,5 +60,46 @@ public class BrokerOrderState /// The broker's own words for a closing status, e.g. the reject reason. /// public string Message { get; set; } + + /// + /// Creates an empty state the caller fills through the properties. + /// + public BrokerOrderState() + { + } + + /// + /// Creates a state with no fill numbers: the id, the status, the time and the broker's words + /// for a closing status. + /// + /// The brokerage order id. + /// The Lean status the brokerage maps its broker's own status to. + /// When the brokerage reported this state, in UTC. + /// The broker's own words for a closing status. + public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, string message) + : this(brokerageOrderId, status, timeUtc, filledQuantity: null, fillPrice: null, message: message) + { + } + + /// + /// Creates a state from what one read saw. Only the first three are always known; the fill + /// numbers and the message stay null when the read does not carry them. + /// + /// The brokerage order id. + /// The Lean status the brokerage maps its broker's own status to. + /// When the brokerage reported this state, in UTC. + /// The total absolute quantity filled so far. + /// The price the broker reports for the fills. + /// The broker's own words for a closing status. + public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, + decimal? filledQuantity = null, decimal? fillPrice = null, string message = null) + { + BrokerageOrderId = brokerageOrderId; + Status = status; + TimeUtc = timeUtc; + FilledQuantity = filledQuantity; + FillPrice = fillPrice; + Message = message; + } } } diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/BrokerageOrderPollingService.cs index 0709a8b78f1c..78885b3f71b4 100644 --- a/Brokerages/Services/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/BrokerageOrderPollingService.cs @@ -495,16 +495,16 @@ public void ProcessOrderState(BrokerOrderState orderState) /// /// The whole handover from a stream to polling, in the only safe order: process what the stream - /// already delivered, seed the registry with one per - /// open Lean order, then . Does nothing while polling already runs. + /// already delivered, pre-load the registry with one + /// per open Lean order, then start the loop. Does nothing while polling already runs. /// /// - /// The stream reported 100 of 233 shares, the seed carries 100, so the first sweep reports only the other 133. + /// The stream reported 100 of 233 shares, the pre-load carries 100, so the first sweep reports only the other 133. /// - /// Builds the seed for one open Lean order: the brokerage id, the order's status - /// and the cumulative filled quantity already reported. A null return skips the order, and a null - /// callback skips seeding. - public void SeedAndStart(Func seed = null) + /// Builds the state another path already reported for one open Lean + /// order: the brokerage id, the order's status and the cumulative filled quantity. A null return + /// skips the order, and a null callback pre-loads nothing. + public void Start(Func preLoadOpenOrders) { if (IsPolling) { @@ -512,19 +512,27 @@ public void SeedAndStart(Func seed = null) } // an empty locked block waits for any order request in flight and processes the stream messages - // it buffered, so the seeds below count every fill the stream delivered + // it buffered, so the pre-load below counts every fill the stream delivered _messageHandler?.WithLockedStream(() => { }); - if (seed != null) + if (preLoadOpenOrders != null) { + Log.Trace($"{GetType().Name}.{nameof(Start)}(): pre-loading the open orders."); + + var openOrderCount = 0; + var preLoadedCount = 0; foreach (var openLeanOrder in _orderProvider?.GetOpenOrders() ?? []) { - var lastSeen = seed(openLeanOrder); + openOrderCount++; + var lastSeen = preLoadOpenOrders(openLeanOrder); if (lastSeen != null && !string.IsNullOrEmpty(lastSeen.BrokerageOrderId)) { Watch(lastSeen.BrokerageOrderId, lastSeen); + preLoadedCount++; } } + + Log.Trace($"{GetType().Name}.{nameof(Start)}(): pre-loaded {preLoadedCount} of {openOrderCount} open order(s)."); } Start(); diff --git a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs index b143fc932ef5..d444ce831617 100644 --- a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs @@ -60,15 +60,8 @@ private Order AddOrder(decimal quantity, string brokerageId, OrderStatus status private static BrokerOrderState State(string brokerageId, OrderStatus status, decimal? filled = null, decimal? price = null, string message = null) { - return new BrokerOrderState - { - BrokerageOrderId = brokerageId, - Status = status, - FilledQuantity = filled, - FillPrice = price, - TimeUtc = new DateTime(2026, 8, 12, 14, 30, 0, DateTimeKind.Utc), - Message = message - }; + return new BrokerOrderState(brokerageId, status, new DateTime(2026, 8, 12, 14, 30, 0, DateTimeKind.Utc), + filledQuantity: filled, fillPrice: price, message: message); } [Test] @@ -638,12 +631,12 @@ public void StartAndStopAreIdempotentAndDisposeIsFinal() } [Test] - public void SeedAndStartSeedsThenStarts() + public void StartWithPreLoadSeedsTheRegistryThenStarts() { AddOrder(233m, "42", OrderStatus.Submitted); var seeded = new List(); - _service.SeedAndStart(order => + _service.Start(order => { seeded.Add(order.BrokerId[0]); // the stream already reported 100 of the 233 shares @@ -661,7 +654,7 @@ public void SeedAndStartSeedsThenStarts() } [Test] - public void SeedAndStartDrainsTheMessageHandlerBeforeSeeding() + public void StartWithPreLoadDrainsTheMessageHandlerBeforeSeeding() { AddOrder(233m, "42", OrderStatus.Submitted); var steps = new List(); @@ -689,7 +682,7 @@ public void SeedAndStartDrainsTheMessageHandlerBeforeSeeding() // the stream delivered a fill that is still buffered behind the order request handler.HandleNewMessage(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); - var seedAndStart = Task.Run(() => service.SeedAndStart(openOrder => + var seededStart = Task.Run(() => service.Start(openOrder => { lock (steps) { @@ -699,10 +692,10 @@ public void SeedAndStartDrainsTheMessageHandlerBeforeSeeding() })); // the handover waits for the order request instead of seeding early - Assert.IsFalse(seedAndStart.Wait(TimeSpan.FromMilliseconds(100))); + Assert.IsFalse(seededStart.Wait(TimeSpan.FromMilliseconds(100))); releaseEvent.Set(); - Task.WaitAll(orderRequest, seedAndStart); + Task.WaitAll(orderRequest, seededStart); Assert.IsTrue(service.IsPolling); lock (steps) @@ -713,22 +706,22 @@ public void SeedAndStartDrainsTheMessageHandlerBeforeSeeding() } [Test] - public void SeedAndStartWithoutSeedJustStarts() + public void StartWithNullPreLoadJustStarts() { - _service.SeedAndStart(); + _service.Start(preLoadOpenOrders: null); Assert.IsTrue(_service.IsPolling); Assert.IsEmpty(_orderEvents); } [Test] - public void SeedAndStartWhilePollingDoesNothing() + public void StartWithPreLoadWhilePollingDoesNothing() { AddOrder(100m, "42"); _service.Start(); var callCount = 0; - _service.SeedAndStart(_ => { callCount++; return null; }); + _service.Start(_ => { callCount++; return null; }); Assert.IsTrue(_service.IsPolling); Assert.AreEqual(0, callCount); @@ -740,7 +733,7 @@ public void SeedWithoutBrokerageIdIsSkipped() AddOrder(100m, "42"); AddOrder(100m, "43"); - _service.SeedAndStart(seed: order => order.BrokerId[0] == "42" + _service.Start(preLoadOpenOrders: order => order.BrokerId[0] == "42" ? null : new BrokerOrderState { Status = OrderStatus.Submitted }); From e274b06ab3f376d2eb037fd3c8ca6174f9fc6b5a Mon Sep 17 00:00:00 2001 From: Romazes Date: Tue, 18 Aug 2026 16:52:56 +0300 Subject: [PATCH 15/25] docs: refresh the adr for the adoptions, testing run and api polish - record public.com's live-first testing run and the offline checklist additions - tradier reads per id and has draft pr; stale refs and tenses fixed - the pre-loading start, the state constructors and the price-recovery note --- .../0001-brokerage-order-polling-service.md | 178 +++++++++++------- 1 file changed, 115 insertions(+), 63 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index d861f6afd9f8..cb933206d2fd 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -8,7 +8,7 @@ Pilot implemented - 2026-08-13: CharlesSchwab is the first plugin on the service (`Lean.Brokerages.CharlesSchwab`, draft PR #107). Its own `OrderUpdatePollingService` and diff are deleted; what remains in the plugin is what this document says remains - the read with its sweep window, the model-to-state mapping, the leg id assignment shared with the stream's `OrderAccepted`, and the seeded -stream-to-polling handover through `SeedAndStart`. +stream-to-polling handover through the pre-loading `Start` overload. Replacement watch implemented - 2026-08-14: a polled replace goes through `WatchReplacement` and the diff reports the update submit. The plugin's own replace reporting and its by-position leg id derivation are @@ -34,7 +34,7 @@ CharlesSchwab and Public.com were moved onto it: the creation, the three event f call left both plugins (see "Wiring, per plugin"). Third plugin adopted - 2026-08-17: Tradier runs on `PerOrderIdPollingService` -(`Lean.Brokerages.Tradier`, branch `feature-core-order-polling-service`). Its fill timer, `CheckForFills` +(`Lean.Brokerages.Tradier`, draft PR #54). Its fill timer, `CheckForFills` diff, order cache and unknown-id verification are deleted (~230 lines); the plugin keeps the get-order read and the mapping. Tradier is the first adopter that splits orders across zero: the legs chain from the read through the base cross-zero helpers, and the second leg's watch seed carries what the first leg @@ -42,6 +42,25 @@ filled (see "A cross-zero order, two ids"). Two behavior changes are intentional Lean are ignored - a per-id read never sees them, where the old code raised a fatal "UnknownOrderId" error - and the fee attaches once per Lean order instead of once per broker leg. +Testing process run end to end - 2026-08-17: Public.com is the first plugin tested by this document's own +order - live capture first, offline replay second (`PublicBrokerageOrderPollingTests.cs`, in draft PR #6). +No recorded payloads existed, so two Explicit live tests ran by hand with debug logging on, and three +capture runs recorded the get-order bodies: an equity place-update-cancel and a market fill, an option +update whose fill beat the cancel, and a multi-leg cancel. Eleven offline tests replay those bodies. The +runs confirmed the per-id mode's edges on a real account: a market order filled on its first read reports +the submit and the fill in one batch, a `PENDING_CANCEL` read between the request and the cancel reports +nothing, and one canceled shared-id combo reports one `Canceled` per leg. + +Seeded start folded into Start - 2026-08-18: `SeedAndStart(seed)` became the `Start(preLoadOpenOrders)` +overload, so starting is one method with two shapes: the plain `Start()` resumes the loop, and the overload +runs the handover first. Behavior unchanged; Schwab's call site renamed with it. + +State constructors and wiring traces - 2026-08-18: `BrokerOrderState` gained two constructors - the +always-known facts positionally, and an overload taking the message without the fill numbers - and all +three adopters build through them. The wiring now traces the created mode class with its intervals, and +the pre-loading `Start` traces how many open orders it pre-loaded. Schwab moved its mapping into +`CharlesSchwabExtensions.ToLegOrderStates`, called as `brokerageOrder.ToLegOrderStates()` from the read. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -77,30 +96,32 @@ own pull request. | Public.com | `OrderPollingService.cs` | 268 lines | per order id | `OrderPollingInterval` ctor argument | `_messageHandler.HandleNewMessage` | | Tradier | inline, `TradierBrokerage.cs:1240-1284` | ~45 lines | per order id, one shot | `Task.Delay(2s)` | direct | -The two service classes are near copies outside their two callbacks. Both have: a background task, a -`CancellationTokenSource` recreated by `Start` and cleared by `Stop`, an idempotent `Start`, a `Stop` that cancels and -waits up to 2 seconds before disposing the source, a `Dispose` that refuses to start again, a loop that logs and -retries on a failed read instead of dying, and the same two trace lines. Even the comments match, because the second +All three copies are deleted by their adoptions; the table records what stood before. The two service +classes were near copies outside their two callbacks. Both had: a background task, a +`CancellationTokenSource` recreated by `Start` and cleared by `Stop`, an idempotent `Start`, a `Stop` that canceled and +waited up to 2 seconds before disposing the source, a `Dispose` that refused to start again, a loop that logged and +retried on a failed read instead of dying, and the same two trace lines. Even the comments matched, because the second one was written from the first. -What actually differs is small and none of it is a design decision worth keeping twice: `Task.Run` against +What actually differed was small and none of it was a design decision worth keeping twice: `Task.Run` against `Task.Factory.StartNew(LongRunning)`, an async fetch against a sync one, `Task.Delay` against -`cancellationToken.WaitHandle.WaitOne`, and a failure counter that only Schwab has. Public has one real extra: a +`cancellationToken.WaitHandle.WaitOne`, and a failure counter that only Schwab had. Public had one real extra: a registry of watched brokerage ids with the last state seen for each (`Models/OrderSnapshot.cs`: status, cumulative filled quantity, average price). -Tradier's version is the same idea again in miniature. When a fill arrives for a brokerage id Lean does not know, it -waits 2 seconds, re-checks `_orderProvider.GetOrdersByBrokerageId`, and re-requests the orders from the API -(`TradierBrokerage.cs:1240-1284`). +Tradier's version was the same idea again in miniature. When a fill arrived for a brokerage id Lean did not know, it +waited 2 seconds, re-checked `_orderProvider.GetOrdersByBrokerageId`, and re-requested the orders from the API +(`TradierBrokerage.cs:1240-1284` before its adoption deleted the path). ### 2. Two brokerages block the order thread for minutes, then kill the run Both CharlesSchwab and InteractiveBrokers place the order, then **block inside the order method** waiting for the -broker to confirm it on the real-time channel. Neither of them ever asks the broker over HTTP instead. +broker to confirm it on the real-time channel. Neither of them ever asked the broker over HTTP instead — +this service is that ask, and Schwab's polled mode now answers the same wait. | Brokerage | Waits for | How long | Where the number comes from | When it expires | | --- | --- | --- | --- | --- | -| CharlesSchwab | `OrderAccepted` on the account activity stream | **3 minutes** | hardcoded `TimeSpan.FromMinutes(3)`, `CharlesSchwabBrokerage.cs:478` | `Error` `MissingWebSocketResponse` (`:480`) | +| CharlesSchwab | `OrderAccepted` on the account activity stream | **3 minutes** | hardcoded `TimeSpan.FromMinutes(3)`, `CharlesSchwabBrokerage.cs:486` | `Error` `MissingWebSocketResponse` (`:488`) | | InteractiveBrokers | `openOrder` / `orderStatus` / `execDetails` callback | **5 minutes** | `ib-response-timeout`, default `300` seconds, `InteractiveBrokersBrokerage.cs:84` | `Error` `NoBrokerageResponse` (`:1659`) | | InteractiveBrokers, `MarketOnOpen` / `ComboLegLimit` / `ComboMarket` / `ComboLimit` | same | 10 seconds | `ib-no-submission-orders-response-timeout`, `:90` | Lean **invents** a `Submitted` event (`:1649-1652`) | @@ -200,8 +221,8 @@ away**, because a Lean `Order` has nowhere to put it. All that survives is the w So the plugins already have these numbers. There is just no field to store them in. That is the whole fix: instead of the service reading Lean `Order` objects, **the brokerage converts its own order model into one small shared snapshot and passes that to the service.** The snapshot has fields for the -status, the total filled quantity and the fill price — the three numbers Public's poller already tracks -per order (`Models/OrderSnapshot.cs`) — so one compare in Lean core works for every brokerage and can report real +status, the total filled quantity and the fill price — the three numbers Public's replaced poller tracked +per order (its since-deleted `Models/OrderSnapshot.cs`) — so one compare in Lean core works for every brokerage and can report real fills, not only that the order exists. Two rules shape what the service does with a snapshot: @@ -230,8 +251,8 @@ namespace `QuantConnect.Brokerages.Services`. `Services` is a new folder under `Brokerages`, and it follows the convention the other subfolders there already use: `Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. It also matches where -the two plugins keep this class today — CharlesSchwab has it in `QuantConnect.CharlesSchwabBrokerage/Services/`, so -the move up to core keeps the same path. +the two plugins kept this class before the move — CharlesSchwab had it in `QuantConnect.CharlesSchwabBrokerage/Services/`, so +the move up to core kept the same path. Tests go to `Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs`. @@ -259,10 +280,9 @@ generic: nothing broker-specific ever enters it, because the brokerage translate before the service looks at it. And because the service is the one calling the read, a read that throws is caught and counted by the service itself — the repeated-failure warning below works without any extra code in the plugin. -Both existing pollers already have exactly this flow. Schwab's loop hands each polled order to -`_messageHandler.HandleNewMessage` and the diff runs when the handler dequeues it -(`CharlesSchwabBrokerage.OrderUpdatePolling.cs`), and Public wires its poller the same way -(`PublicBrokerage.cs:182`). +Both replaced pollers had exactly this flow: Schwab's loop handed each polled order to +`_messageHandler.HandleNewMessage` and the diff ran when the handler dequeued it, and Public wired its +poller the same way. The service's own loop does the enqueue now. ### The broker order state @@ -294,6 +314,13 @@ public class BrokerOrderState /// The broker's own words for a closing status, e.g. the reject reason. public string Message { get; set; } + + /// The always-known facts positionally - id, status, time - with the fill numbers and the + /// message defaulting to null. An empty constructor stays for filling through the properties, and an + /// overload takes the message without the fill numbers. + public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, + decimal? filledQuantity = null, decimal? fillPrice = null, string message = null); + public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, string message); } ``` @@ -313,7 +340,7 @@ instead of carrying a fee field nobody fills. Combos come in two shapes, and the state carries both without any extra field. Schwab gives every leg its own brokerage id (`mainId + legId - 1`), so the plugin passes one state per leg. Public.com gives the whole combo -**one** id (`PublicBrokerage.Brokerage.cs:328`), so the plugin passes one state and the service fans it out: +**one** id (`PublicBrokerage.Brokerage.cs:430-441`), so the plugin passes one state and the service fans it out: `GetOrdersByBrokerageId` returns every Lean leg order behind the id, and each leg's share of a new fill is ``` @@ -322,8 +349,8 @@ legFill = leanOrder.Quantity * newPart / abs(leanOrder.GroupOrderManager.Quantit The state has no quantity field because the service does not need one: it already knows the brokerage id, so it reads the group quantity from the Lean orders themselves. That number equals the broker's own order quantity — the -group quantity is exactly what the combo was placed with. Public's diff already splits fills this way today -(`PublicBrokerage.Brokerage.cs:700-711`). One rule for the mapping follows: +group quantity is exactly what the combo was placed with. Public's replaced diff split fills this way, and +the rule moved into the service's fan-out (`Brokerages/Services/BrokerageOrderPollingService.cs`). One rule for the mapping follows: `FilledQuantity` of a shared-id combo is in strategy units, the same units the group quantity counts in. A worked example, from a real Public.com order — 5 AAPL strangles, one brokerage id, a put leg and a call leg with @@ -476,9 +503,9 @@ public abstract class BrokerageOrderPollingService : IDisposable public void ProcessOrderState(BrokerOrderState orderState); /// The whole handover from a stream to polling, in the only safe order: process what the - /// stream already delivered, seed one watch per open Lean order, then Start. The seed callback is - /// optional. See "Seed before Start". - public void SeedAndStart(Func seed = null); + /// stream already delivered, pre-load one watch per open Lean order, then start the loop. A null + /// callback pre-loads nothing. See "Seed before Start". + public void Start(Func preLoadOpenOrders); public void Start(); public void Stop(); @@ -515,8 +542,9 @@ The watch registry, the compare with the last state seen and the seeding of orde The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `Sweep`: - **`PerOrderIdPollingService`** — `Func`: the sweep loops the watched ids and calls the - read once per id. Nothing watched, nothing requested. Public.com's own service has exactly this constructor - today — `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)`. + read once per id. Nothing watched, nothing requested. Public.com's replaced service had exactly this + constructor — `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)` — + and its adoption passes the same read to `CreateOrderPollingService`. - **`AllOrdersPollingService`** — `Func>`: the sweep calls the read once, and the read returns everything the broker lists. @@ -534,7 +562,7 @@ the watch timeout: the service still checks that every watched id shows up in th | CharlesSchwab | bulk | all | one request returns the whole account, and Schwab is the order path for the run | | Public.com | per id | watched | Public.com has a get-order endpoint and only cares about its own orders | | InteractiveBrokers | bulk | watched | IB has **no** per-order request — `reqOpenOrders` always returns everything | -| Tradier | bulk | watched | same as IB: the unknown ids are re-checked against a full read (`GetIntradayAndPendingOrders`, `TradierBrokerage.cs:1266`) | +| Tradier | per id | watched | adopted: `GetOrder` asks about one id, so a sweep reads only the watched orders and an idle account sends no request at all (the plan here said bulk; the per-order endpoint made per id the better fit) | | Webull | bulk | watched | its only order read is `GetOpenOrders` (`Api/ApiClient.cs:666`); order updates come over a gRPC stream, and the poll covers its drops | | TradeStation | bulk | watched | one `GetOrders` request returns the account's orders (`Api/TradeStationApiClient.cs:185`); the stream stays the first path | | Alpaca | per id | watched | the SDK has `GetOrderAsync` and the plugin already calls it (`AlpacaBrokerage.cs:546`), so a watch asks only about its own orders | @@ -661,13 +689,19 @@ identical to the first and be lost. Pricing is deliberately simple: the new part takes the state's `FillPrice`, as the broker reported it. When several fills land inside one sweep, the quantity is still exact and the price is the broker's reported price at sweep -time, not each fill's own — Tradier's poller ships exactly this trade-off today and documents it -(`TradierBrokerage.cs:1469-1472`). Public recovers the exact increment price from the change of the average; +time, not each fill's own — Tradier's replaced poller shipped exactly this trade-off, and its adoption +keeps it, documented in the mapping (`TradierBrokerage.cs:1169-1171`). Public recovers the exact increment price from the change of the average; the service refuses that arithmetic on purpose — it amplifies broker rounding and can even go negative on a tiny part, while the simple price needs no guard at all. Public's adoption kept the recovery anyway, but inside its own mapping: the state's `FillPrice` arrives already recovered, with a guard that falls back to the plain average when the new part is not positive. The service still only copies the price it is given. +The recovery also cannot move into the service today. The state has no average field, so the service never +sees the broker's raw average — the mapping turns it into a fill price first. Keeping it in Public costs one +small map (the previous read's cumulative and average) and one fallback guard only the plugin can judge. It +moves into the service the day a second average-only broker adopts — IB's `orderStatus` reports an average +fill price — as one additive change: a nullable `AveragePrice` on the state and the arithmetic in the service. + One more detail is load-bearing. **State outlives the terminal event.** Forgetting an order the moment its `Canceled` goes out re-reports every fill if the next sweep lands before Lean applies the event — Schwab's own ADR documents exactly this race. So state is dropped only when a compare sees the order closed **in Lean**, never at @@ -677,7 +711,7 @@ emission. The "none found" branch skips today, but it is also an opening. An id the order provider keeps not knowing is most likely an order the user placed outside Lean, in the broker's own app — and Lean already has a door for those: -`OnNewBrokerageOrderNotification` (`Brokerages/Brokerage.cs:256`). The transaction handler picks it up +`OnNewBrokerageOrderNotification` (`Brokerages/Brokerage.cs:257`). The transaction handler picks it up (`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:190`, `:1674`), asks the algorithm's brokerage message handler whether to accept the order, and adopts it with `AddOpenOrder`. TradeStation already raises it from its stream (`TradeStationBrokerage.cs:1088`); a poll can feed the same door. @@ -775,7 +809,7 @@ facts that matter to a poll: does the id change, where does the new id come from The rows in code: IB re-sends the same broker id (`InteractiveBrokersBrokerage.cs:1599,1626`), TradeStation PUTs to the existing id and never reads the reply's `OrderID` (`Api/TradeStationApiClient.cs:329-335`), -Tradier's PUT has no quantity parameter at all (`TradierBrokerage.cs:486-507`), Public replaces in place +Tradier's PUT has no quantity parameter at all (`TradierBrokerage.cs:475-496`), Public replaces in place with the id echoed back (`Api/ApiClient.cs:351-362`), Webull keeps the `client_order_id` that is the Lean `BrokerId` (`Api/ApiClient.cs:595-605`), ByBit amends futures under the same id (`Api/BybitTradeApiEndpoint.cs:94-100`) and cannot amend spot (`BybitBrokerage.Brokerage.cs:200-204`), and @@ -787,7 +821,7 @@ count continues on it — all six carry-over cells are verified in code, e.g. Tr itself can never come from a sweep: the state carries a status and two fill numbers, no price and no quantity, so a modified order polls exactly like an unmodified one. The rule for a same-id plugin is Tradier's and Public's, already shipping: report `UpdateSubmitted` right after the REST reply -(`TradierBrokerage.cs:970-971`, `PublicBrokerage.Brokerage.cs:687`) and leave the registry alone — Public +(`TradierBrokerage.cs:947-948`, `PublicBrokerage.Brokerage.cs:692`) and leave the registry alone — Public runs on this service today and its `UpdateOrder` makes no service call. The two that report from the stream instead (IB `InteractiveBrokersBrokerage.cs:2359,2380`, Webull `WebullBrokerage.Brokerage.cs:448-453`) simply have no update report while their channel is down; on adoption they move the report next to the REST @@ -852,34 +886,36 @@ own the pending leg: `UpdateOrderState`, so the sweep's own diff stays silent. - The second leg's watch seed carries what the first leg filled, and its reads add the same offset to the leg's own cumulative. The diff's frame is whole again: the leg's last fill reaches the Lean quantity and - reports `Filled`. + reports `Filled`. A second leg the order provider has not indexed yet resolves through the base + cross-zero map instead, and the hook reports its closing fill itself, marking it reported so the diff + stays silent. The service itself needs nothing new for this — the seed, `UpdateOrderState` and `TryGetLastOrderState` -were already there. What it costs the plugin is one map (the first leg's filled quantity, keyed by the -Lean order id between the second leg's request and its watch, then by the second leg's brokerage id for -the reads) and one closed-order hook in the read. +were already there. What it costs the plugin is two small maps holding the first leg's filled quantity — +one keyed by the Lean order id between the second leg's request and its watch, one keyed by the second +leg's brokerage id for the reads — and one closed-order hook in the read. ### Seed before Start Polling never starts first: the stream reported orders before it, and the registry must know what was already reported before the first sweep runs. So every `Start` that follows stream time begins with a handover, and the -service owns its order — `SeedAndStart(seed)` does nothing while polling already runs, and otherwise runs three +service owns its order — `Start(preLoadOpenOrders)` does nothing while polling already runs, and otherwise runs three steps: 1. Drain the message handler — the service runs an empty `WithLockedStream` block on the handler it was built with, so every fill the stream already delivered is counted. Nothing slips in after it, because the switch runs on the stream's own thread — the only thread that delivers stream messages. -2. One `seed(openLeanOrder)` call per open Lean order, each becoming a `Watch(id, lastSeen)`: the plugin returns - the brokerage id, the order's status and the cumulative filled quantity from its own bookkeeping. Orders the - stream already closed need no seed — the diff skips every order Lean has closed, and a null return skips the - order. -3. `Start`. - -The seed callback is optional: a plugin whose stream reported nothing passes no seed, and without a message -handler there is nothing to drain. The first sweep then continues from what the stream reported instead of repeating it. A -fallback-mode plugin makes this one call, because its stream never comes back — Schwab's `ToSeedState` is the -working example of the seed callback. A gap-mode plugin repeats the call on every drop, and `Stop`s when the -stream returns. +2. One `preLoadOpenOrders(openLeanOrder)` call per open Lean order, each becoming a `Watch(id, lastSeen)`: the + plugin returns the brokerage id, the order's status and the cumulative filled quantity from its own + bookkeeping. Orders the stream already closed need no seed — the diff skips every order Lean has closed, + and a null return skips the order. +3. Start the loop. + +A plugin with nothing to hand over — its stream reported nothing, or it has no stream — calls the plain +`Start()` instead, and a null callback pre-loads nothing. The first sweep then continues from what the +stream reported instead of repeating it. A fallback-mode plugin makes this one call, because its stream +never comes back — Schwab's `ToPreLoadState` is the working example of the callback. A gap-mode plugin repeats +the call on every drop, and `Stop`s when the stream returns. The seed source already exists in most streaming plugins, because they keep the same bookkeeping Schwab does — the cumulative quantity already reported, per Lean order: Webull's `_orderIdToPreviousCumulativeQuantity` @@ -901,7 +937,7 @@ does. `Dispose` is the one-way door. Three rules for a plugin that uses this: - **Seed before every `Start`.** While the socket was up the stream was reporting and the registry was not - listening. `SeedAndStart` hands over what was reported, in the right order — see "Seed before Start". + listening. The pre-loading `Start` overload hands over what was reported, in the right order — see "Seed before Start". - **Sweep once after the stream returns, before stopping.** The socket coming back does not replay what it missed, and no plugin re-reads orders on reconnect today. One last sweep closes the gap. - **Coming back is not always allowed.** Schwab must stay on polling for the rest of the run, because reconnecting @@ -974,9 +1010,10 @@ A per-id broker only changes the read — one call replaces Public.com's constru forwards and its dispose line: ```csharp -CreateOrderPollingService(ReadOrderState, _messageHandler, _orderProvider); +CreateOrderPollingService(ReadOrderState, _messageHandler, _orderProvider, pollInterval: OrderPollingInterval); // BrokerOrderState ReadOrderState(string brokerageId): get-order by id, a 404 maps to null. -// Nothing else passed: brokerage-order-poll-interval-ms decides the interval, default 3000 ms. +// Public keeps its own key, public-order-poll-interval-ms, default 1000 ms. A plugin that passes no +// interval gets the shared brokerage-order-poll-interval-ms default, 3000 ms. ``` What silence means is one override per plugin: `OnOrderPollingNotAcknowledged` defaults to a single @@ -1008,7 +1045,7 @@ The same two lines cover a dropped socket, in any streaming plugin: ```csharp // where the brokerage already handles the connection going down; see "Seed before Start" -OrderPollingService.SeedAndStart(ToSeedState); +OrderPollingService.Start(ToPreLoadState); // on reconnect: one last sweep for the gap, then hand the job back to the stream OrderPollingService.Stop(); @@ -1042,7 +1079,9 @@ again: it tests its read, its mapping and its wiring **through** the service. Th the everyday coverage; the live tests prove the whole path against the real broker — and for a plugin starting from nothing they come first, because their runs record the data the offline tests replay. The reference is the Schwab pilot's fixture, `Lean.Brokerages.CharlesSchwab/QuantConnect.CharlesSchwabBrokerage.Tests/CharlesSchwabBrokerageOrderUpdatePollingTests.cs`; -its mock classes sit next to it in `Tests/Models`. +its mock classes sit next to it in `Tests/Models`. For a polling-primary, per-id plugin the reference is +Public.com's fixture, `Lean.Brokerages.Public/QuantConnect.PublicBrokerage.Tests/PublicBrokerageOrderPollingTests.cs` — +the first one built in this section's order, live capture first. ### The test doubles @@ -1091,6 +1130,11 @@ and the offline tests come second, replaying it. Getting a payload is a manual, `BuildRigMarketOrderJson(status, filledQuantity, executionActivities)` replays a real market order's five executions one sweep at a time. +Copy the nulls too. A real Public.com order carries `"filledQuantity": null` until something fills, +`"legs": null` outside a combo, and `"limitPrice": null` on a market order; a builder that writes `"0"` or `[]` +instead feeds the code a shape the broker never sends, and the test passes on the wrong parse path. The +staged skeleton keeps every field exactly as recorded and changes only what the stage changes. + A note for the AI model driving an adoption: the log only records what the code writes to it. Before the capture run, go through every request path the polling touches — the order read, place, replace, cancel, and the socket's message handling — and check that each one traces the broker's raw response @@ -1109,9 +1153,15 @@ leaves the test. What they must show, per plugin — the list is the checklist f once; a rejected snapshot reports `Invalid` carrying the broker's words; the fills of one sweep arrive as one event; a staged sequence reports every partial fill; a working order without fill data does not stop the sweep. A combo plugin adds the leg id assignment from the snapshot by symbol. +- **Cancel and its races**: the cancel request stays quiet and the poll reports the `Canceled` once; an + intermediate `CancelPending` read reports nothing; a fill that beats the cancel ends the order `Filled` + with no `Canceled` at all; a canceled shared-id combo reports one `Canceled` per leg. +- **An id the broker does not know yet**: the read returns null (Public.com's get-order 404) and the + sweep asks again until the order appears — the submit is reported then, not before. - **Stream-to-polling handover**, for a hybrid plugin: the stream reports part of a fill and the seeded poll reports only the rest; the same fill split differently by the two paths stays consistent; a leg - the stream already closed is not repeated by the poll. + the stream already closed is not repeated by the poll. A polling-primary plugin proves the seed with + the orders adopted at startup: one adopted half-filled reports only the part that fills after. - **Event ordering**: a fill arriving within the first sweep still reports `Submitted` before `Filled`, and `UpdateSubmitted` before `Filled` after a replace — the proof the message-handler lock holds. - **The whole lifecycle**: place, update, cancel while polling, and the same on the stream, so the two @@ -1138,7 +1188,9 @@ streaming, and every test asserts the same lifecycle on both. What proved worth spread for the whole balance may catch a partial fill, but nothing can force one, so a hard assert would only make the test flaky. The developer adjusts the price to the live quote before running. - One live run doubles as the recorder: its debug log is where the offline tests' REST and socket - payloads come from. + payloads come from. A polling-primary plugin turns the logging on inside the test itself + (`Log.DebuggingEnabled = true`), so every hand run records — and every extra asset class is one more + capture run, the way Public.com recorded its option and its multi-leg bodies after the equity ones. ## Alternatives not taken @@ -1170,7 +1222,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth wrap every stream message in a new closure — one allocation per message on the hottest path a brokerage has. The non-generic handler keeps the message itself in the buffer and allocates only at registration. - **Put it on the `Brokerage` base class, driven by the engine, like the cash sync.** - `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:480`, called from + `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:577`, called from `Engine/TransactionHandlers/BrokerageTransactionHandler.cs:731`) is the existing shape for core-driven periodic work, and it was the obvious candidate. Rejected: cash sync runs once a day on a schedule core can decide, while the right poll interval here is a property of the broker's rate limits and of whether its stream is alive. It also @@ -1191,7 +1243,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth already raises `OnOrderIdChangedEvent` when a replace re-keys a Lean order, so the service could subscribe to it instead of offering a method. Rejected three times over. The event does not mean "replace": Lean core raises it on a plain place — the cross-zero flow assigns the second part's id through it - (`Brokerages/Brokerage.cs:842`) — and Binance assigns every initial id with it, so the service would + (`Brokerages/Brokerage.cs:939`) — and Binance assigns every initial id with it, so the service would report an update submit for a placement. The event does not carry the previous id, and the service cannot recover it: the transaction handler subscribes first and swaps `order.BrokerId` before a later subscriber runs (`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:1497`), so the old registry entries could @@ -1260,12 +1312,12 @@ dispose call. What it cannot absorb is everything else the plugin writes: the re lines in Schwab, ~45 in Public), the `Watch` calls inside the order methods, and the start trigger, which is broker policy — Public starts polling in `Connect` because the service is its connection, Schwab starts it when the stream is taken away. The class cannot even own the stop: `Disconnect` is abstract on `Brokerage` -(`Brokerages/Brokerage.cs:149`), so `Stop` stays a line the plugin writes either way. An abstract read would +(`Brokerages/Brokerage.cs:150`), so `Stop` stays a line the plugin writes either way. An abstract read would guard the one step nobody gets wrong, and guard it for a third of the plugins. The place that reaches every row of the table is the root `Brokerage` class, the way the cross-zero helpers and `CreateOAuthTokenHandler` already sit there as protected members only some plugins use -(`Brokerages/Brokerage.cs:604`, `:342`): one protected creation helper per mode, whose read-callback +(`Brokerages/Brokerage.cs:701`, `:343`): one protected creation helper per mode, whose read-callback signature picks the class, the service as a protected property, and a `Dispose` that covers it. Every brokerage derives from `Brokerage`, the sealed IB class included, and constructing the service directly stays possible — the helper is additive. Implemented that way on 2026-08-17: the seam is the two @@ -1279,7 +1331,7 @@ stays possible — the helper is additive. Implemented that way on 2026-08-17: t | --- | --- | | A plugin maps a broker status to the wrong Lean status | The mapping is the same one its streaming path already needs, written once per plugin and covered by its own tests. The service only emits transitions, so a wrong mapping surfaces once, not as a flood. | | A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the watch timeout still fires. | -| Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier ships this trade-off today (`TradierBrokerage.cs:1469-1472`); a shorter poll interval narrows it. | +| Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier kept this trade-off through its adoption (`TradierBrokerage.cs:1169-1171`); a shorter poll interval narrows it. | | Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order is unacknowledged, and the interval is a constructor argument the plugin picks. | | A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | | A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; Tradier adopted with null — its submit event goes out before the watch begins, so the poll cannot outrun it — and IB's rollout step still names adding one. | From 0839dbcc205774f8d519495dcc3c9e29c6316e72 Mon Sep 17 00:00:00 2001 From: Romazes Date: Tue, 18 Aug 2026 23:21:32 +0300 Subject: [PATCH 16/25] refactor: split service folders and rework the watch alarm - move the service to Brokerages/Services/OrderPolling with its shapes in Models - rename OrderNotAcknowledged to BrokerageOrderNeverNotified across the family - the args resolve the Lean order, carry WatchDuration and print via ToString - the default warning uses the args ToString; the message code string stays --- Brokerages/Brokerage.cs | 19 +++--- .../Services/OrderNotAcknowledgedEventArgs.cs | 48 -------------- .../AllOrdersPollingService.cs | 5 +- .../BrokerageOrderPollingService.cs | 35 +++++----- .../Models}/BrokerOrderState.cs | 2 +- .../BrokerageOrderNeverNotifiedEventArgs.cs | 65 +++++++++++++++++++ .../PerOrderIdPollingService.cs | 5 +- .../BrokerageOrderPollingServiceTests.cs | 35 +++++++--- 8 files changed, 128 insertions(+), 86 deletions(-) delete mode 100644 Brokerages/Services/OrderNotAcknowledgedEventArgs.cs rename Brokerages/Services/{ => OrderPolling}/AllOrdersPollingService.cs (91%) rename Brokerages/Services/{ => OrderPolling}/BrokerageOrderPollingService.cs (95%) rename Brokerages/Services/{ => OrderPolling/Models}/BrokerOrderState.cs (98%) create mode 100644 Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs rename Brokerages/Services/{ => OrderPolling}/PerOrderIdPollingService.cs (93%) rename Tests/Brokerages/Services/{ => OrderPolling}/BrokerageOrderPollingServiceTests.cs (95%) diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index 87fb79f3401a..0b5ba9379a05 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -29,7 +29,8 @@ using QuantConnect.Api; using QuantConnect.Brokerages.Authentication; using QuantConnect.Brokerages.CrossZero; -using QuantConnect.Brokerages.Services; +using QuantConnect.Brokerages.Services.OrderPolling; +using QuantConnect.Brokerages.Services.OrderPolling.Models; using QuantConnect.Util; namespace QuantConnect.Brokerages @@ -367,7 +368,7 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// Creates a , for a broker with a get-order endpoint, and wires it /// into this brokerage: polled order events reach , polling outage warnings /// reach , and an order the broker never reports goes to - /// . The created service is kept in . + /// . The created service is kept in . /// /// Reads the current state of one order by its brokerage id. A null /// return means the broker does not know the id. @@ -377,7 +378,7 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before - /// is called. Null falls back to one minute. + /// is called. Null falls back to one minute. /// The created and wired service. protected PerOrderIdPollingService CreateOrderPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, @@ -399,7 +400,7 @@ protected PerOrderIdPollingService CreateOrderPollingService(FuncHow long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before - /// is called. Null falls back to one minute. + /// is called. Null falls back to one minute. /// The created and wired service. protected AllOrdersPollingService CreateOrderPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, @@ -423,7 +424,7 @@ private void WireOrderPollingService(BrokerageOrderPollingService service) service.OrderEvents += (_, orderEvents) => OnOrderEvents(orderEvents); service.Message += (_, message) => OnMessage(message); - service.OrderNotAcknowledged += (_, notAcknowledged) => OnOrderPollingNotAcknowledged(notAcknowledged); + service.BrokerageOrderNeverNotified += (_, neverNotified) => OnBrokerageOrderNeverNotified(neverNotified); OrderPollingService = service; Log.Trace($"Brokerage.WireOrderPollingService(): {Name} created a {service.GetType().Name}: " + @@ -434,12 +435,12 @@ private void WireOrderPollingService(BrokerageOrderPollingService service) /// Called when the broker never reported a watched order for the whole watch timeout. The default /// sends one warning message; override it to decide what the silence means for this broker. /// - /// The brokerage order id and how long it was watched. - protected virtual void OnOrderPollingNotAcknowledged(OrderNotAcknowledgedEventArgs notAcknowledged) + /// The brokerage order id and how long it was watched. + protected virtual void OnBrokerageOrderNeverNotified(BrokerageOrderNeverNotifiedEventArgs neverNotified) { OnMessage(new BrokerageMessageEvent(BrokerageMessageType.Warning, "OrderNotAcknowledged", - $"{Name} never reported order '{notAcknowledged.BrokerageOrderId}' after " + - $"{notAcknowledged.WatchedFor.TotalSeconds:F0} seconds of polling. The order may not have been accepted, verify it manually.")); + $"{Name} was never notified about the order ({neverNotified}). " + + $"It may not have been accepted, verify it manually.")); } #endregion diff --git a/Brokerages/Services/OrderNotAcknowledgedEventArgs.cs b/Brokerages/Services/OrderNotAcknowledgedEventArgs.cs deleted file mode 100644 index d6bd0b33649f..000000000000 --- a/Brokerages/Services/OrderNotAcknowledgedEventArgs.cs +++ /dev/null @@ -1,48 +0,0 @@ -/* - * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. - * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. -*/ - -using System; - -namespace QuantConnect.Brokerages.Services -{ - /// - /// Raised when a watched brokerage order id went unreported for the whole watch timeout of polling. - /// This is a question, not a verdict: the service does not know whether the order never reached the - /// broker or closed before the first sweep saw it. The brokerage decides what to do next. - /// - public class OrderNotAcknowledgedEventArgs : EventArgs - { - /// - /// The brokerage order id nothing ever reported. - /// - public string BrokerageOrderId { get; } - - /// - /// How long the id was watched, in polling time, before the timeout fired. - /// - public TimeSpan WatchedFor { get; } - - /// - /// Creates a new . - /// - /// The brokerage order id nothing ever reported. - /// How long the id was watched, in polling time, before the timeout fired. - public OrderNotAcknowledgedEventArgs(string brokerageOrderId, TimeSpan watchedFor) - { - BrokerageOrderId = brokerageOrderId; - WatchedFor = watchedFor; - } - } -} diff --git a/Brokerages/Services/AllOrdersPollingService.cs b/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs similarity index 91% rename from Brokerages/Services/AllOrdersPollingService.cs rename to Brokerages/Services/OrderPolling/AllOrdersPollingService.cs index 6c28a7429608..ae2633b1038e 100644 --- a/Brokerages/Services/AllOrdersPollingService.cs +++ b/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs @@ -16,8 +16,9 @@ using System; using QuantConnect.Securities; using System.Collections.Generic; +using QuantConnect.Brokerages.Services.OrderPolling.Models; -namespace QuantConnect.Brokerages.Services +namespace QuantConnect.Brokerages.Services.OrderPolling { /// /// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched, and the @@ -40,7 +41,7 @@ public class AllOrdersPollingService : BrokerageOrderPollingService /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before - /// is raised. Null falls back to one minute. + /// is raised. Null falls back to one minute. public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) : base(messageHandler, orderProvider, pollInterval, watchTimeout) diff --git a/Brokerages/Services/BrokerageOrderPollingService.cs b/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs similarity index 95% rename from Brokerages/Services/BrokerageOrderPollingService.cs rename to Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs index 78885b3f71b4..4ec42ec23a76 100644 --- a/Brokerages/Services/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs @@ -14,9 +14,11 @@ */ using System; +using System.Linq; using System.Threading; using QuantConnect.Util; using QuantConnect.Orders; +using QuantConnect.Brokerages.Services.OrderPolling.Models; using QuantConnect.Logging; using System.Threading.Tasks; using QuantConnect.Securities; @@ -24,7 +26,7 @@ using System.Collections.Generic; using QuantConnect.Configuration; -namespace QuantConnect.Brokerages.Services +namespace QuantConnect.Brokerages.Services.OrderPolling { /// /// Reads orders from the brokerage on an interval and turns the returned states into order events. @@ -104,7 +106,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// A watched order that nothing reported for of polling. Raised once; /// the id is unwatched with it. The brokerage decides what the silence means. /// - public event EventHandler OrderNotAcknowledged; + public event EventHandler BrokerageOrderNeverNotified; /// /// Several reads in a row failed, so the run currently has no order updates. Raised once per @@ -124,7 +126,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// /// How long a watched order may stay completely unreported, in polling time, before - /// is raised for it. + /// is raised for it. /// public TimeSpan WatchTimeout { get; } @@ -141,7 +143,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before - /// is raised. Null falls back to one minute. + /// is raised. Null falls back to one minute. protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) { @@ -191,7 +193,7 @@ protected List GetWatchedBrokerageIds() /// /// Watches a brokerage order id, with nothing seen for it yet, so the first state to carry the id /// acknowledges the order and of silence raises - /// . Idempotent: watching an already-watched id never overwrites + /// . Idempotent: watching an already-watched id never overwrites /// its state. /// /// The brokerage order id to watch. @@ -680,12 +682,12 @@ private async Task PollLoop(CancellationToken cancellationToken) /// /// Counts one interval of silence for every watched order the broker never reported, and raises - /// once for each one that reached the watch timeout. Called only + /// once for each one that reached the watch timeout. Called only /// after a successful sweep, because only a read that succeeded proves the silence is real. /// private void CheckWatchTimeouts() { - List expired = null; + List<(string BrokerageId, TimeSpan WatchDuration)> expired = null; lock (_lock) { foreach (var (brokerageId, entry) in _orderStates) @@ -695,27 +697,30 @@ private void CheckWatchTimeouts() continue; } - entry.UnacknowledgedFor += PollInterval; - if (entry.UnacknowledgedFor >= WatchTimeout) + entry.UnacknowledgedDuration += PollInterval; + if (entry.UnacknowledgedDuration >= WatchTimeout) { - (expired ??= []).Add(new OrderNotAcknowledgedEventArgs(brokerageId, entry.UnacknowledgedFor)); + (expired ??= []).Add((brokerageId, entry.UnacknowledgedDuration)); } } if (expired != null) { - foreach (var eventArgs in expired) + foreach (var (brokerageId, _) in expired) { - _orderStates.Remove(eventArgs.BrokerageOrderId); + _orderStates.Remove(brokerageId); } } } if (expired != null) { - foreach (var eventArgs in expired) + foreach (var (brokerageId, watchDuration) in expired) { - OrderNotAcknowledged?.Invoke(this, eventArgs); + // Resolved outside the registry lock. A placement whose id assignment was itself the + // thing that never happened resolves to no Lean order, so the args carry null then. + var leanOrder = _orderProvider?.GetOrdersByBrokerageId(brokerageId)?.FirstOrDefault(); + BrokerageOrderNeverNotified?.Invoke(this, new BrokerageOrderNeverNotifiedEventArgs(brokerageId, leanOrder, watchDuration)); } } } @@ -767,7 +772,7 @@ private class OrderStateEntry /// /// How long the order has been watched with nothing reporting it, in polling time. /// - public TimeSpan UnacknowledgedFor; + public TimeSpan UnacknowledgedDuration; } } } diff --git a/Brokerages/Services/BrokerOrderState.cs b/Brokerages/Services/OrderPolling/Models/BrokerOrderState.cs similarity index 98% rename from Brokerages/Services/BrokerOrderState.cs rename to Brokerages/Services/OrderPolling/Models/BrokerOrderState.cs index cf54beb16672..d18808f4baf1 100644 --- a/Brokerages/Services/BrokerOrderState.cs +++ b/Brokerages/Services/OrderPolling/Models/BrokerOrderState.cs @@ -16,7 +16,7 @@ using System; using QuantConnect.Orders; -namespace QuantConnect.Brokerages.Services +namespace QuantConnect.Brokerages.Services.OrderPolling.Models { /// /// One order, as the brokerage last saw it. The brokerage converts its own order model into this shape diff --git a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs new file mode 100644 index 000000000000..d669451fd696 --- /dev/null +++ b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs @@ -0,0 +1,65 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using QuantConnect.Orders; + +namespace QuantConnect.Brokerages.Services.OrderPolling.Models +{ + /// + /// Raised when no read saw a watched brokerage order id for the whole watch timeout. A question, not + /// a verdict: the order may never have reached the broker, or closed before the first sweep. The + /// brokerage decides what to do next. + /// + public class BrokerageOrderNeverNotifiedEventArgs : EventArgs + { + /// + /// The brokerage order id no read ever saw. + /// + public string BrokerageOrderId { get; } + + /// + /// The Lean order behind the id, or null when nothing resolves - a placement whose id was never assigned. + /// + public Order Order { get; } + + /// + /// How long the id was watched, in polling time. + /// + public TimeSpan WatchDuration { get; } + + /// + /// Creates a new . + /// + /// The brokerage order id no read ever saw. + /// The Lean order behind the id, or null when nothing resolves. + /// How long the id was watched, in polling time. + public BrokerageOrderNeverNotifiedEventArgs(string brokerageOrderId, Order order, TimeSpan watchDuration) + { + BrokerageOrderId = brokerageOrderId; + Order = order; + WatchDuration = watchDuration; + } + + /// + /// The order and the watch duration, ready for a log line or a warning. + /// + public override string ToString() + { + var order = Order?.ToString() ?? $"brokerage order id '{BrokerageOrderId}'"; + return $"{order}, watched for {WatchDuration.TotalSeconds:F0} seconds"; + } + } +} diff --git a/Brokerages/Services/PerOrderIdPollingService.cs b/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs similarity index 93% rename from Brokerages/Services/PerOrderIdPollingService.cs rename to Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs index 9489a29ef179..73fe8f442670 100644 --- a/Brokerages/Services/PerOrderIdPollingService.cs +++ b/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs @@ -17,8 +17,9 @@ using QuantConnect.Logging; using QuantConnect.Securities; using System.Collections.Generic; +using QuantConnect.Brokerages.Services.OrderPolling.Models; -namespace QuantConnect.Brokerages.Services +namespace QuantConnect.Brokerages.Services.OrderPolling { /// /// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id - @@ -43,7 +44,7 @@ public class PerOrderIdPollingService : BrokerageOrderPollingService /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before - /// is raised. Null falls back to one minute. + /// is raised. Null falls back to one minute. public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) : base(messageHandler, orderProvider, pollInterval, watchTimeout) diff --git a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs similarity index 95% rename from Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs rename to Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs index d444ce831617..977a4ffaaa51 100644 --- a/Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs @@ -21,9 +21,10 @@ using QuantConnect.Orders; using System.Collections.Generic; using QuantConnect.Brokerages; -using QuantConnect.Brokerages.Services; +using QuantConnect.Brokerages.Services.OrderPolling; +using QuantConnect.Brokerages.Services.OrderPolling.Models; -namespace QuantConnect.Tests.Brokerages.Services +namespace QuantConnect.Tests.Brokerages.Services.OrderPolling { [TestFixture] public class BrokerageOrderPollingServiceTests @@ -454,27 +455,28 @@ public void WatchNeverOverwritesExistingState() [Test] public void WatchTimeoutFiresOnceAndUnwatchesTheId() { - using var notAcknowledged = new ManualResetEventSlim(false); - var raised = new List(); + var order = AddOrder(100m, "77"); + using var neverNotified = new ManualResetEventSlim(false); + var raised = new List(); using var service = new PerOrderIdPollingService( _ => null, // the broker never knows the id messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25), watchTimeout: TimeSpan.FromMilliseconds(75)); - service.OrderNotAcknowledged += (_, eventArgs) => + service.BrokerageOrderNeverNotified += (_, eventArgs) => { lock (raised) { raised.Add(eventArgs); } - notAcknowledged.Set(); + neverNotified.Set(); }; service.Watch("77"); service.Start(); - Assert.IsTrue(notAcknowledged.Wait(TimeSpan.FromSeconds(5)), "the watch timeout never fired"); + Assert.IsTrue(neverNotified.Wait(TimeSpan.FromSeconds(5)), "the watch timeout never fired"); // let a few more sweeps run: the id was unwatched with the event, so it fires exactly once Thread.Sleep(200); @@ -484,11 +486,26 @@ public void WatchTimeoutFiresOnceAndUnwatchesTheId() { Assert.AreEqual(1, raised.Count); Assert.AreEqual("77", raised[0].BrokerageOrderId); - Assert.GreaterOrEqual(raised[0].WatchedFor, TimeSpan.FromMilliseconds(75)); + Assert.AreSame(order, raised[0].Order); + Assert.GreaterOrEqual(raised[0].WatchDuration, TimeSpan.FromMilliseconds(75)); } Assert.IsFalse(service.TryGetLastOrderState("77", out _)); } + [Test] + public void NeverNotifiedArgsPrintTheOrderAndTheWatchDuration() + { + var order = AddOrder(100m, "42"); + + // the message the default warning embeds: the Lean order's own ToString plus the watch duration + var withOrder = new BrokerageOrderNeverNotifiedEventArgs("42", order, TimeSpan.FromSeconds(60)); + Assert.AreEqual($"{order}, watched for 60 seconds", withOrder.ToString()); + + // a placement whose id was never assigned resolves to no Lean order: the id carries the identity + var withoutOrder = new BrokerageOrderNeverNotifiedEventArgs("42", order: null, TimeSpan.FromSeconds(60)); + Assert.AreEqual("brokerage order id '42', watched for 60 seconds", withoutOrder.ToString()); + } + [Test] public void AcknowledgedWatchNeverTimesOut() { @@ -499,7 +516,7 @@ public void AcknowledgedWatchNeverTimesOut() _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25), watchTimeout: TimeSpan.FromMilliseconds(75)); - service.OrderNotAcknowledged += (_, _) => Interlocked.Increment(ref fired); + service.BrokerageOrderNeverNotified += (_, _) => Interlocked.Increment(ref fired); // the stream acknowledged the order right after it was watched service.Watch("77"); From 34f4c77e2b8c24064bbf68601acce7c22a075a71 Mon Sep 17 00:00:00 2001 From: Romazes Date: Tue, 18 Aug 2026 23:21:33 +0300 Subject: [PATCH 17/25] docs: record the folder split and the watch alarm rework in the adr --- .../0001-brokerage-order-polling-service.md | 86 +++++++++++-------- 1 file changed, 50 insertions(+), 36 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index cb933206d2fd..55fab6d96acc 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -29,7 +29,7 @@ moved into its mapping (see "The diff", pricing). Wiring moved into core - 2026-08-17: the root `Brokerage` class gained the seam the base-class section describes - two protected `CreateOrderPollingService` overloads whose read-callback signature picks the mode, the service as a protected `OrderPollingService` property with `IsOrderPolling`, a virtual -`OnOrderPollingNotAcknowledged` for the silence warning, and a `Dispose` that covers the service. +`OnBrokerageOrderNeverNotified` for the silence warning, and a `Dispose` that covers the service. CharlesSchwab and Public.com were moved onto it: the creation, the three event forwards and the dispose call left both plugins (see "Wiring, per plugin"). @@ -61,6 +61,17 @@ three adopters build through them. The wiring now traces the created mode class the pre-loading `Start` traces how many open orders it pre-loaded. Schwab moved its mapping into `CharlesSchwabExtensions.ToLegOrderStates`, called as `brokerageOrder.ToLegOrderStates()` from the read. +Folders split per service - 2026-08-18: the service moved to `Brokerages/Services/OrderPolling/`, with its +data shapes in `Models/` under it, and the namespaces follow the folders. `Services` holds one subfolder +per plugin service from now on, so the next service gets a sibling folder instead of growing one flat +namespace. All three adopters moved onto the new usings with it (see "Where it lives"). + +Watch alarm renamed and enriched - 2026-08-18: `OrderNotAcknowledged` became `BrokerageOrderNeverNotified` - +the brokerage never notified about the order - with the virtual and the args renamed with it. The args now +resolve the Lean order behind the id (null for a placement whose id was never assigned), carry the watch +duration, and print themselves through `ToString`, which the default warning uses. The message code string +stays `"OrderNotAcknowledged"`, so live logs keep their vocabulary. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -121,7 +132,7 @@ this service is that ask, and Schwab's polled mode now answers the same wait. | Brokerage | Waits for | How long | Where the number comes from | When it expires | | --- | --- | --- | --- | --- | -| CharlesSchwab | `OrderAccepted` on the account activity stream | **3 minutes** | hardcoded `TimeSpan.FromMinutes(3)`, `CharlesSchwabBrokerage.cs:486` | `Error` `MissingWebSocketResponse` (`:488`) | +| CharlesSchwab | `OrderAccepted` on the account activity stream | **3 minutes** | hardcoded `TimeSpan.FromMinutes(3)`, `CharlesSchwabBrokerage.cs:483` | `Error` `MissingWebSocketResponse` (`:485`) | | InteractiveBrokers | `openOrder` / `orderStatus` / `execDetails` callback | **5 minutes** | `ib-response-timeout`, default `300` seconds, `InteractiveBrokersBrokerage.cs:84` | `Error` `NoBrokerageResponse` (`:1659`) | | InteractiveBrokers, `MarketOnOpen` / `ComboLegLimit` / `ComboMarket` / `ComboLimit` | same | 10 seconds | `ib-no-submission-orders-response-timeout`, `:90` | Lean **invents** a `Submitted` event (`:1649-1652`) | @@ -245,16 +256,17 @@ choice, made when it picks the class. ### Where it lives -`Lean/Brokerages/Services/BrokerageOrderPollingService.cs` — the base class — with -`PerOrderIdPollingService.cs`, `AllOrdersPollingService.cs` and `BrokerOrderState.cs` next to it, all in -namespace `QuantConnect.Brokerages.Services`. +`Lean/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs` — the base class — with +`PerOrderIdPollingService.cs` and `AllOrdersPollingService.cs` next to it, in namespace +`QuantConnect.Brokerages.Services.OrderPolling`. The data shapes live one level deeper, in `Models/` with +the matching namespace suffix: `BrokerOrderState.cs` and `BrokerageOrderNeverNotifiedEventArgs.cs`. `Services` is a new folder under `Brokerages`, and it follows the convention the other subfolders there already use: -`Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. It also matches where -the two plugins kept this class before the move — CharlesSchwab had it in `QuantConnect.CharlesSchwabBrokerage/Services/`, so -the move up to core kept the same path. +`Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. Each service +takes its own subfolder inside it — `OrderPolling` is the first — so the next plugin service gets a sibling +folder instead of growing one flat namespace. -Tests go to `Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs`. +Tests go to `Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs`. ### How a poll flows @@ -287,7 +299,7 @@ poller the same way. The service's own loop does the enqueue now. ### The broker order state ```csharp -namespace QuantConnect.Brokerages.Services; +namespace QuantConnect.Brokerages.Services.OrderPolling.Models; /// /// One order, as the brokerage last saw it. The brokerage converts its own order model into this shape @@ -340,7 +352,7 @@ instead of carrying a fee field nobody fills. Combos come in two shapes, and the state carries both without any extra field. Schwab gives every leg its own brokerage id (`mainId + legId - 1`), so the plugin passes one state per leg. Public.com gives the whole combo -**one** id (`PublicBrokerage.Brokerage.cs:430-441`), so the plugin passes one state and the service fans it out: +**one** id (`PublicBrokerage.Brokerage.cs:431-442`), so the plugin passes one state and the service fans it out: `GetOrdersByBrokerageId` returns every Lean leg order behind the id, and each leg's share of a new fill is ``` @@ -350,7 +362,7 @@ legFill = leanOrder.Quantity * newPart / abs(leanOrder.GroupOrderManager.Quantit The state has no quantity field because the service does not need one: it already knows the brokerage id, so it reads the group quantity from the Lean orders themselves. That number equals the broker's own order quantity — the group quantity is exactly what the combo was placed with. Public's replaced diff split fills this way, and -the rule moved into the service's fan-out (`Brokerages/Services/BrokerageOrderPollingService.cs`). One rule for the mapping follows: +the rule moved into the service's fan-out (`Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs`). One rule for the mapping follows: `FilledQuantity` of a shared-id combo is in strategy units, the same units the group quantity counts in. A worked example, from a real Public.com order — 5 AAPL strangles, one brokerage id, a put leg and a call leg with @@ -424,7 +436,7 @@ Lean orders it already looks up. ### The class ```csharp -namespace QuantConnect.Brokerages.Services; +namespace QuantConnect.Brokerages.Services.OrderPolling; /// /// Reads orders from the brokerage on an interval and turns the returned snapshots into order events. @@ -456,7 +468,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// A watched order that nothing reported for watchTimeout of polling. /// Raised once; the id is unwatched with it. - public event EventHandler OrderNotAcknowledged; + public event EventHandler BrokerageOrderNeverNotified; /// Several reads in a row failed, so the run currently has no order updates. public event EventHandler Message; @@ -591,7 +603,7 @@ placement's assignment is the same code the stream's `OrderAccepted` runs, and i plugin's place wait. A replace has a mirror of it, fed from its own pending record: its assignment moves each Lean order onto its new id and marks it through `WatchReplacement`, so the diff reports the update submit instead of a plain submit. Nothing waits on a replacement - the replace reply already confirmed it, -and the watch raises `OrderNotAcknowledged` if the broker never lists it. Schwab's fallback path works this +and the watch raises `BrokerageOrderNeverNotified` if the broker never lists it. Schwab's fallback path works this way today. ### One lock for two message types @@ -690,7 +702,7 @@ identical to the first and be lost. Pricing is deliberately simple: the new part takes the state's `FillPrice`, as the broker reported it. When several fills land inside one sweep, the quantity is still exact and the price is the broker's reported price at sweep time, not each fill's own — Tradier's replaced poller shipped exactly this trade-off, and its adoption -keeps it, documented in the mapping (`TradierBrokerage.cs:1169-1171`). Public recovers the exact increment price from the change of the average; +keeps it, documented in the mapping (`TradierBrokerage.cs:1170-1172`). Public recovers the exact increment price from the change of the average; the service refuses that arithmetic on purpose — it amplifies broker rounding and can even go negative on a tiny part, while the simple price needs no guard at all. Public's adoption kept the recovery anyway, but inside its own mapping: the state's `FillPrice` arrives already recovered, with a guard that falls back to the plain @@ -711,7 +723,7 @@ emission. The "none found" branch skips today, but it is also an opening. An id the order provider keeps not knowing is most likely an order the user placed outside Lean, in the broker's own app — and Lean already has a door for those: -`OnNewBrokerageOrderNotification` (`Brokerages/Brokerage.cs:257`). The transaction handler picks it up +`OnNewBrokerageOrderNotification` (`Brokerages/Brokerage.cs:258`). The transaction handler picks it up (`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:190`, `:1674`), asks the algorithm's brokerage message handler whether to accept the order, and adopts it with `AddOpenOrder`. TradeStation already raises it from its stream (`TradeStationBrokerage.cs:1088`); a poll can feed the same door. @@ -731,18 +743,20 @@ diffs. With the snapshot the diff is shared, whole, and the plugins keep only th `Watch(brokerageId)` is called by `PlaceOrder` right after the request returns an id. From then on: -- A snapshot arrives for the id, or the stream records one through `UpdateOrderState` → the order is acknowledged, +- A snapshot arrives for the id, or the stream records one through `UpdateOrderState` → the order was seen, and the id stays watched until the order closes. - The order is closed in Lean → `Unwatch`, and its state is dropped. -- `watchTimeout` of polling passes and nothing ever carried the id → `OrderNotAcknowledged` is raised once, with - the id and how long it was watched, and the id is unwatched. +- `watchTimeout` of polling passes and nothing ever carried the id → `BrokerageOrderNeverNotified` is raised once, with + the id, the Lean order behind it when one resolves (null for a placement whose id assignment never + happened), and how long it was watched; the args print all of it through `ToString`, and the id is + unwatched. The timeout only counts while the service is polling, and only on sweeps whose read succeeded — a failed read asked the broker nothing, so it proves no silence. A watch set while the service is stopped does not count — otherwise every healthy order would hit the timeout the moment polling starts. So a plugin in watch mode calls `Start` together with `Watch` (calling `Start` twice is fine) and may `Stop` once nothing is watched. -`OrderNotAcknowledged` is a question, not a verdict — this is rule 2 again, a missing order proves nothing. The service does +`BrokerageOrderNeverNotified` is a question, not a verdict — this is rule 2 again, a missing order proves nothing. The service does not know whether the order never arrived or filled instantly, so it does not decide. The brokerage handles it: Public can call its get-order endpoint, Schwab can read the order with its executions, IB can use `reqExecutions`. A brokerage that handles nothing raises a `Warning` and the run keeps going, which is already better than today's @@ -784,7 +798,7 @@ The seed rule stays for what a plugin still reports itself: it seeds in the same sweep does not repeat it. The assignment also releases the three minute place wait, so the same `MissingWebSocketResponse` error guards a placement nothing ever confirms, on the stream and on the poll alike. A replacement has no wait of its own - the replace reply already confirmed it. The watch doubles as -the alarm: an id the broker never lists raises `OrderNotAcknowledged` instead of staying silent. +the alarm: an id the broker never lists raises `BrokerageOrderNeverNotified` instead of staying silent. ### A replace, across the brokers @@ -809,7 +823,7 @@ facts that matter to a poll: does the id change, where does the new id come from The rows in code: IB re-sends the same broker id (`InteractiveBrokersBrokerage.cs:1599,1626`), TradeStation PUTs to the existing id and never reads the reply's `OrderID` (`Api/TradeStationApiClient.cs:329-335`), -Tradier's PUT has no quantity parameter at all (`TradierBrokerage.cs:475-496`), Public replaces in place +Tradier's PUT has no quantity parameter at all (`TradierBrokerage.cs:476-497`), Public replaces in place with the id echoed back (`Api/ApiClient.cs:351-362`), Webull keeps the `client_order_id` that is the Lean `BrokerId` (`Api/ApiClient.cs:595-605`), ByBit amends futures under the same id (`Api/BybitTradeApiEndpoint.cs:94-100`) and cannot amend spot (`BybitBrokerage.Brokerage.cs:200-204`), and @@ -821,7 +835,7 @@ count continues on it — all six carry-over cells are verified in code, e.g. Tr itself can never come from a sweep: the state carries a status and two fill numbers, no price and no quantity, so a modified order polls exactly like an unmodified one. The rule for a same-id plugin is Tradier's and Public's, already shipping: report `UpdateSubmitted` right after the REST reply -(`TradierBrokerage.cs:947-948`, `PublicBrokerage.Brokerage.cs:692`) and leave the registry alone — Public +(`TradierBrokerage.cs:948-949`, `PublicBrokerage.Brokerage.cs:693`) and leave the registry alone — Public runs on this service today and its `UpdateOrder` makes no service call. The two that report from the stream instead (IB `InteractiveBrokersBrokerage.cs:2359,2380`, Webull `WebullBrokerage.Brokerage.cs:448-453`) simply have no update report while their channel is down; on adoption they move the report next to the REST @@ -1016,7 +1030,7 @@ CreateOrderPollingService(ReadOrderState, _messageHandler, _orderProvider, pollI // interval gets the shared brokerage-order-poll-interval-ms default, 3000 ms. ``` -What silence means is one override per plugin: `OnOrderPollingNotAcknowledged` defaults to a single +What silence means is one override per plugin: `OnBrokerageOrderNeverNotified` defaults to a single warning built from the brokerage name, and Schwab overrides it to keep the wording its live pilot verified. @@ -1068,13 +1082,13 @@ fill timer and `CheckForFills` diff the same way; its cross-zero split is "A cro - Reporting without the stream: the plugin reports neither the place nor the replace itself. It watches the main id, the first sweep assigns the leg ids by symbol — a replacement's through `WatchReplacement` — and the diff reports the submit or the update submit. -- Deciding what an unacknowledged order means — the `OnOrderPollingNotAcknowledged` override; the default +- Deciding what a never-notified order means — the `OnBrokerageOrderNeverNotified` override; the default is one warning built from the brokerage name. ## Testing the polling in a plugin The service's own behavior — the diff, the watch, the registry — is covered once, in Lean's -`Tests/Brokerages/Services/BrokerageOrderPollingServiceTests.cs`. A plugin does not test the service +`Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs`. A plugin does not test the service again: it tests its read, its mapping and its wiring **through** the service. The offline tests carry the everyday coverage; the live tests prove the whole path against the real broker — and for a plugin starting from nothing they come first, because their runs record the data the offline tests replay. The reference is the Schwab pilot's fixture, @@ -1222,7 +1236,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth wrap every stream message in a new closure — one allocation per message on the hottest path a brokerage has. The non-generic handler keeps the message itself in the buffer and allocates only at registration. - **Put it on the `Brokerage` base class, driven by the engine, like the cash sync.** - `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:577`, called from + `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:578`, called from `Engine/TransactionHandlers/BrokerageTransactionHandler.cs:731`) is the existing shape for core-driven periodic work, and it was the obvious candidate. Rejected: cash sync runs once a day on a schedule core can decide, while the right poll interval here is a property of the broker's rate limits and of whether its stream is alive. It also @@ -1243,7 +1257,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth already raises `OnOrderIdChangedEvent` when a replace re-keys a Lean order, so the service could subscribe to it instead of offering a method. Rejected three times over. The event does not mean "replace": Lean core raises it on a plain place — the cross-zero flow assigns the second part's id through it - (`Brokerages/Brokerage.cs:939`) — and Binance assigns every initial id with it, so the service would + (`Brokerages/Brokerage.cs:940`) — and Binance assigns every initial id with it, so the service would report an update submit for a placement. The event does not carry the previous id, and the service cannot recover it: the transaction handler subscribes first and swaps `order.BrokerId` before a later subscriber runs (`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:1497`), so the old registry entries could @@ -1256,7 +1270,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth ### A base brokerage class that owns the wiring Every adopter wires the service the same way: create the mode class with the read callback, forward -`OrderEvents` to `OnOrderEvents` and `Message` to `OnMessage`, decide what `OrderNotAcknowledged` means, and +`OrderEvents` to `OnOrderEvents` and `Message` to `OnMessage`, decide what `BrokerageOrderNeverNotified` means, and dispose the service with the brokerage. So the tempting next step is to move that wiring into an inheritance layer — one abstract brokerage class that owns the service and asks the plugin only for overrides: @@ -1312,17 +1326,17 @@ dispose call. What it cannot absorb is everything else the plugin writes: the re lines in Schwab, ~45 in Public), the `Watch` calls inside the order methods, and the start trigger, which is broker policy — Public starts polling in `Connect` because the service is its connection, Schwab starts it when the stream is taken away. The class cannot even own the stop: `Disconnect` is abstract on `Brokerage` -(`Brokerages/Brokerage.cs:150`), so `Stop` stays a line the plugin writes either way. An abstract read would +(`Brokerages/Brokerage.cs:151`), so `Stop` stays a line the plugin writes either way. An abstract read would guard the one step nobody gets wrong, and guard it for a third of the plugins. The place that reaches every row of the table is the root `Brokerage` class, the way the cross-zero helpers and `CreateOAuthTokenHandler` already sit there as protected members only some plugins use -(`Brokerages/Brokerage.cs:701`, `:343`): one protected creation helper per mode, whose read-callback +(`Brokerages/Brokerage.cs:702`, `:344`): one protected creation helper per mode, whose read-callback signature picks the class, the service as a protected property, and a `Dispose` that covers it. Every brokerage derives from `Brokerage`, the sealed IB class included, and constructing the service directly stays possible — the helper is additive. Implemented that way on 2026-08-17: the seam is the two `CreateOrderPollingService` overloads, `OrderPollingService`, `IsOrderPolling` and the virtual -`OnOrderPollingNotAcknowledged`, and Schwab, Public.com and Tradier all create their service through it +`OnBrokerageOrderNeverNotified`, and Schwab, Public.com and Tradier all create their service through it (see "Wiring, per plugin"). The abstract class this section declines stays declined. ## Risks @@ -1331,12 +1345,12 @@ stays possible — the helper is additive. Implemented that way on 2026-08-17: t | --- | --- | | A plugin maps a broker status to the wrong Lean status | The mapping is the same one its streaming path already needs, written once per plugin and covered by its own tests. The service only emits transitions, so a wrong mapping surfaces once, not as a flood. | | A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the watch timeout still fires. | -| Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier kept this trade-off through its adoption (`TradierBrokerage.cs:1169-1171`); a shorter poll interval narrows it. | -| Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order is unacknowledged, and the interval is a constructor argument the plugin picks. | +| Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier kept this trade-off through its adoption (`TradierBrokerage.cs:1170-1172`); a shorter poll interval narrows it. | +| Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order still waits for its first report, and the interval is a constructor argument the plugin picks. | | A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | | A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; Tradier adopted with null — its submit event goes out before the watch begins, so the poll cannot outrun it — and IB's rollout step still names adding one. | | The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | -| The watch timeout cannot tell "never arrived" from "filled instantly" | It does not try. `OrderNotAcknowledged` hands the question to the brokerage, which has the endpoints to answer it. | +| The watch timeout cannot tell "never arrived" from "filled instantly" | It does not try. `BrokerageOrderNeverNotified` hands the question to the brokerage, which has the endpoints to answer it. | | Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | | A plugin starts polling on disconnect and forgets to stop on reconnect | Both paths are one line and sit next to the connection handling the plugin already has. The poll side repeats nothing the registry already holds, so the cost of forgetting is extra requests, not extra events. | From 9b74a382f77b32db14d9e80c2823679a8dccdfc0 Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 00:26:17 +0300 Subject: [PATCH 18/25] refactor: rename the state to BrokerageOrderSnapshot and default its time - rename BrokerOrderState to BrokerageOrderSnapshot across the service and tests - drop the property-fill constructor, every builder goes through the constructors - timeUtc is nullable in the constructors and defaults to DateTime.UtcNow - shorten the snapshot xml docs --- Brokerages/Brokerage.cs | 4 +- .../OrderPolling/AllOrdersPollingService.cs | 6 +-- .../BrokerageOrderPollingService.cs | 22 ++++---- ...rderState.cs => BrokerageOrderSnapshot.cs} | 53 +++++++------------ .../OrderPolling/PerOrderIdPollingService.cs | 8 +-- .../BrokerageOrderPollingServiceTests.cs | 12 ++--- 6 files changed, 46 insertions(+), 59 deletions(-) rename Brokerages/Services/OrderPolling/Models/{BrokerOrderState.cs => BrokerageOrderSnapshot.cs} (54%) diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index 0b5ba9379a05..5840ec2d34ad 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -380,7 +380,7 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// How long a watched order may stay unreported before /// is called. Null falls back to one minute. /// The created and wired service. - protected PerOrderIdPollingService CreateOrderPollingService(Func readOrder, + protected PerOrderIdPollingService CreateOrderPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) { @@ -402,7 +402,7 @@ protected PerOrderIdPollingService CreateOrderPollingService(FuncHow long a watched order may stay unreported before /// is called. Null falls back to one minute. /// The created and wired service. - protected AllOrdersPollingService CreateOrderPollingService(Func> readAllOrders, + protected AllOrdersPollingService CreateOrderPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) { diff --git a/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs b/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs index ae2633b1038e..6b80bcc14bc4 100644 --- a/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs +++ b/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs @@ -29,7 +29,7 @@ public class AllOrdersPollingService : BrokerageOrderPollingService /// /// Reads every order the broker lists. /// - private readonly Func> _readAllOrders; + private readonly Func> _readAllOrders; /// /// Creates a new . @@ -42,7 +42,7 @@ public class AllOrdersPollingService : BrokerageOrderPollingService /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is raised. Null falls back to one minute. - public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, + public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) : base(messageHandler, orderProvider, pollInterval, watchTimeout) { @@ -52,7 +52,7 @@ public AllOrdersPollingService(Func> readAllOrders /// /// Calls the read once for the whole sweep. /// - protected override IEnumerable Sweep() + protected override IEnumerable Sweep() { return _readAllOrders(); } diff --git a/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs b/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs index 4ec42ec23a76..bd9ec9db6081 100644 --- a/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs @@ -65,7 +65,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// Where each state a sweep returns goes: the message handler, or without one straight into /// . /// - private readonly Action _route; + private readonly Action _route; /// /// Resolves brokerage order ids to Lean orders on every compare, so the service never drifts from @@ -151,7 +151,7 @@ protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler message { _messageHandler = messageHandler; _route = messageHandler.HandleNewMessage; - messageHandler.Register(ProcessOrderState); + messageHandler.Register(ProcessOrderState); } else { @@ -167,7 +167,7 @@ protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler message /// , hands each state to the route, and counts a throw as one failed sweep. /// A null state is skipped: in per-id mode it means the broker does not know the id yet. /// - protected abstract IEnumerable Sweep(); + protected abstract IEnumerable Sweep(); /// /// A copy of the brokerage order ids a sweep still has to read: everything tracked whose end was @@ -210,7 +210,7 @@ public void Watch(string brokerageId) /// /// The brokerage order id to watch. /// The state another path already reported for the order. - public void Watch(string brokerageId, BrokerOrderState lastSeen) + public void Watch(string brokerageId, BrokerageOrderSnapshot lastSeen) { lock (_lock) { @@ -238,7 +238,7 @@ public void Watch(string brokerageId, BrokerOrderState lastSeen) /// The first state to carry the new id reports the order as update submitted, which a stream /// would otherwise do. The new id starts with no fill state, because a replacement that counts /// its executions from zero must not inherit the old order's numbers; a broker that carries the - /// fills across a replace seeds with instead. + /// fills across a replace seeds with instead. /// /// The brokerage order id the replacement runs under. /// The replaced brokerage order id, or null when it is unknown. @@ -279,7 +279,7 @@ public void Unwatch(string brokerageId) /// /// The brokerage order id the state belongs to. /// The cumulative state the other path reported. - public void UpdateOrderState(string brokerageId, BrokerOrderState orderState) + public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderState) { lock (_lock) { @@ -317,7 +317,7 @@ public void UpdateOrderState(string brokerageId, BrokerOrderState orderState) /// The brokerage order id to look up. /// When this method returns true, the last state seen; otherwise null. /// true when a state was ever recorded for the id; otherwise false. - public bool TryGetLastOrderState(string brokerageId, out BrokerOrderState lastSeen) + public bool TryGetLastOrderState(string brokerageId, out BrokerageOrderSnapshot lastSeen) { lock (_lock) { @@ -338,7 +338,7 @@ public bool TryGetLastOrderState(string brokerageId, out BrokerOrderState lastSe /// handler only the poll loop calls it. /// /// The state a sweep read from the broker. - public void ProcessOrderState(BrokerOrderState orderState) + public void ProcessOrderState(BrokerageOrderSnapshot orderState) { if (orderState == null || string.IsNullOrEmpty(orderState.BrokerageOrderId)) { @@ -497,7 +497,7 @@ public void ProcessOrderState(BrokerOrderState orderState) /// /// The whole handover from a stream to polling, in the only safe order: process what the stream - /// already delivered, pre-load the registry with one + /// already delivered, pre-load the registry with one /// per open Lean order, then start the loop. Does nothing while polling already runs. /// /// @@ -506,7 +506,7 @@ public void ProcessOrderState(BrokerOrderState orderState) /// Builds the state another path already reported for one open Lean /// order: the brokerage id, the order's status and the cumulative filled quantity. A null return /// skips the order, and a null callback pre-loads nothing. - public void Start(Func preLoadOpenOrders) + public void Start(Func preLoadOpenOrders) { if (IsPolling) { @@ -734,7 +734,7 @@ private class OrderStateEntry /// The last state seen for the order, from any path. Null when nothing was seen yet, so the /// submit is still due. /// - public BrokerOrderState LastSeen; + public BrokerageOrderSnapshot LastSeen; /// /// The cumulative filled quantity already reported to Lean, by any path. Never shrinks. diff --git a/Brokerages/Services/OrderPolling/Models/BrokerOrderState.cs b/Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs similarity index 54% rename from Brokerages/Services/OrderPolling/Models/BrokerOrderState.cs rename to Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs index d18808f4baf1..064eca3727d7 100644 --- a/Brokerages/Services/OrderPolling/Models/BrokerOrderState.cs +++ b/Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs @@ -19,40 +19,36 @@ namespace QuantConnect.Brokerages.Services.OrderPolling.Models { /// - /// One order, as the brokerage last saw it. The brokerage converts its own order model into this shape - /// and passes it to , which compares it with the last state - /// seen for the same order and reports only what is new. Every field except the id and the status is - /// optional, and null means "my read does not know", never "zero". + /// One order, as the brokerage last saw it. The plugin converts its broker model into this shape; + /// diffs it against the last snapshot seen and reports + /// only what is new. Null in an optional field means "my read does not know", never "zero". /// - public class BrokerOrderState + public class BrokerageOrderSnapshot { /// - /// The brokerage order id. Some brokers give every combo leg its own id, some give the whole - /// combo one id; the state carries whatever the broker uses. + /// The brokerage order id - per combo leg or one for the whole combo, whatever the broker uses. /// public string BrokerageOrderId { get; set; } /// - /// The Lean status the brokerage maps its broker's own status to. + /// The Lean status the plugin maps its broker's own status to. /// public OrderStatus Status { get; set; } /// - /// The total absolute quantity filled so far, never the size of the last fill. Null when the - /// read does not carry it - a status without this number can - /// not produce the fill event that closes the order. For a combo that shares one brokerage id - /// across its legs, this counts in strategy units, the same units the group quantity counts in. + /// The total absolute quantity filled so far, never the last fill's size. Null when the read + /// does not carry it - no fill event goes out then. A shared-id combo counts in strategy units. /// public decimal? FilledQuantity { get; set; } /// - /// The price the broker reports for the fills. Null when the read does not carry it, and no - /// fill event goes out without it - the service never invents a number. + /// The price the broker reports for the fills. No fill event goes out without it - the service + /// never invents a number. /// public decimal? FillPrice { get; set; } /// - /// When the brokerage reported this state, in UTC. + /// When the brokerage reported this snapshot, in UTC. /// public DateTime TimeUtc { get; set; } @@ -62,41 +58,32 @@ public class BrokerOrderState public string Message { get; set; } /// - /// Creates an empty state the caller fills through the properties. - /// - public BrokerOrderState() - { - } - - /// - /// Creates a state with no fill numbers: the id, the status, the time and the broker's words - /// for a closing status. + /// Creates a snapshot with no fill numbers. /// /// The brokerage order id. - /// The Lean status the brokerage maps its broker's own status to. - /// When the brokerage reported this state, in UTC. + /// The Lean status the plugin maps its broker's own status to. + /// When the brokerage reported this snapshot, in UTC. Null takes . /// The broker's own words for a closing status. - public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, string message) + public BrokerageOrderSnapshot(string brokerageOrderId, OrderStatus status, DateTime? timeUtc, string message) : this(brokerageOrderId, status, timeUtc, filledQuantity: null, fillPrice: null, message: message) { } /// - /// Creates a state from what one read saw. Only the first three are always known; the fill - /// numbers and the message stay null when the read does not carry them. + /// Creates a snapshot from what one read saw; only the id and the status are always known. /// /// The brokerage order id. - /// The Lean status the brokerage maps its broker's own status to. - /// When the brokerage reported this state, in UTC. + /// The Lean status the plugin maps its broker's own status to. + /// When the brokerage reported this snapshot, in UTC. Null takes . /// The total absolute quantity filled so far. /// The price the broker reports for the fills. /// The broker's own words for a closing status. - public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, + public BrokerageOrderSnapshot(string brokerageOrderId, OrderStatus status, DateTime? timeUtc = null, decimal? filledQuantity = null, decimal? fillPrice = null, string message = null) { BrokerageOrderId = brokerageOrderId; Status = status; - TimeUtc = timeUtc; + TimeUtc = timeUtc ?? DateTime.UtcNow; FilledQuantity = filledQuantity; FillPrice = fillPrice; Message = message; diff --git a/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs b/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs index 73fe8f442670..65a9dfd7a766 100644 --- a/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs +++ b/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs @@ -31,7 +31,7 @@ public class PerOrderIdPollingService : BrokerageOrderPollingService /// /// Reads the current state of one order by its brokerage id. /// - private readonly Func _readOrder; + private readonly Func _readOrder; /// /// Creates a new . @@ -45,7 +45,7 @@ public class PerOrderIdPollingService : BrokerageOrderPollingService /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is raised. Null falls back to one minute. - public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, + public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) : base(messageHandler, orderProvider, pollInterval, watchTimeout) { @@ -57,10 +57,10 @@ public PerOrderIdPollingService(Func readOrder, Broker /// so it cannot starve the other watched orders; the sweep only counts as failed when every read /// of the sweep failed. /// - protected override IEnumerable Sweep() + protected override IEnumerable Sweep() { var brokerageIds = GetWatchedBrokerageIds(); - var orderStates = new List(brokerageIds.Count); + var orderStates = new List(brokerageIds.Count); var failedReadCount = 0; var lastError = default(Exception); foreach (var brokerageId in brokerageIds) diff --git a/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs index 977a4ffaaa51..52624da0d479 100644 --- a/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs @@ -39,7 +39,7 @@ public void SetUp() _orderProvider = new OrderProvider(); _orderEvents = new List(); // the read is unused: these tests drive the diff directly through ProcessOrderState - _service = new AllOrdersPollingService(() => Array.Empty(), messageHandler: null, _orderProvider, + _service = new AllOrdersPollingService(() => Array.Empty(), messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(50), watchTimeout: TimeSpan.FromMilliseconds(120)); _service.OrderEvents += (_, orderEvents) => _orderEvents.AddRange(orderEvents); } @@ -59,9 +59,9 @@ private Order AddOrder(decimal quantity, string brokerageId, OrderStatus status return order; } - private static BrokerOrderState State(string brokerageId, OrderStatus status, decimal? filled = null, decimal? price = null, string message = null) + private static BrokerageOrderSnapshot State(string brokerageId, OrderStatus status, decimal? filled = null, decimal? price = null, string message = null) { - return new BrokerOrderState(brokerageId, status, new DateTime(2026, 8, 12, 14, 30, 0, DateTimeKind.Utc), + return new BrokerageOrderSnapshot(brokerageId, status, new DateTime(2026, 8, 12, 14, 30, 0, DateTimeKind.Utc), filledQuantity: filled, fillPrice: price, message: message); } @@ -536,7 +536,7 @@ public void RepeatedReadFailuresRaiseOneWarningPerOutage() var warnings = new List(); using var warned = new AutoResetEvent(false); using var service = new AllOrdersPollingService( - () => fail ? throw new Exception("read failed") : Array.Empty(), + () => fail ? throw new Exception("read failed") : Array.Empty(), messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25)); @@ -676,7 +676,7 @@ public void StartWithPreLoadDrainsTheMessageHandlerBeforeSeeding() AddOrder(233m, "42", OrderStatus.Submitted); var steps = new List(); using var handler = new BrokerageConcurrentMessageHandler(); - using var service = new AllOrdersPollingService(() => Array.Empty(), handler, _orderProvider, + using var service = new AllOrdersPollingService(() => Array.Empty(), handler, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(50)); service.OrderEvents += (_, orderEvents) => { @@ -752,7 +752,7 @@ public void SeedWithoutBrokerageIdIsSkipped() _service.Start(preLoadOpenOrders: order => order.BrokerId[0] == "42" ? null - : new BrokerOrderState { Status = OrderStatus.Submitted }); + : new BrokerageOrderSnapshot(brokerageOrderId: null, OrderStatus.Submitted)); Assert.IsTrue(_service.IsPolling); Assert.IsFalse(_service.TryGetLastOrderState("42", out _)); From 3ef90039427592648f0581daac072416f8d1bff1 Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 00:26:17 +0300 Subject: [PATCH 19/25] docs: record the snapshot rename and the time default in the adr --- .../0001-brokerage-order-polling-service.md | 62 ++++++++++--------- 1 file changed, 34 insertions(+), 28 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 55fab6d96acc..f714efe945f7 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -55,7 +55,7 @@ Seeded start folded into Start - 2026-08-18: `SeedAndStart(seed)` became the `St overload, so starting is one method with two shapes: the plain `Start()` resumes the loop, and the overload runs the handover first. Behavior unchanged; Schwab's call site renamed with it. -State constructors and wiring traces - 2026-08-18: `BrokerOrderState` gained two constructors - the +State constructors and wiring traces - 2026-08-18: `BrokerageOrderSnapshot` gained two constructors - the always-known facts positionally, and an overload taking the message without the fill numbers - and all three adopters build through them. The wiring now traces the created mode class with its intervals, and the pre-loading `Start` traces how many open orders it pre-loaded. Schwab moved its mapping into @@ -72,6 +72,12 @@ resolve the Lean order behind the id (null for a placement whose id was never as duration, and print themselves through `ToString`, which the default warning uses. The message code string stays `"OrderNotAcknowledged"`, so live logs keep their vocabulary. +State renamed to snapshot - 2026-08-19: `BrokerOrderState` became `BrokerageOrderSnapshot` - this +document's own word for it, with the `Brokerage` prefix the rest of its family already uses. The +property-fill constructor left with it: every builder goes through the constructors now, and the +constructors default the time to `DateTime.UtcNow` when none is passed. All three adopters and the +tests renamed and simplified with it. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -259,7 +265,7 @@ choice, made when it picks the class. `Lean/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs` — the base class — with `PerOrderIdPollingService.cs` and `AllOrdersPollingService.cs` next to it, in namespace `QuantConnect.Brokerages.Services.OrderPolling`. The data shapes live one level deeper, in `Models/` with -the matching namespace suffix: `BrokerOrderState.cs` and `BrokerageOrderNeverNotifiedEventArgs.cs`. +the matching namespace suffix: `BrokerageOrderSnapshot.cs` and `BrokerageOrderNeverNotifiedEventArgs.cs`. `Services` is a new folder under `Brokerages`, and it follows the convention the other subfolders there already use: `Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. Each service @@ -273,7 +279,7 @@ Tests go to `Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingService ``` poll thread — one sweep every PollInterval service calls the read: once per watched id, or once for all orders (by class) - brokerage the read asks the broker and converts each order to a BrokerOrderState + brokerage the read asks the broker and converts each order to a BrokerageOrderSnapshot service sends each state to the brokerage's message handler (waits in the queue while an order request holds the lock) service counts failed reads: three sweeps in a row -> one Warning @@ -296,7 +302,7 @@ Both replaced pollers had exactly this flow: Schwab's loop handed each polled or `_messageHandler.HandleNewMessage` and the diff ran when the handler dequeued it, and Public wired its poller the same way. The service's own loop does the enqueue now. -### The broker order state +### The brokerage order snapshot ```csharp namespace QuantConnect.Brokerages.Services.OrderPolling.Models; @@ -306,7 +312,7 @@ namespace QuantConnect.Brokerages.Services.OrderPolling.Models; /// and passes it to the service, which compares it with the last snapshot seen for the same order and /// reports only what is new. /// -public class BrokerOrderState +public class BrokerageOrderSnapshot { /// The brokerage order id. Some brokers give every combo leg its own id, some give the /// whole combo one id; the snapshot carries whatever the broker uses. @@ -321,18 +327,18 @@ public class BrokerOrderState /// The price the broker reports for the fills. Null when the read does not carry it. public decimal? FillPrice { get; set; } - /// When the brokerage reported this state, in UTC. + /// When the brokerage reported this snapshot, in UTC. public DateTime TimeUtc { get; set; } /// The broker's own words for a closing status, e.g. the reject reason. public string Message { get; set; } - /// The always-known facts positionally - id, status, time - with the fill numbers and the - /// message defaulting to null. An empty constructor stays for filling through the properties, and an - /// overload takes the message without the fill numbers. - public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, + /// The always-known facts positionally - id and status - with the time defaulting to + /// and the fill numbers and message to null; an overload takes the + /// message without the fill numbers. + public BrokerageOrderSnapshot(string brokerageOrderId, OrderStatus status, DateTime? timeUtc = null, decimal? filledQuantity = null, decimal? fillPrice = null, string message = null); - public BrokerOrderState(string brokerageOrderId, OrderStatus status, DateTime timeUtc, string message); + public BrokerageOrderSnapshot(string brokerageOrderId, OrderStatus status, DateTime? timeUtc, string message); } ``` @@ -457,7 +463,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// One read of the broker, giving the states the sweep saw. The loop calls it every /// poll interval, hands each state to the message handler, and counts a throw as one failed sweep. - protected abstract IEnumerable Sweep(); + protected abstract IEnumerable Sweep(); /// A copy of the ids a sweep still has to read: everything tracked whose end was not /// reported yet. @@ -489,7 +495,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// Watches a brokerage order id, seeded with what another path already reported, so the /// next poll does not repeat it. Used for orders adopted at startup, for a submit reported from /// the request path, and to move state onto the new id of a replace. - public void Watch(string brokerageId, BrokerOrderState lastSeen); + public void Watch(string brokerageId, BrokerageOrderSnapshot lastSeen); /// Watches the new brokerage order id of a replace and drops the replaced id in the same /// step, so the first state to carry the new id reports the update submit. The new id starts with @@ -501,23 +507,23 @@ public abstract class BrokerageOrderPollingService : IDisposable /// Records what another path already reported for an order, so the next poll does not /// repeat it. Called by the streaming path while the stream lives. - public void UpdateOrderState(string brokerageId, BrokerOrderState orderState); + public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderState); /// The last state seen for an order, from any path. The streaming path reads it for its /// own duplicate check, and a replace reads it to move the state to the new id. - public bool TryGetLastOrderState(string brokerageId, out BrokerOrderState lastSeen); + public bool TryGetLastOrderState(string brokerageId, out BrokerageOrderSnapshot lastSeen); /// /// Compares a snapshot with the last state seen for the same order and raises /// with what is new. The constructor registers it on the message /// handler, so polled orders queue behind an order request that holds the stream lock. /// - public void ProcessOrderState(BrokerOrderState orderState); + public void ProcessOrderState(BrokerageOrderSnapshot orderState); /// The whole handover from a stream to polling, in the only safe order: process what the /// stream already delivered, pre-load one watch per open Lean order, then start the loop. A null /// callback pre-loads nothing. See "Seed before Start". - public void Start(Func preLoadOpenOrders); + public void Start(Func preLoadOpenOrders); public void Start(); public void Stop(); @@ -532,7 +538,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// public class PerOrderIdPollingService : BrokerageOrderPollingService { - public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, + public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); } @@ -541,7 +547,7 @@ public class PerOrderIdPollingService : BrokerageOrderPollingService /// public class AllOrdersPollingService : BrokerageOrderPollingService { - public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, + public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); } ``` @@ -553,11 +559,11 @@ The watch registry, the compare with the last state seen and the seeding of orde The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `Sweep`: -- **`PerOrderIdPollingService`** — `Func`: the sweep loops the watched ids and calls the +- **`PerOrderIdPollingService`** — `Func`: the sweep loops the watched ids and calls the read once per id. Nothing watched, nothing requested. Public.com's replaced service had exactly this constructor — `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)` — and its adoption passes the same read to `CreateOrderPollingService`. -- **`AllOrdersPollingService`** — `Func>`: the sweep calls the read once, and the +- **`AllOrdersPollingService`** — `Func>`: the sweep calls the read once, and the read returns everything the broker lists. Two classes instead of one class with both reads, because a bulk read does not fit a per-id shape: called once @@ -612,7 +618,7 @@ way today. differ had no way to push both through one lock. Schwab works around it with a marker interface: its stream model and its polled model both implement `IOrderUpdateMessage`, and the handler is typed to that. This works inside one plugin that owns both models, and it cannot be the shared answer — every plugin that adopts the service would -have to add the interface to its own wire models, and the core `BrokerOrderState` cannot implement a per-plugin +have to add the interface to its own wire models, and the core `BrokerageOrderSnapshot` cannot implement a per-plugin interface. So a second, non-generic `BrokerageConcurrentMessageHandler` ships with the service, in the same file @@ -623,7 +629,7 @@ register, one per message type, and every source enqueues through one `HandleNew ```csharp _messageHandler = new BrokerageConcurrentMessageHandler(concurrencyEnabled); _messageHandler.Register(OnAccountContent); // the stream's own type -// the polling service registers its own BrokerOrderState listener itself, in its constructor +// the polling service registers its own BrokerageOrderSnapshot listener itself, in its constructor // one method for both sources _messageHandler.HandleNewMessage(accountContent); @@ -1025,7 +1031,7 @@ forwards and its dispose line: ```csharp CreateOrderPollingService(ReadOrderState, _messageHandler, _orderProvider, pollInterval: OrderPollingInterval); -// BrokerOrderState ReadOrderState(string brokerageId): get-order by id, a 404 maps to null. +// BrokerageOrderSnapshot ReadOrderState(string brokerageId): get-order by id, a 404 maps to null. // Public keeps its own key, public-order-poll-interval-ms, default 1000 ms. A plugin that passes no // interval gets the shared brokerage-order-poll-interval-ms default, 3000 ms. ``` @@ -1213,7 +1219,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth could only emit `Submitted`, and every brokerage with richer data had to subclass the service and override the diff. The snapshot carries the same numbers the plugins already read and today throw away, so the subclass and the override are gone. -- **A core interface the wire model implements** (`OrderResponse : IBrokerOrderState`), so the plugin passes its +- **A core interface the wire model implements** (`OrderResponse : IBrokerageOrderSnapshot`), so the plugin passes its API model straight in with no conversion. Checked against all eight surveyed plugins and rejected on the evidence. Two cannot implement it at all: IB's order model is compiled into the vendor `CSharpAPI.dll`, and Alpaca's `JsonOrder` is `internal sealed` inside the SDK — both would need a wrapper class, which is the same @@ -1226,7 +1232,7 @@ streaming, and every test asserts the same lifecycle on both. What proved worth class the plugin fills is one pattern that works for all eight. - **A marker interface as the message handler's type** — Schwab's current answer to two message types in one handler (`IOrderUpdateMessage`). As the shared answer it fails the same way the core interface does: every - plugin edits its wire models, and the core `BrokerOrderState` cannot implement a per-plugin interface. + plugin edits its wire models, and the core `BrokerageOrderSnapshot` cannot implement a per-plugin interface. Replaced by the non-generic multi-source handler (see "One lock for two message types"). - **A dual-generic handler**, `BrokerageConcurrentMessageHandler` with the stream type and the polled type. Rejected: every existing plugin migrates to the new shape even with no polling, a plugin with no stream (IB) @@ -1278,8 +1284,8 @@ layer — one abstract brokerage class that owns the service and asks the plugin public abstract class BaseWebSocketsAndPollingServiceBrokerage : BaseWebsocketsBrokerage { // owns the service, forwards its events, disposes it with the brokerage - protected abstract IEnumerable ReadOrderStates(); // or a per-id read - protected virtual BrokerOrderState ToSeedState(Order order) => null; + protected abstract IEnumerable ReadOrderStates(); // or a per-id read + protected virtual BrokerageOrderSnapshot ToSeedState(Order order) => null; } ``` From 1954e827e3ba3699bd51b092a020de3553b0893e Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 01:18:09 +0300 Subject: [PATCH 20/25] refactor: clearer polling names, ctor chains and ms defaults - SingleOrderPollingService and BulkOrdersPollingService, base takes the Base prefix - callbacks are getBrokerageOrderById and getAllBrokerageOrders; watchTimeout is notificationTimeout - both mode classes chain three constructors instead of optional parameters - defaults read 3 * 1000 and 60 * 1000 ms; usage remarks with a wiring example on both modes --- Brokerages/Brokerage.cs | 36 ++--- .../OrderPolling/AllOrdersPollingService.cs | 60 --------- ...cs => BaseBrokerageOrderPollingService.cs} | 38 +++--- .../OrderPolling/BulkOrdersPollingService.cs | 101 ++++++++++++++ .../BrokerageOrderNeverNotifiedEventArgs.cs | 2 +- .../Models/BrokerageOrderSnapshot.cs | 2 +- .../OrderPolling/PerOrderIdPollingService.cs | 88 ------------ .../OrderPolling/SingleOrderPollingService.cs | 127 ++++++++++++++++++ .../BrokerageOrderPollingServiceTests.cs | 24 ++-- 9 files changed, 279 insertions(+), 199 deletions(-) delete mode 100644 Brokerages/Services/OrderPolling/AllOrdersPollingService.cs rename Brokerages/Services/OrderPolling/{BrokerageOrderPollingService.cs => BaseBrokerageOrderPollingService.cs} (96%) create mode 100644 Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs delete mode 100644 Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs create mode 100644 Brokerages/Services/OrderPolling/SingleOrderPollingService.cs diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index 5840ec2d34ad..1fbb359b9696 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -357,7 +357,7 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// its own lifecycle points; /// disposes it. /// - protected BrokerageOrderPollingService OrderPollingService { get; private set; } + protected BaseBrokerageOrderPollingService OrderPollingService { get; private set; } /// /// Returns true while the order polling service is running @@ -365,48 +365,48 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC protected bool IsOrderPolling => OrderPollingService?.IsPolling == true; /// - /// Creates a , for a broker with a get-order endpoint, and wires it + /// Creates a , for a broker with a get-order endpoint, and wires it /// into this brokerage: polled order events reach , polling outage warnings /// reach , and an order the broker never reports goes to /// . The created service is kept in . /// - /// Reads the current state of one order by its brokerage id. A null + /// Reads the current state of one order by its brokerage id. A null /// return means the broker does not know the id. /// The brokerage's message handler; the service registers itself and /// enqueues every polled state through it. Null processes each state directly. /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before - /// is called. Null falls back to one minute. + /// How long a watched order may stay unreported before + /// is called. Null takes 60000 ms. /// The created and wired service. - protected PerOrderIdPollingService CreateOrderPollingService(Func readOrder, + protected SingleOrderPollingService CreateOrderPollingService(Func getBrokerageOrderById, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, - TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null) { - var service = new PerOrderIdPollingService(readOrder, messageHandler, orderProvider, pollInterval, watchTimeout); + var service = new SingleOrderPollingService(getBrokerageOrderById, messageHandler, orderProvider, pollInterval, notificationTimeout); WireOrderPollingService(service); return service; } /// - /// Creates an , for a broker with only a bulk orders endpoint, and + /// Creates an , for a broker with only a bulk orders endpoint, and /// wires it into this brokerage the same way as the per-order overload. /// - /// Reads every order the broker lists, one state per brokerage order id. + /// Reads every order the broker lists, one state per brokerage order id. /// The brokerage's message handler; the service registers itself and /// enqueues every polled state through it. Null processes each state directly. /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before - /// is called. Null falls back to one minute. + /// How long a watched order may stay unreported before + /// is called. Null takes 60000 ms. /// The created and wired service. - protected AllOrdersPollingService CreateOrderPollingService(Func> readAllOrders, + protected BulkOrdersPollingService CreateOrderPollingService(Func> getAllBrokerageOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, - TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null) { - var service = new AllOrdersPollingService(readAllOrders, messageHandler, orderProvider, pollInterval, watchTimeout); + var service = new BulkOrdersPollingService(getAllBrokerageOrders, messageHandler, orderProvider, pollInterval, notificationTimeout); WireOrderPollingService(service); return service; } @@ -415,7 +415,7 @@ protected AllOrdersPollingService CreateOrderPollingService(Func /// The service one of the create overloads built. - private void WireOrderPollingService(BrokerageOrderPollingService service) + private void WireOrderPollingService(BaseBrokerageOrderPollingService service) { if (OrderPollingService != null) { @@ -428,11 +428,11 @@ private void WireOrderPollingService(BrokerageOrderPollingService service) OrderPollingService = service; Log.Trace($"Brokerage.WireOrderPollingService(): {Name} created a {service.GetType().Name}: " + - $"poll interval {service.PollInterval.TotalMilliseconds}ms, watch timeout {service.WatchTimeout.TotalSeconds}s."); + $"poll interval {service.PollInterval.TotalMilliseconds}ms, notification timeout {service.NotificationTimeout.TotalSeconds}s."); } /// - /// Called when the broker never reported a watched order for the whole watch timeout. The default + /// Called when the broker never reported a watched order for the whole notification timeout. The default /// sends one warning message; override it to decide what the silence means for this broker. /// /// The brokerage order id and how long it was watched. diff --git a/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs b/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs deleted file mode 100644 index 6b80bcc14bc4..000000000000 --- a/Brokerages/Services/OrderPolling/AllOrdersPollingService.cs +++ /dev/null @@ -1,60 +0,0 @@ -/* - * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. - * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. -*/ - -using System; -using QuantConnect.Securities; -using System.Collections.Generic; -using QuantConnect.Brokerages.Services.OrderPolling.Models; - -namespace QuantConnect.Brokerages.Services.OrderPolling -{ - /// - /// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched, and the - /// read returns everything the broker lists. - /// - public class AllOrdersPollingService : BrokerageOrderPollingService - { - /// - /// Reads every order the broker lists. - /// - private readonly Func> _readAllOrders; - - /// - /// Creates a new . - /// - /// Reads every order the broker lists, one state per brokerage order id. - /// The brokerage's message handler; the service registers itself and - /// enqueues every polled state through it. Null processes each state directly. - /// Resolves brokerage order ids to Lean orders. - /// How long the loop sleeps between sweeps. Null falls back to the - /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before - /// is raised. Null falls back to one minute. - public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, - IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) - : base(messageHandler, orderProvider, pollInterval, watchTimeout) - { - _readAllOrders = readAllOrders; - } - - /// - /// Calls the read once for the whole sweep. - /// - protected override IEnumerable Sweep() - { - return _readAllOrders(); - } - } -} diff --git a/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs b/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs similarity index 96% rename from Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs rename to Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs index bd9ec9db6081..4111961cf121 100644 --- a/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs @@ -33,9 +33,9 @@ namespace QuantConnect.Brokerages.Services.OrderPolling /// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order /// the broker never replied about. This base class owns everything both modes share - the loop, the /// watch registry, the compare and the events. What one sweep reads is the subclass: - /// or . + /// or . /// - public abstract class BrokerageOrderPollingService : IDisposable + public abstract class BaseBrokerageOrderPollingService : IDisposable { /// /// How many sweeps in a row have to fail before the failure is reported through . @@ -45,7 +45,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// /// Guards the registry and the polling task against the poll loop, the handler thread inside /// , the order threads through / - /// / , and the watch-timeout check. + /// / , and the notification-timeout check. /// private readonly object _lock = new object(); @@ -103,7 +103,7 @@ public abstract class BrokerageOrderPollingService : IDisposable public event EventHandler> OrderEvents; /// - /// A watched order that nothing reported for of polling. Raised once; + /// A watched order that nothing reported for of polling. Raised once; /// the id is unwatched with it. The brokerage decides what the silence means. /// public event EventHandler BrokerageOrderNeverNotified; @@ -128,7 +128,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// How long a watched order may stay completely unreported, in polling time, before /// is raised for it. /// - public TimeSpan WatchTimeout { get; } + public TimeSpan NotificationTimeout { get; } /// /// Initializes what both modes share: the message handler wiring, the order provider, and the two @@ -142,10 +142,10 @@ public abstract class BrokerageOrderPollingService : IDisposable /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between sweeps. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before - /// is raised. Null falls back to one minute. - protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, - TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) + /// How long a watched order may stay unreported before + /// is raised. Null takes 60000 ms. + protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, + TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null) { if (messageHandler != null) { @@ -158,8 +158,8 @@ protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler message _route = ProcessOrderState; } _orderProvider = orderProvider; - PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3000)); - WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); + PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3 * 1000)); + NotificationTimeout = notificationTimeout ?? TimeSpan.FromMilliseconds(60 * 1000); } /// @@ -192,7 +192,7 @@ protected List GetWatchedBrokerageIds() /// /// Watches a brokerage order id, with nothing seen for it yet, so the first state to carry the id - /// acknowledges the order and of silence raises + /// acknowledges the order and of silence raises /// . Idempotent: watching an already-watched id never overwrites /// its state. /// @@ -347,7 +347,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) var brokerageId = orderState.BrokerageOrderId; - // record the id as seen: the broker knows the order, so the watch timeout stops counting + // record the id as seen: the broker knows the order, so the notification timeout stops counting lock (_lock) { if (_orderStates.TryGetValue(brokerageId, out var seenEntry)) @@ -649,7 +649,7 @@ private async Task PollLoop(CancellationToken cancellationToken) // the broker nothing, so it must not count against a watched order if (!cancellationToken.IsCancellationRequested) { - CheckWatchTimeouts(); + CheckNotificationTimeouts(); } } catch (Exception ex) @@ -682,10 +682,10 @@ private async Task PollLoop(CancellationToken cancellationToken) /// /// Counts one interval of silence for every watched order the broker never reported, and raises - /// once for each one that reached the watch timeout. Called only + /// once for each one that reached the notification timeout. Called only /// after a successful sweep, because only a read that succeeded proves the silence is real. /// - private void CheckWatchTimeouts() + private void CheckNotificationTimeouts() { List<(string BrokerageId, TimeSpan WatchDuration)> expired = null; lock (_lock) @@ -698,7 +698,7 @@ private void CheckWatchTimeouts() } entry.UnacknowledgedDuration += PollInterval; - if (entry.UnacknowledgedDuration >= WatchTimeout) + if (entry.UnacknowledgedDuration >= NotificationTimeout) { (expired ??= []).Add((brokerageId, entry.UnacknowledgedDuration)); } @@ -753,7 +753,7 @@ private class OrderStateEntry public bool TerminalReported; /// - /// Set by : the watch timeout only applies to explicitly watched orders. + /// Set by : the notification timeout only applies to explicitly watched orders. /// public bool Watched; @@ -765,7 +765,7 @@ private class OrderStateEntry /// /// Set once anything carried the id: a polled state, a stream write, or a seed. Stops the - /// watch timeout. + /// notification timeout. /// public bool Acknowledged; diff --git a/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs b/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs new file mode 100644 index 000000000000..6f4db45a410d --- /dev/null +++ b/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs @@ -0,0 +1,101 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using QuantConnect.Securities; +using System.Collections.Generic; +using QuantConnect.Brokerages.Services.OrderPolling.Models; + +namespace QuantConnect.Brokerages.Services.OrderPolling +{ + /// + /// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched, and the + /// read returns everything the broker lists. + /// + /// + /// Use it when the broker cannot be asked about one order id, or when one request already returns the + /// whole account cheaply. The sweep runs even with nothing watched, so it can be the order path of a + /// whole run. + /// + /// CreateOrderPollingService(() => _api.GetAllOrders().Select(ToOrderSnapshot), _messageHandler, _orderProvider); + /// + /// + public class BulkOrdersPollingService : BaseBrokerageOrderPollingService + { + /// + /// Reads every order the broker lists. + /// + private readonly Func> _getAllBrokerageOrders; + + /// + /// Creates a new with the default poll interval and + /// notification timeout. + /// + /// Reads every order the broker lists, one snapshot per brokerage order id. + /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. + /// Resolves brokerage order ids to Lean orders. + public BulkOrdersPollingService( + Func> getAllBrokerageOrders, + BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider) + : this(getAllBrokerageOrders, messageHandler, orderProvider, pollInterval: null, notificationTimeout: null) + { + } + + /// + /// Creates a new with the default notification timeout. + /// + /// Reads every order the broker lists, one snapshot per brokerage order id. + /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. + /// Resolves brokerage order ids to Lean orders. + /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + public BulkOrdersPollingService( + Func> getAllBrokerageOrders, + BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider, + TimeSpan? pollInterval) + : this(getAllBrokerageOrders, messageHandler, orderProvider, pollInterval, notificationTimeout: null) + { + } + + /// + /// Creates a new . + /// + /// Reads every order the broker lists, one snapshot per brokerage order id. + /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. + /// Resolves brokerage order ids to Lean orders. + /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + /// The silence that raises + /// for a watched order. Null takes 60000 ms. + public BulkOrdersPollingService( + Func> getAllBrokerageOrders, + BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider, + TimeSpan? pollInterval, + TimeSpan? notificationTimeout) + : base(messageHandler, orderProvider, pollInterval, notificationTimeout) + { + _getAllBrokerageOrders = getAllBrokerageOrders; + } + + /// + /// Calls the read once for the whole sweep. + /// + protected override IEnumerable Sweep() + { + return _getAllBrokerageOrders(); + } + } +} diff --git a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs index d669451fd696..e3db500ea705 100644 --- a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs +++ b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs @@ -19,7 +19,7 @@ namespace QuantConnect.Brokerages.Services.OrderPolling.Models { /// - /// Raised when no read saw a watched brokerage order id for the whole watch timeout. A question, not + /// Raised when no read saw a watched brokerage order id for the whole notification timeout. A question, not /// a verdict: the order may never have reached the broker, or closed before the first sweep. The /// brokerage decides what to do next. /// diff --git a/Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs b/Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs index 064eca3727d7..7d10206dd5a1 100644 --- a/Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs +++ b/Brokerages/Services/OrderPolling/Models/BrokerageOrderSnapshot.cs @@ -20,7 +20,7 @@ namespace QuantConnect.Brokerages.Services.OrderPolling.Models { /// /// One order, as the brokerage last saw it. The plugin converts its broker model into this shape; - /// diffs it against the last snapshot seen and reports + /// diffs it against the last snapshot seen and reports /// only what is new. Null in an optional field means "my read does not know", never "zero". /// public class BrokerageOrderSnapshot diff --git a/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs b/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs deleted file mode 100644 index 65a9dfd7a766..000000000000 --- a/Brokerages/Services/OrderPolling/PerOrderIdPollingService.cs +++ /dev/null @@ -1,88 +0,0 @@ -/* - * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. - * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. -*/ - -using System; -using QuantConnect.Logging; -using QuantConnect.Securities; -using System.Collections.Generic; -using QuantConnect.Brokerages.Services.OrderPolling.Models; - -namespace QuantConnect.Brokerages.Services.OrderPolling -{ - /// - /// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id - - /// no request when nothing is watched. A null return means the broker does not know the id, so the - /// watch timeout keeps counting. - /// - public class PerOrderIdPollingService : BrokerageOrderPollingService - { - /// - /// Reads the current state of one order by its brokerage id. - /// - private readonly Func _readOrder; - - /// - /// Creates a new . - /// - /// Reads the current state of one order by its brokerage id. A null - /// return means the broker does not know the id. - /// The brokerage's message handler; the service registers itself and - /// enqueues every polled state through it. Null processes each state directly. - /// Resolves brokerage order ids to Lean orders. - /// How long the loop sleeps between sweeps. Null falls back to the - /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before - /// is raised. Null falls back to one minute. - public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, - IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null) - : base(messageHandler, orderProvider, pollInterval, watchTimeout) - { - _readOrder = readOrder; - } - - /// - /// Calls the read once per watched brokerage id. One id whose read throws is logged and skipped, - /// so it cannot starve the other watched orders; the sweep only counts as failed when every read - /// of the sweep failed. - /// - protected override IEnumerable Sweep() - { - var brokerageIds = GetWatchedBrokerageIds(); - var orderStates = new List(brokerageIds.Count); - var failedReadCount = 0; - var lastError = default(Exception); - foreach (var brokerageId in brokerageIds) - { - try - { - orderStates.Add(_readOrder(brokerageId)); - } - catch (Exception ex) - { - failedReadCount++; - lastError = ex; - Log.Error($"{nameof(PerOrderIdPollingService)}.{nameof(Sweep)}(): failed to read order '{brokerageId}': {ex.Message}"); - } - } - - if (failedReadCount > 0 && failedReadCount == brokerageIds.Count) - { - throw lastError; - } - - return orderStates; - } - } -} diff --git a/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs b/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs new file mode 100644 index 000000000000..43d1b53da089 --- /dev/null +++ b/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs @@ -0,0 +1,127 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; +using QuantConnect.Logging; +using QuantConnect.Securities; +using System.Collections.Generic; +using QuantConnect.Brokerages.Services.OrderPolling.Models; + +namespace QuantConnect.Brokerages.Services.OrderPolling +{ + /// + /// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id - + /// no request when nothing is watched. A null return means the broker does not know the id, so the + /// notification timeout keeps counting. + /// + /// + /// Use it when the broker can be asked about one order id. The sweep reads only the watched orders, + /// so an idle account sends no requests and rate limits stay untouched. + /// + /// CreateOrderPollingService(id => ToOrderSnapshot(_api.GetOrderById(id)), _messageHandler, _orderProvider); + /// + /// + public class SingleOrderPollingService : BaseBrokerageOrderPollingService + { + /// + /// Reads the current state of one order by its brokerage id. + /// + private readonly Func _getBrokerageOrderById; + + /// + /// Creates a new with the default poll interval and + /// notification timeout. + /// + /// Reads one order by its brokerage id. A null return means the broker does not know the id. + /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. + /// Resolves brokerage order ids to Lean orders. + public SingleOrderPollingService( + Func getBrokerageOrderById, + BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider) + : this(getBrokerageOrderById, messageHandler, orderProvider, pollInterval: null, notificationTimeout: null) + { + } + + /// + /// Creates a new with the default notification timeout. + /// + /// Reads one order by its brokerage id. A null return means the broker does not know the id. + /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. + /// Resolves brokerage order ids to Lean orders. + /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + public SingleOrderPollingService( + Func getBrokerageOrderById, + BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider, + TimeSpan? pollInterval) + : this(getBrokerageOrderById, messageHandler, orderProvider, pollInterval, notificationTimeout: null) + { + } + + /// + /// Creates a new . + /// + /// Reads one order by its brokerage id. A null return means the broker does not know the id. + /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. + /// Resolves brokerage order ids to Lean orders. + /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + /// The silence that raises + /// for a watched order. Null takes 60000 ms. + public SingleOrderPollingService( + Func getBrokerageOrderById, + BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider, + TimeSpan? pollInterval, + TimeSpan? notificationTimeout) + : base(messageHandler, orderProvider, pollInterval, notificationTimeout) + { + _getBrokerageOrderById = getBrokerageOrderById; + } + + /// + /// Calls the read once per watched brokerage id. One id whose read throws is logged and skipped, + /// so it cannot starve the other watched orders; the sweep only counts as failed when every read + /// of the sweep failed. + /// + protected override IEnumerable Sweep() + { + var brokerageIds = GetWatchedBrokerageIds(); + var orderStates = new List(brokerageIds.Count); + var failedReadCount = 0; + var lastError = default(Exception); + foreach (var brokerageId in brokerageIds) + { + try + { + orderStates.Add(_getBrokerageOrderById(brokerageId)); + } + catch (Exception ex) + { + failedReadCount++; + lastError = ex; + Log.Error($"{nameof(SingleOrderPollingService)}.{nameof(Sweep)}(): failed to read order '{brokerageId}': {ex.Message}"); + } + } + + if (failedReadCount > 0 && failedReadCount == brokerageIds.Count) + { + throw lastError; + } + + return orderStates; + } + } +} diff --git a/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs index 52624da0d479..a92984dd36fd 100644 --- a/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs @@ -30,7 +30,7 @@ namespace QuantConnect.Tests.Brokerages.Services.OrderPolling public class BrokerageOrderPollingServiceTests { private OrderProvider _orderProvider; - private AllOrdersPollingService _service; + private BulkOrdersPollingService _service; private List _orderEvents; [SetUp] @@ -39,8 +39,8 @@ public void SetUp() _orderProvider = new OrderProvider(); _orderEvents = new List(); // the read is unused: these tests drive the diff directly through ProcessOrderState - _service = new AllOrdersPollingService(() => Array.Empty(), messageHandler: null, _orderProvider, - pollInterval: TimeSpan.FromMilliseconds(50), watchTimeout: TimeSpan.FromMilliseconds(120)); + _service = new BulkOrdersPollingService(() => Array.Empty(), messageHandler: null, _orderProvider, + pollInterval: TimeSpan.FromMilliseconds(50), notificationTimeout: TimeSpan.FromMilliseconds(120)); _service.OrderEvents += (_, orderEvents) => _orderEvents.AddRange(orderEvents); } @@ -453,17 +453,17 @@ public void WatchNeverOverwritesExistingState() } [Test] - public void WatchTimeoutFiresOnceAndUnwatchesTheId() + public void NotificationTimeoutFiresOnceAndUnwatchesTheId() { var order = AddOrder(100m, "77"); using var neverNotified = new ManualResetEventSlim(false); var raised = new List(); - using var service = new PerOrderIdPollingService( + using var service = new SingleOrderPollingService( _ => null, // the broker never knows the id messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25), - watchTimeout: TimeSpan.FromMilliseconds(75)); + notificationTimeout: TimeSpan.FromMilliseconds(75)); service.BrokerageOrderNeverNotified += (_, eventArgs) => { lock (raised) @@ -476,7 +476,7 @@ public void WatchTimeoutFiresOnceAndUnwatchesTheId() service.Watch("77"); service.Start(); - Assert.IsTrue(neverNotified.Wait(TimeSpan.FromSeconds(5)), "the watch timeout never fired"); + Assert.IsTrue(neverNotified.Wait(TimeSpan.FromSeconds(5)), "the notification timeout never fired"); // let a few more sweeps run: the id was unwatched with the event, so it fires exactly once Thread.Sleep(200); @@ -510,12 +510,12 @@ public void NeverNotifiedArgsPrintTheOrderAndTheWatchDuration() public void AcknowledgedWatchNeverTimesOut() { var fired = 0; - using var service = new PerOrderIdPollingService( + using var service = new SingleOrderPollingService( _ => null, messageHandler: null, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(25), - watchTimeout: TimeSpan.FromMilliseconds(75)); + notificationTimeout: TimeSpan.FromMilliseconds(75)); service.BrokerageOrderNeverNotified += (_, _) => Interlocked.Increment(ref fired); // the stream acknowledged the order right after it was watched @@ -535,7 +535,7 @@ public void RepeatedReadFailuresRaiseOneWarningPerOutage() var fail = true; var warnings = new List(); using var warned = new AutoResetEvent(false); - using var service = new AllOrdersPollingService( + using var service = new BulkOrdersPollingService( () => fail ? throw new Exception("read failed") : Array.Empty(), messageHandler: null, _orderProvider, @@ -581,7 +581,7 @@ public void PerOrderIdSweepReadsOnlyWatchedIdsAndProcessesTheStates() var readIds = new List(); using var processed = new ManualResetEventSlim(false); // no message handler: the loop hands each state straight into the diff - using var service = new PerOrderIdPollingService( + using var service = new SingleOrderPollingService( brokerageId => { lock (readIds) @@ -676,7 +676,7 @@ public void StartWithPreLoadDrainsTheMessageHandlerBeforeSeeding() AddOrder(233m, "42", OrderStatus.Submitted); var steps = new List(); using var handler = new BrokerageConcurrentMessageHandler(); - using var service = new AllOrdersPollingService(() => Array.Empty(), handler, _orderProvider, + using var service = new BulkOrdersPollingService(() => Array.Empty(), handler, _orderProvider, pollInterval: TimeSpan.FromMilliseconds(50)); service.OrderEvents += (_, orderEvents) => { From fa6681538e341ac1395491e50e5998864c627eb1 Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 01:18:09 +0300 Subject: [PATCH 21/25] docs: record the polling api polish in the adr --- .../0001-brokerage-order-polling-service.md | 78 +++++++++++-------- 1 file changed, 45 insertions(+), 33 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index f714efe945f7..5354df16e46c 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -19,7 +19,7 @@ connection running next to a streaming one, on real CharlesSchwab accounts. The showed up as designed: a market order already filled on its first listing reports the submit and the fill in one batch, and the executions of one sweep arrive as one event. -Second plugin adopted - 2026-08-15: Public.com runs on `PerOrderIdPollingService` +Second plugin adopted - 2026-08-15: Public.com runs on `SingleOrderPollingService` (`Lean.Brokerages.Public`, draft PR #6). Its own service class, diff and snapshot model are deleted; the plugin keeps the get-order read and the model-to-state mapping. Its same-id replace stays plugin-side, so `WatchReplacement` is not wired, and a get-order 404 maps to a null state - the contract's "the broker @@ -33,7 +33,7 @@ mode, the service as a protected `OrderPollingService` property with `IsOrderPol CharlesSchwab and Public.com were moved onto it: the creation, the three event forwards and the dispose call left both plugins (see "Wiring, per plugin"). -Third plugin adopted - 2026-08-17: Tradier runs on `PerOrderIdPollingService` +Third plugin adopted - 2026-08-17: Tradier runs on `SingleOrderPollingService` (`Lean.Brokerages.Tradier`, draft PR #54). Its fill timer, `CheckForFills` diff, order cache and unknown-id verification are deleted (~230 lines); the plugin keeps the get-order read and the mapping. Tradier is the first adopter that splits orders across zero: the legs chain from @@ -78,6 +78,16 @@ property-fill constructor left with it: every builder goes through the construct constructors default the time to `DateTime.UtcNow` when none is passed. All three adopters and the tests renamed and simplified with it. +Mode classes renamed - 2026-08-19: `PerOrderIdPollingService` and `AllOrdersPollingService` became +`SingleOrderPollingService` and `BulkOrdersPollingService` - named for the request shape one sweep +sends: one order per request, or one request for everything the broker lists. The abstract parent took +Lean's `Base` prefix with it: `BaseBrokerageOrderPollingService`. + +Constructor chains and clearer names - 2026-08-19: `watchTimeout` became `notificationTimeout` - the +silence that raises `BrokerageOrderNeverNotified`; the read callbacks became `getBrokerageOrderById` and +`getAllBrokerageOrders`; both mode classes offer three chained constructors instead of optional +parameters; and the defaults read `3 * 1000` and `60 * 1000` milliseconds in code and docs alike. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -93,7 +103,7 @@ happened to an order, and it is not reliable.** It can be absent, it can be take and it can stay up while quietly missing an update. In all four shapes the broker still knows the answer over HTTP, and nobody asks. -This document proposes one helper class in Lean core, `BrokerageOrderPollingService`, that any brokerage can create +This document proposes one helper class in Lean core, `BaseBrokerageOrderPollingService`, that any brokerage can create and use. The brokerage picks one of two classes — read one order by its brokerage id, or read all orders — and hands the service a read callback that converts each order from the broker's own model into one shared snapshot shape. Every N seconds the service runs the read, and the snapshots travel through the brokerage's message handler @@ -262,8 +272,8 @@ choice, made when it picks the class. ### Where it lives -`Lean/Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs` — the base class — with -`PerOrderIdPollingService.cs` and `AllOrdersPollingService.cs` next to it, in namespace +`Lean/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs` — the base class — with +`SingleOrderPollingService.cs` and `BulkOrdersPollingService.cs` next to it, in namespace `QuantConnect.Brokerages.Services.OrderPolling`. The data shapes live one level deeper, in `Models/` with the matching namespace suffix: `BrokerageOrderSnapshot.cs` and `BrokerageOrderNeverNotifiedEventArgs.cs`. @@ -368,7 +378,7 @@ legFill = leanOrder.Quantity * newPart / abs(leanOrder.GroupOrderManager.Quantit The state has no quantity field because the service does not need one: it already knows the brokerage id, so it reads the group quantity from the Lean orders themselves. That number equals the broker's own order quantity — the group quantity is exactly what the combo was placed with. Public's replaced diff split fills this way, and -the rule moved into the service's fan-out (`Brokerages/Services/OrderPolling/BrokerageOrderPollingService.cs`). One rule for the mapping follows: +the rule moved into the service's fan-out (`Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs`). One rule for the mapping follows: `FilledQuantity` of a shared-id combo is in strategy units, the same units the group quantity counts in. A worked example, from a real Public.com order — 5 AAPL strangles, one brokerage id, a put leg and a call leg with @@ -449,17 +459,17 @@ namespace QuantConnect.Brokerages.Services.OrderPolling; /// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order /// the broker never replied about. The base class owns everything both modes share — the loop, the /// watch registry, the compare and the events. What one sweep reads is the subclass: -/// or . +/// or . /// -public abstract class BrokerageOrderPollingService : IDisposable +public abstract class BaseBrokerageOrderPollingService : IDisposable { /// Initializes what both modes share: the message handler wiring, the order provider, and /// the two time settings with their defaults. The service wires the handler both ways itself: it /// registers and enqueues every polled snapshot, so one handler /// serializes polled snapshots with everything else the brokerage processes. A null handler routes /// each snapshot straight into . - protected BrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, - TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, + TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null); /// One read of the broker, giving the states the sweep saw. The loop calls it every /// poll interval, hands each state to the message handler, and counts a throw as one failed sweep. @@ -472,7 +482,7 @@ public abstract class BrokerageOrderPollingService : IDisposable /// The order events one snapshot produced. Raised inside , never empty. public event EventHandler> OrderEvents; - /// A watched order that nothing reported for watchTimeout of polling. + /// A watched order that nothing reported for notificationTimeout of polling. /// Raised once; the id is unwatched with it. public event EventHandler BrokerageOrderNeverNotified; @@ -486,7 +496,7 @@ public abstract class BrokerageOrderPollingService : IDisposable public TimeSpan PollInterval { get; } /// How long a watched order may stay completely unreported, in polling time. - public TimeSpan WatchTimeout { get; } + public TimeSpan NotificationTimeout { get; } /// Watches a brokerage order id, with nothing seen for it yet. Idempotent: watching an /// already-watched id never overwrites its state. @@ -533,22 +543,24 @@ public abstract class BrokerageOrderPollingService : IDisposable /// /// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id — /// no request when nothing is watched. A null return means the broker does not know the id, so the -/// watch timeout keeps counting. A read that throws is logged and skipped, so one bad id cannot +/// notification timeout keeps counting. A read that throws is logged and skipped, so one bad id cannot /// starve the others; the sweep only counts as failed when every read of the sweep failed. /// -public class PerOrderIdPollingService : BrokerageOrderPollingService +public class SingleOrderPollingService : BaseBrokerageOrderPollingService { - public PerOrderIdPollingService(Func readOrder, BrokerageConcurrentMessageHandler messageHandler, - IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + public SingleOrderPollingService(Func getBrokerageOrderById, BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider, TimeSpan? pollInterval, TimeSpan? notificationTimeout); + // two shorter overloads chain to it: (read, handler, provider) and (read, handler, provider, pollInterval) } /// /// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched. /// -public class AllOrdersPollingService : BrokerageOrderPollingService +public class BulkOrdersPollingService : BaseBrokerageOrderPollingService { - public AllOrdersPollingService(Func> readAllOrders, BrokerageConcurrentMessageHandler messageHandler, - IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? watchTimeout = null); + public BulkOrdersPollingService(Func> getAllBrokerageOrders, BrokerageConcurrentMessageHandler messageHandler, + IOrderProvider orderProvider, TimeSpan? pollInterval, TimeSpan? notificationTimeout); + // two shorter overloads chain to it: (read, handler, provider) and (read, handler, provider, pollInterval) } ``` @@ -559,11 +571,11 @@ The watch registry, the compare with the last state seen and the seeding of orde The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `Sweep`: -- **`PerOrderIdPollingService`** — `Func`: the sweep loops the watched ids and calls the +- **`SingleOrderPollingService`** — `Func`: the sweep loops the watched ids and calls the read once per id. Nothing watched, nothing requested. Public.com's replaced service had exactly this constructor — `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)` — and its adoption passes the same read to `CreateOrderPollingService`. -- **`AllOrdersPollingService`** — `Func>`: the sweep calls the read once, and the +- **`BulkOrdersPollingService`** — `Func>`: the sweep calls the read once, and the read returns everything the broker lists. Two classes instead of one class with both reads, because a bulk read does not fit a per-id shape: called once @@ -573,7 +585,7 @@ each class clean: one class with both reads would hold a null field for the unus to pick the right read; a subclass holds only its own read. The watch registry serves both modes. In per-id mode it is also the read list. In all-orders mode it only feeds -the watch timeout: the service still checks that every watched id shows up in the snapshots sooner or later. +the notification timeout: the service still checks that every watched id shows up in the snapshots sooner or later. | Plugin | The sweep reads | Scope | Why | | --- | --- | --- | --- | @@ -655,7 +667,7 @@ the polling service moves to the non-generic class, one plugin at a time. `ProcessOrderState` compares the snapshot with the last state it has seen for that brokerage id: ``` -record the id as seen, for the watch timeout +record the id as seen, for the notification timeout find the Lean orders by brokerage id (IOrderProvider.GetOrdersByBrokerageId — a list, because combo legs can share one id and each leg is its own Lean order) none found -> skip and write nothing: not ours, or ours with the id not on the @@ -752,7 +764,7 @@ diffs. With the snapshot the diff is shared, whole, and the plugins keep only th - A snapshot arrives for the id, or the stream records one through `UpdateOrderState` → the order was seen, and the id stays watched until the order closes. - The order is closed in Lean → `Unwatch`, and its state is dropped. -- `watchTimeout` of polling passes and nothing ever carried the id → `BrokerageOrderNeverNotified` is raised once, with +- `notificationTimeout` of polling passes and nothing ever carried the id → `BrokerageOrderNeverNotified` is raised once, with the id, the Lean order behind it when one resolves (null for a placement whose id assignment never happened), and how long it was watched; the args print all of it through `ToString`, and the id is unwatched. @@ -986,7 +998,7 @@ does **not** own the Lean order state: that is read from `IOrderProvider` on eve drifts from what Lean actually knows. Four kinds of thread touch that state: the poll loop, the handler thread inside `ProcessOrderState`, the order -threads through `Watch`/`Unwatch`/`UpdateOrderState`, and the watch-timeout check. One internal lock protects the +threads through `Watch`/`Unwatch`/`UpdateOrderState`, and the notification-timeout check. One internal lock protects the registry from all of them. A per-id sweep copies the watched ids under that lock before it reads the broker, so `PlaceOrder` can watch a new id while a sweep runs. `ProcessOrderState` must not run twice at the same time, and the service does not guard that itself: the message handler already runs it one call at a time, and a plugin @@ -1000,8 +1012,8 @@ plugin that passes nothing gets the shared defaults, resolved once in the base c chain to: ```csharp -PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3000)); -WatchTimeout = watchTimeout ?? TimeSpan.FromMinutes(1); +PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3 * 1000)); +NotificationTimeout = notificationTimeout ?? TimeSpan.FromMilliseconds(60 * 1000); ``` `brokerage-order-poll-interval-ms` is one new generic config entry, so the interval can be tuned once for every @@ -1014,7 +1026,7 @@ brokerage that uses the default. Core helpers already work this way: the message The general shape, for a streaming brokerage with a bulk endpoint. The root `Brokerage` class owns the wiring: one protected create call builds the service, forwards its events onto the brokerage events, stores it in the protected `OrderPollingService` property and hands it to `Dispose`. The read callback's -signature picks the mode — a bulk read can only build an `AllOrdersPollingService`: +signature picks the mode — a bulk read can only build an `BulkOrdersPollingService`: ```csharp // bulk broker: one request per sweep reads the whole account. The service wires itself onto the @@ -1023,7 +1035,7 @@ signature picks the mode — a bulk read can only build an `AllOrdersPollingServ CreateOrderPollingService( () => _apiClient.GetAllOrders().Select(ToOrderState), // read: model -> snapshot _messageHandler, - _orderProvider, pollInterval: TimeSpan.FromSeconds(3), watchTimeout: TimeSpan.FromMinutes(1)); + _orderProvider, pollInterval: TimeSpan.FromSeconds(3), notificationTimeout: TimeSpan.FromMinutes(1)); ``` A per-id broker only changes the read — one call replaces Public.com's construction, its three event @@ -1040,7 +1052,7 @@ What silence means is one override per plugin: `OnBrokerageOrderNeverNotified` d warning built from the brokerage name, and Schwab overrides it to keep the wording its live pilot verified. -InteractiveBrokers, the plugin with no answer today, adopts `AllOrdersPollingService` (it has no per-id request) with +InteractiveBrokers, the plugin with no answer today, adopts `BulkOrdersPollingService` (it has no per-id request) with the thinnest possible mapping — id and status from the orders `reqAllOpenOrders` returns, fill fields left null. The fill numbers are not out of reach: the captured `IBApi.Order` already carries a `FilledQuantity` field, and the paired `orderStatus` callback adds the average fill price, so the mapping can grow later without a new endpoint. @@ -1350,13 +1362,13 @@ stays possible — the helper is additive. Implemented that way on 2026-08-17: t | Risk | What we do about it | | --- | --- | | A plugin maps a broker status to the wrong Lean status | The mapping is the same one its streaming path already needs, written once per plugin and covered by its own tests. The service only emits transitions, so a wrong mapping surfaces once, not as a flood. | -| A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the watch timeout still fires. | +| A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the notification timeout still fires. | | Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier kept this trade-off through its adoption (`TradierBrokerage.cs:1170-1172`); a shorter poll interval narrows it. | | Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order still waits for its first report, and the interval is a constructor argument the plugin picks. | | A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | | A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; Tradier adopted with null — its submit event goes out before the watch begins, so the poll cannot outrun it — and IB's rollout step still names adding one. | | The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | -| The watch timeout cannot tell "never arrived" from "filled instantly" | It does not try. `BrokerageOrderNeverNotified` hands the question to the brokerage, which has the endpoints to answer it. | +| The notification timeout cannot tell "never arrived" from "filled instantly" | It does not try. `BrokerageOrderNeverNotified` hands the question to the brokerage, which has the endpoints to answer it. | | Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | | A plugin starts polling on disconnect and forgets to stop on reconnect | Both paths are one line and sit next to the connection handling the plugin already has. The poll side repeats nothing the registry already holds, so the cost of forgetting is extra requests, not extra events. | @@ -1375,7 +1387,7 @@ stays possible — the helper is additive. Implemented that way on 2026-08-17: t prices become per-sweep prices while the quantities stay exact; Public kept its change-of-average price recovery inside its mapping, so its part prices stay exact too. 4. Tradier (done, further than planned here): the step was a watch for submissions with fills staying on its own - path. The adoption moved the fill path itself onto `PerOrderIdPollingService` and handles the cross-zero split + path. The adoption moved the fill path itself onto `SingleOrderPollingService` and handles the cross-zero split as "A cross-zero order, two ids" describes. Two costs are accepted: a sweep is one gated request per watched order instead of one bulk request — the one-order-per-symbol rule keeps that count small, and an idle account now polls nothing at all — and orders placed outside Lean are ignored, where the old code raised a fatal From 705dc73a33dc2565fadb65eeaf1ae710cf0daa3d Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 16:58:13 +0300 Subject: [PATCH 22/25] refactor: poll vocabulary, tracking entry model and safer dispose - rename Sweep() to GetOrderSnapshots() and drop "sweep" from docs and comments - extract the registry record to Models/OrderTrackingEntry with LastSnapshot - GetWatchedBrokerageIds becomes GetOpenBrokerageIds returning IEnumerable, copying the entries under the lock and filtering outside it - Dispose always disposes the cancellation source - clearer private names: MaxFailedPollsBeforeWarning, _orderEntries, _processSnapshot --- Brokerages/Brokerage.cs | 4 +- .../BaseBrokerageOrderPollingService.cs | 202 +++++++----------- .../OrderPolling/BulkOrdersPollingService.cs | 12 +- .../BrokerageOrderNeverNotifiedEventArgs.cs | 2 +- .../OrderPolling/Models/OrderTrackingEntry.cs | 70 ++++++ .../OrderPolling/SingleOrderPollingService.cs | 24 +-- 6 files changed, 169 insertions(+), 145 deletions(-) create mode 100644 Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index 1fbb359b9696..aa0f4e5e56e4 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -375,7 +375,7 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// The brokerage's message handler; the service registers itself and /// enqueues every polled state through it. Null processes each state directly. /// Resolves brokerage order ids to Lean orders. - /// How long the loop sleeps between sweeps. Null falls back to the + /// How long the loop sleeps between polls. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is called. Null takes 60000 ms. @@ -397,7 +397,7 @@ protected SingleOrderPollingService CreateOrderPollingService(FuncThe brokerage's message handler; the service registers itself and /// enqueues every polled state through it. Null processes each state directly. /// Resolves brokerage order ids to Lean orders. - /// How long the loop sleeps between sweeps. Null falls back to the + /// How long the loop sleeps between polls. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is called. Null takes 60000 ms. diff --git a/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs b/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs index 4111961cf121..76d1ca3b7d9d 100644 --- a/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs @@ -13,46 +13,46 @@ * limitations under the License. */ -using System; -using System.Linq; -using System.Threading; -using QuantConnect.Util; -using QuantConnect.Orders; using QuantConnect.Brokerages.Services.OrderPolling.Models; +using QuantConnect.Configuration; using QuantConnect.Logging; -using System.Threading.Tasks; -using QuantConnect.Securities; +using QuantConnect.Orders; using QuantConnect.Orders.Fees; +using QuantConnect.Securities; +using QuantConnect.Util; +using System; using System.Collections.Generic; -using QuantConnect.Configuration; +using System.Linq; +using System.Threading; +using System.Threading.Tasks; namespace QuantConnect.Brokerages.Services.OrderPolling { /// - /// Reads orders from the brokerage on an interval and turns the returned states into order events. - /// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order - /// the broker never replied about. This base class owns everything both modes share - the loop, the - /// watch registry, the compare and the events. What one sweep reads is the subclass: + /// Reads orders from the brokerage on an interval and turns what it reads into order events. + /// Use it when a brokerage has no order stream, when the stream goes down, or to check an order + /// the broker never replied about. This base class holds what both modes share: the loop, the + /// watch registry, the compare and the events. The subclass decides what one poll reads: /// or . /// public abstract class BaseBrokerageOrderPollingService : IDisposable { /// - /// How many sweeps in a row have to fail before the failure is reported through . + /// How many polls must fail one after another to raise the warning. /// - private const int ConsecutiveFailuresBeforeReport = 3; + private const int MaxFailedPollsBeforeWarning = 3; /// /// Guards the registry and the polling task against the poll loop, the handler thread inside /// , the order threads through / /// / , and the notification-timeout check. /// - private readonly object _lock = new object(); + private readonly Lock _lock = new(); /// /// The registry: per brokerage order id, the last state seen and what was already reported for it. /// - private readonly Dictionary _orderStates = new(); + private readonly Dictionary _orderEntries = []; /// /// The brokerage's message handler, when it has one. The constructor wires it both ways: polled @@ -62,10 +62,10 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable private readonly BrokerageConcurrentMessageHandler _messageHandler; /// - /// Where each state a sweep returns goes: the message handler, or without one straight into + /// Where each state a poll returns goes: the message handler, or without one straight into /// . /// - private readonly Action _route; + private readonly Action _processSnapshot; /// /// Resolves brokerage order ids to Lean orders on every compare, so the service never drifts from @@ -120,7 +120,7 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable public bool IsPolling => _isPolling; /// - /// How long the loop sleeps between sweeps. + /// How long the loop sleeps between polls. /// public TimeSpan PollInterval { get; } @@ -140,7 +140,7 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable /// state straight into - only the poll loop calls it then, so the /// calls are still one at a time. /// Resolves brokerage order ids to Lean orders. - /// How long the loop sleeps between sweeps. Null falls back to the + /// How long the loop sleeps between polls. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. /// How long a watched order may stay unreported before /// is raised. Null takes 60000 ms. @@ -150,12 +150,12 @@ protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler mes if (messageHandler != null) { _messageHandler = messageHandler; - _route = messageHandler.HandleNewMessage; + _processSnapshot = messageHandler.HandleNewMessage; messageHandler.Register(ProcessOrderState); } else { - _route = ProcessOrderState; + _processSnapshot = ProcessOrderState; } _orderProvider = orderProvider; PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3 * 1000)); @@ -163,30 +163,31 @@ protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler mes } /// - /// One read of the broker, giving the states the sweep saw. The loop calls it every - /// , hands each state to the route, and counts a throw as one failed sweep. - /// A null state is skipped: in per-id mode it means the broker does not know the id yet. + /// One read of the broker, giving the current order snapshots. The loop calls it every + /// , hands each snapshot on for processing, and counts a throw as one failed poll. + /// A null snapshot is skipped: in per-id mode it means the broker does not know the id yet. /// - protected abstract IEnumerable Sweep(); + protected abstract IEnumerable GetOrderSnapshots(); /// - /// A copy of the brokerage order ids a sweep still has to read: everything tracked whose end was - /// not reported yet. Taken under the registry lock, so an order placed mid-sweep is picked up by - /// the next one. + /// The brokerage order ids a poll still has to read: everything tracked whose end was not + /// reported yet. The entries are copied under the registry lock and filtered outside it, so + /// the order paths never wait on the filter. /// - protected List GetWatchedBrokerageIds() + protected IEnumerable GetOpenBrokerageIds() { + KeyValuePair[] entries; lock (_lock) { - var brokerageIds = new List(_orderStates.Count); - foreach (var (brokerageId, entry) in _orderStates) + entries = [.. _orderEntries]; + } + + foreach (var (brokerageId, entry) in entries) + { + if (!entry.TerminalReported) { - if (!entry.TerminalReported) - { - brokerageIds.Add(brokerageId); - } + yield return brokerageId; } - return brokerageIds; } } @@ -214,12 +215,12 @@ public void Watch(string brokerageId, BrokerageOrderSnapshot lastSeen) { lock (_lock) { - if (!_orderStates.TryGetValue(brokerageId, out var entry)) + if (!_orderEntries.TryGetValue(brokerageId, out var entry)) { - entry = new OrderStateEntry(); + entry = new OrderTrackingEntry(); if (lastSeen != null) { - entry.LastSeen = lastSeen; + entry.LastSnapshot = lastSeen; entry.ReportedFilledQuantity = lastSeen.FilledQuantity ?? 0m; // seeded means another path already heard from the broker about this order, // and a seed carrying the order's end means the end was already reported @@ -227,7 +228,7 @@ public void Watch(string brokerageId, BrokerageOrderSnapshot lastSeen) entry.SubmitReported = lastSeen.Status != OrderStatus.New; entry.TerminalReported = lastSeen.Status == OrderStatus.Canceled || lastSeen.Status == OrderStatus.Invalid; } - _orderStates[brokerageId] = entry; + _orderEntries[brokerageId] = entry; } entry.Watched = true; } @@ -248,13 +249,13 @@ public void WatchReplacement(string brokerageId, string previousBrokerageId) { if (previousBrokerageId != null) { - _orderStates.Remove(previousBrokerageId); + _orderEntries.Remove(previousBrokerageId); } - if (!_orderStates.TryGetValue(brokerageId, out var entry)) + if (!_orderEntries.TryGetValue(brokerageId, out var entry)) { - entry = new OrderStateEntry(); - _orderStates[brokerageId] = entry; + entry = new OrderTrackingEntry(); + _orderEntries[brokerageId] = entry; } entry.Watched = true; entry.IsReplacement = true; @@ -269,7 +270,7 @@ public void Unwatch(string brokerageId) { lock (_lock) { - _orderStates.Remove(brokerageId); + _orderEntries.Remove(brokerageId); } } @@ -283,13 +284,13 @@ public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderSta { lock (_lock) { - if (!_orderStates.TryGetValue(brokerageId, out var entry)) + if (!_orderEntries.TryGetValue(brokerageId, out var entry)) { - entry = new OrderStateEntry(); - _orderStates[brokerageId] = entry; + entry = new OrderTrackingEntry(); + _orderEntries[brokerageId] = entry; } - entry.LastSeen = orderState; + entry.LastSnapshot = orderState; entry.Acknowledged = true; // the already-reported quantity never shrinks var filledQuantity = orderState.FilledQuantity ?? 0m; @@ -298,7 +299,7 @@ public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderSta entry.ReportedFilledQuantity = filledQuantity; } // a state written by the other path means its submit is out, and a terminal state means - // the end was already reported - a later sweep must not repeat either + // the end was already reported - a later poll must not repeat either if (orderState.Status != OrderStatus.New) { entry.SubmitReported = true; @@ -322,9 +323,9 @@ public bool TryGetLastOrderState(string brokerageId, out BrokerageOrderSnapshot lock (_lock) { lastSeen = null; - if (_orderStates.TryGetValue(brokerageId, out var entry)) + if (_orderEntries.TryGetValue(brokerageId, out var entry)) { - lastSeen = entry.LastSeen; + lastSeen = entry.LastSnapshot; } return lastSeen != null; } @@ -337,7 +338,7 @@ public bool TryGetLastOrderState(string brokerageId, out BrokerageOrderSnapshot /// safe to run twice at the same time - the handler runs it one call at a time, and without a /// handler only the poll loop calls it. /// - /// The state a sweep read from the broker. + /// The state a poll read from the broker. public void ProcessOrderState(BrokerageOrderSnapshot orderState) { if (orderState == null || string.IsNullOrEmpty(orderState.BrokerageOrderId)) @@ -350,7 +351,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) // record the id as seen: the broker knows the order, so the notification timeout stops counting lock (_lock) { - if (_orderStates.TryGetValue(brokerageId, out var seenEntry)) + if (_orderEntries.TryGetValue(brokerageId, out var seenEntry)) { seenEntry.Acknowledged = true; } @@ -360,7 +361,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) var leanOrders = _orderProvider?.GetOrdersByBrokerageId(brokerageId); if (leanOrders == null || leanOrders.Count == 0) { - // not ours, or ours with the id not on the Lean order yet - the next sweep sees it again + // not ours, or ours with the id not on the Lean order yet - the next poll sees it again return; } @@ -379,10 +380,10 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) // through UpdateOrderState can never interleave with the diff's read-then-write bookkeeping lock (_lock) { - if (!_orderStates.TryGetValue(brokerageId, out var entry)) + if (!_orderEntries.TryGetValue(brokerageId, out var entry)) { - entry = new OrderStateEntry(); - _orderStates[brokerageId] = entry; + entry = new OrderTrackingEntry(); + _orderEntries[brokerageId] = entry; } entry.Acknowledged = true; @@ -392,7 +393,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) // where the Lean order is already past New: there the state proves the replacement is live, // so the update submit goes out instead. if (!entry.SubmitReported - && (entry.LastSeen == null || entry.LastSeen.Status == OrderStatus.New) + && (entry.LastSnapshot == null || entry.LastSnapshot.Status == OrderStatus.New) && orderState.Status != OrderStatus.Invalid) { foreach (var leanOrder in leanOrders) @@ -470,7 +471,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) // the end of the order last, once. The id leaves the read list, but its state stays until a // compare sees the Lean order closed - forgetting it here would re-report every fill if the - // next sweep lands before Lean applies this event. + // next poll lands before Lean applies this event. if ((orderState.Status == OrderStatus.Canceled || orderState.Status == OrderStatus.Invalid) && !entry.TerminalReported) { foreach (var leanOrder in leanOrders) @@ -486,7 +487,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) entry.TerminalReported = true; } - entry.LastSeen = orderState; + entry.LastSnapshot = orderState; } if (orderEvents.Count > 0) @@ -501,7 +502,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) /// per open Lean order, then start the loop. Does nothing while polling already runs. /// /// - /// The stream reported 100 of 233 shares, the pre-load carries 100, so the first sweep reports only the other 133. + /// The stream reported 100 of 233 shares, the pre-load carries 100, so the first poll reports only the other 133. /// /// Builds the state another path already reported for one open Lean /// order: the brokerage id, the order's status and the cumulative filled quantity. A null return @@ -564,12 +565,12 @@ public void Start() } /// - /// Stops the polling loop but keeps the service usable, so a later resumes + /// Stops the polling loop but keeps the service usable, so a later resumes /// polling. The registry survives a stop, so nothing already reported repeats after a restart. /// public void Stop() { - Task pollingTask; + Task pollingTask; CancellationTokenSource cancellationTokenSource; lock (_lock) { @@ -600,6 +601,7 @@ public void Stop() /// public void Dispose() { + CancellationTokenSource cancellationTokenSource; lock (_lock) { if (_disposed) @@ -607,13 +609,15 @@ public void Dispose() return; } _disposed = true; + cancellationTokenSource = _cancellationTokenSource; } Stop(); + cancellationTokenSource?.Dispose(); } /// - /// Re-reads the broker on each sweep and routes every state, until cancelled. + /// Re-reads the broker on each poll and routes every state, until cancelled. /// /// Cancelled to stop the loop. private async Task PollLoop(CancellationToken cancellationToken) @@ -628,7 +632,7 @@ private async Task PollLoop(CancellationToken cancellationToken) { try { - foreach (var orderState in Sweep()) + foreach (var orderState in GetOrderSnapshots()) { if (cancellationToken.IsCancellationRequested) { @@ -638,14 +642,14 @@ private async Task PollLoop(CancellationToken cancellationToken) // a per-id read returns null when the broker does not know the id yet if (orderState != null) { - _route(orderState); + _processSnapshot(orderState); } } consecutiveFailureCount = 0; isPollingFailureReported = false; - // silence only means something after a read that succeeded: a failed sweep asked + // silence only means something after a read that succeeded: a failed poll asked // the broker nothing, so it must not count against a watched order if (!cancellationToken.IsCancellationRequested) { @@ -654,12 +658,12 @@ private async Task PollLoop(CancellationToken cancellationToken) } catch (Exception ex) { - // A transient read failure must not kill the loop: log and try again next sweep. + // A transient read failure must not kill the loop: log and try again next poll. Log.Error($"{GetType().Name}.{nameof(PollLoop)}(): failed to poll orders: {ex.Message}"); // A failure that keeps coming back is not transient any more, and the run may have no // order updates at all while it lasts, so say so once instead of only a log line. - if (++consecutiveFailureCount >= ConsecutiveFailuresBeforeReport && !isPollingFailureReported) + if (++consecutiveFailureCount >= MaxFailedPollsBeforeWarning && !isPollingFailureReported) { isPollingFailureReported = true; Message?.Invoke(this, new BrokerageMessageEvent(BrokerageMessageType.Warning, "OrderPollingFailed", @@ -683,14 +687,14 @@ private async Task PollLoop(CancellationToken cancellationToken) /// /// Counts one interval of silence for every watched order the broker never reported, and raises /// once for each one that reached the notification timeout. Called only - /// after a successful sweep, because only a read that succeeded proves the silence is real. + /// after a successful poll, because only a read that succeeded proves the silence is real. /// private void CheckNotificationTimeouts() { List<(string BrokerageId, TimeSpan WatchDuration)> expired = null; lock (_lock) { - foreach (var (brokerageId, entry) in _orderStates) + foreach (var (brokerageId, entry) in _orderEntries) { if (!entry.Watched || entry.Acknowledged) { @@ -708,7 +712,7 @@ private void CheckNotificationTimeouts() { foreach (var (brokerageId, _) in expired) { - _orderStates.Remove(brokerageId); + _orderEntries.Remove(brokerageId); } } } @@ -724,55 +728,5 @@ private void CheckNotificationTimeouts() } } } - - /// - /// What the registry keeps per brokerage order id. - /// - private class OrderStateEntry - { - /// - /// The last state seen for the order, from any path. Null when nothing was seen yet, so the - /// submit is still due. - /// - public BrokerageOrderSnapshot LastSeen; - - /// - /// The cumulative filled quantity already reported to Lean, by any path. Never shrinks. - /// - public decimal ReportedFilledQuantity; - - /// - /// Set once the submit was reported for the order, by any path, so it goes out exactly once. - /// - public bool SubmitReported; - - /// - /// Set once the order's end was reported, so the id leaves the read list and a later state - /// for it reports nothing new. - /// - public bool TerminalReported; - - /// - /// Set by : the notification timeout only applies to explicitly watched orders. - /// - public bool Watched; - - /// - /// Set by : the id is the new id of a replace, so the first - /// state to carry it reports the update submit instead of a plain submit. - /// - public bool IsReplacement; - - /// - /// Set once anything carried the id: a polled state, a stream write, or a seed. Stops the - /// notification timeout. - /// - public bool Acknowledged; - - /// - /// How long the order has been watched with nothing reporting it, in polling time. - /// - public TimeSpan UnacknowledgedDuration; - } } } diff --git a/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs b/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs index 6f4db45a410d..7125368affee 100644 --- a/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs +++ b/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs @@ -21,12 +21,12 @@ namespace QuantConnect.Brokerages.Services.OrderPolling { /// - /// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched, and the + /// For a broker with only a bulk endpoint. A poll calls the read once, whatever is watched, and the /// read returns everything the broker lists. /// /// /// Use it when the broker cannot be asked about one order id, or when one request already returns the - /// whole account cheaply. The sweep runs even with nothing watched, so it can be the order path of a + /// whole account cheaply. The poll runs even with nothing watched, so it can be the order path of a /// whole run. /// /// CreateOrderPollingService(() => _api.GetAllOrders().Select(ToOrderSnapshot), _messageHandler, _orderProvider); @@ -60,7 +60,7 @@ public BulkOrdersPollingService( /// Reads every order the broker lists, one snapshot per brokerage order id. /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. /// Resolves brokerage order ids to Lean orders. - /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + /// The sleep between polls. Null takes brokerage-order-poll-interval-ms, default 3000 ms. public BulkOrdersPollingService( Func> getAllBrokerageOrders, BrokerageConcurrentMessageHandler messageHandler, @@ -76,7 +76,7 @@ public BulkOrdersPollingService( /// Reads every order the broker lists, one snapshot per brokerage order id. /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. /// Resolves brokerage order ids to Lean orders. - /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + /// The sleep between polls. Null takes brokerage-order-poll-interval-ms, default 3000 ms. /// The silence that raises /// for a watched order. Null takes 60000 ms. public BulkOrdersPollingService( @@ -91,9 +91,9 @@ public BulkOrdersPollingService( } /// - /// Calls the read once for the whole sweep. + /// Calls the read once per poll, one request for everything the broker lists. /// - protected override IEnumerable Sweep() + protected override IEnumerable GetOrderSnapshots() { return _getAllBrokerageOrders(); } diff --git a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs index e3db500ea705..e6596d97a00d 100644 --- a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs +++ b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs @@ -20,7 +20,7 @@ namespace QuantConnect.Brokerages.Services.OrderPolling.Models { /// /// Raised when no read saw a watched brokerage order id for the whole notification timeout. A question, not - /// a verdict: the order may never have reached the broker, or closed before the first sweep. The + /// a verdict: the order may never have reached the broker, or closed before the first poll. The /// brokerage decides what to do next. /// public class BrokerageOrderNeverNotifiedEventArgs : EventArgs diff --git a/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs b/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs new file mode 100644 index 000000000000..db43a283044f --- /dev/null +++ b/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs @@ -0,0 +1,70 @@ +/* + * QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals. + * Lean Algorithmic Trading Engine v2.0. Copyright 2014 QuantConnect Corporation. + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. +*/ + +using System; + +namespace QuantConnect.Brokerages.Services.OrderPolling.Models +{ + /// + /// What the registry keeps per brokerage order id. + /// + internal class OrderTrackingEntry + { + /// + /// The last snapshot seen for the order, from any path. Null when nothing was seen yet, so the + /// submit is still due. + /// + public BrokerageOrderSnapshot LastSnapshot; + + /// + /// The cumulative filled quantity already reported to Lean, by any path. Never shrinks. + /// + public decimal ReportedFilledQuantity; + + /// + /// Set once the submit was reported for the order, by any path, so it goes out exactly once. + /// + public bool SubmitReported; + + /// + /// Set once the order's end was reported, so the id leaves the read list and a later state + /// for it reports nothing new. + /// + public bool TerminalReported; + + /// + /// Set by : the notification timeout + /// only applies to explicitly watched orders. + /// + public bool Watched; + + /// + /// Set by : the id is the new id + /// of a replace, so the first state to carry it reports the update submit instead of a plain submit. + /// + public bool IsReplacement; + + /// + /// Set once anything carried the id: a polled state, a stream write, or a seed. Stops the + /// notification timeout. + /// + public bool Acknowledged; + + /// + /// How long the order has been watched with nothing reporting it, in polling time. + /// + public TimeSpan UnacknowledgedDuration; + } +} diff --git a/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs b/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs index 43d1b53da089..c4d797cc5aaa 100644 --- a/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs @@ -22,12 +22,12 @@ namespace QuantConnect.Brokerages.Services.OrderPolling { /// - /// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id - + /// For a broker with a get-order endpoint. A poll calls the read once per watched brokerage id - /// no request when nothing is watched. A null return means the broker does not know the id, so the /// notification timeout keeps counting. /// /// - /// Use it when the broker can be asked about one order id. The sweep reads only the watched orders, + /// Use it when the broker can be asked about one order id. The poll reads only the watched orders, /// so an idle account sends no requests and rate limits stay untouched. /// /// CreateOrderPollingService(id => ToOrderSnapshot(_api.GetOrderById(id)), _messageHandler, _orderProvider); @@ -61,7 +61,7 @@ public SingleOrderPollingService( /// Reads one order by its brokerage id. A null return means the broker does not know the id. /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. /// Resolves brokerage order ids to Lean orders. - /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + /// The sleep between polls. Null takes brokerage-order-poll-interval-ms, default 3000 ms. public SingleOrderPollingService( Func getBrokerageOrderById, BrokerageConcurrentMessageHandler messageHandler, @@ -77,7 +77,7 @@ public SingleOrderPollingService( /// Reads one order by its brokerage id. A null return means the broker does not know the id. /// Serializes the snapshots with the brokerage's other messages. Null processes them directly. /// Resolves brokerage order ids to Lean orders. - /// The sleep between sweeps. Null takes brokerage-order-poll-interval-ms, default 3000 ms. + /// The sleep between polls. Null takes brokerage-order-poll-interval-ms, default 3000 ms. /// The silence that raises /// for a watched order. Null takes 60000 ms. public SingleOrderPollingService( @@ -93,17 +93,17 @@ public SingleOrderPollingService( /// /// Calls the read once per watched brokerage id. One id whose read throws is logged and skipped, - /// so it cannot starve the other watched orders; the sweep only counts as failed when every read - /// of the sweep failed. + /// so it cannot starve the other watched orders; the poll only counts as failed when every read failed. /// - protected override IEnumerable Sweep() + protected override IEnumerable GetOrderSnapshots() { - var brokerageIds = GetWatchedBrokerageIds(); - var orderStates = new List(brokerageIds.Count); + var orderStates = new List(); + var readCount = 0; var failedReadCount = 0; var lastError = default(Exception); - foreach (var brokerageId in brokerageIds) + foreach (var brokerageId in GetOpenBrokerageIds()) { + readCount++; try { orderStates.Add(_getBrokerageOrderById(brokerageId)); @@ -112,11 +112,11 @@ protected override IEnumerable Sweep() { failedReadCount++; lastError = ex; - Log.Error($"{nameof(SingleOrderPollingService)}.{nameof(Sweep)}(): failed to read order '{brokerageId}': {ex.Message}"); + Log.Error($"{nameof(SingleOrderPollingService)}.{nameof(GetOrderSnapshots)}(): failed to read order '{brokerageId}': {ex.Message}"); } } - if (failedReadCount > 0 && failedReadCount == brokerageIds.Count) + if (failedReadCount > 0 && failedReadCount == readCount) { throw lastError; } From 9d0cdcd69c2bc82ece6256e067bb0ddf725c6509 Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 16:58:13 +0300 Subject: [PATCH 23/25] docs: record the poll vocabulary round in the adr --- .../0001-brokerage-order-polling-service.md | 24 ++++++++++++++----- 1 file changed, 18 insertions(+), 6 deletions(-) diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md index 5354df16e46c..6bc3e9fbd153 100644 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ b/Documentation/ADR/0001-brokerage-order-polling-service.md @@ -88,6 +88,18 @@ silence that raises `BrokerageOrderNeverNotified`; the read callbacks became `ge `getAllBrokerageOrders`; both mode classes offer three chained constructors instead of optional parameters; and the defaults read `3 * 1000` and `60 * 1000` milliseconds in code and docs alike. +Abstract read renamed - 2026-08-19: the subclass hook `Sweep()` became `GetOrderSnapshots()` - the same +get verb as its read callbacks and `Brokerage.GetOpenOrders`, and it names what one pass returns. "Sweep" +stays the prose word for one polling pass. + +Poll vocabulary and registry names - 2026-08-19: "sweep" left every XML doc and code comment - "poll" is +the code word now, this document keeps sweep for prose. The registry record moved into its own file, +`Models/OrderTrackingEntry` with `LastSnapshot` (was a private nested `OrderStateEntry` with `LastSeen`), +and the private fields follow (`_orderEntries`, `_processSnapshot`, `MaxFailedPollsBeforeWarning`). +`GetWatchedBrokerageIds` became `GetOpenBrokerageIds` - it never filtered on the watch flag, and what it +returns is Lean's "open": the end not reported yet - returning `IEnumerable`, copying the entries +under the registry lock and filtering outside it. `Dispose` now always disposes the cancellation source. + ## Purpose Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: @@ -471,13 +483,13 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null); - /// One read of the broker, giving the states the sweep saw. The loop calls it every - /// poll interval, hands each state to the message handler, and counts a throw as one failed sweep. - protected abstract IEnumerable Sweep(); + /// One read of the broker, giving the current order snapshots. The loop calls it every + /// poll interval, hands each snapshot to the message handler, and counts a throw as one failed poll. + protected abstract IEnumerable GetOrderSnapshots(); - /// A copy of the ids a sweep still has to read: everything tracked whose end was not + /// A copy of the ids a poll still has to read: everything tracked whose end was not /// reported yet. - protected List GetWatchedBrokerageIds(); + protected IEnumerable GetOpenBrokerageIds(); /// The order events one snapshot produced. Raised inside , never empty. public event EventHandler> OrderEvents; @@ -569,7 +581,7 @@ The watch registry, the compare with the last state seen and the seeding of orde ### The two modes -The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `Sweep`: +The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `GetOrderSnapshots`: - **`SingleOrderPollingService`** — `Func`: the sweep loops the watched ids and calls the read once per id. Nothing watched, nothing requested. Public.com's replaced service had exactly this From a6fd466b4fc680cd5534ac39869267bfa398c3a8 Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 18:42:50 +0300 Subject: [PATCH 24/25] refactor: subscribe api, entry constructors and initialize seam - Watch/Unwatch/WatchReplacement become Subscribe/Unsubscribe/SubscribeReplacement, with Subscribed and SubscribedDuration - OrderTrackingEntry builds through three chained constructors; submit flag reads SubmittedOrderEventInvoked - get-or-create collapses into one GetOrCreateEntry helper - CreateOrderPollingService becomes InitializeOrderPollingService - xml docs and comments trimmed to short plain sentences --- Brokerages/Brokerage.cs | 14 +- .../BaseBrokerageOrderPollingService.cs | 196 ++++++++---------- .../OrderPolling/BulkOrdersPollingService.cs | 8 +- .../BrokerageOrderNeverNotifiedEventArgs.cs | 20 +- .../OrderPolling/Models/OrderTrackingEntry.cs | 53 ++++- .../OrderPolling/SingleOrderPollingService.cs | 14 +- .../BrokerageOrderPollingServiceTests.cs | 56 ++--- 7 files changed, 192 insertions(+), 169 deletions(-) diff --git a/Brokerages/Brokerage.cs b/Brokerages/Brokerage.cs index aa0f4e5e56e4..4d00c4647339 100644 --- a/Brokerages/Brokerage.cs +++ b/Brokerages/Brokerage.cs @@ -353,7 +353,7 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// /// The order polling service one of the create overloads built and wired, null until then. The brokerage - /// calls Watch, WatchReplacement, Start - plain or with its pre-load callback - and Stop on it at + /// calls Subscribe, SubscribeReplacement, Start - plain or with its pre-load callback - and Stop on it at /// its own lifecycle points; /// disposes it. /// @@ -377,10 +377,10 @@ protected LeanOAuthTokenHandler CreateOAuthTokenHandler(ApiConnection apiC /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between polls. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before + /// How long a subscribed order may stay unreported before /// is called. Null takes 60000 ms. /// The created and wired service. - protected SingleOrderPollingService CreateOrderPollingService(Func getBrokerageOrderById, + protected SingleOrderPollingService InitializeOrderPollingService(Func getBrokerageOrderById, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null) { @@ -399,10 +399,10 @@ protected SingleOrderPollingService CreateOrderPollingService(FuncResolves brokerage order ids to Lean orders. /// How long the loop sleeps between polls. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before + /// How long a subscribed order may stay unreported before /// is called. Null takes 60000 ms. /// The created and wired service. - protected BulkOrdersPollingService CreateOrderPollingService(Func> getAllBrokerageOrders, + protected BulkOrdersPollingService InitializeOrderPollingService(Func> getAllBrokerageOrders, BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null) { @@ -432,10 +432,10 @@ private void WireOrderPollingService(BaseBrokerageOrderPollingService service) } /// - /// Called when the broker never reported a watched order for the whole notification timeout. The default + /// Called when the broker never reported a subscribed order for the whole notification timeout. The default /// sends one warning message; override it to decide what the silence means for this broker. /// - /// The brokerage order id and how long it was watched. + /// The brokerage order id and how long it was subscribed. protected virtual void OnBrokerageOrderNeverNotified(BrokerageOrderNeverNotifiedEventArgs neverNotified) { OnMessage(new BrokerageMessageEvent(BrokerageMessageType.Warning, "OrderNotAcknowledged", diff --git a/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs b/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs index 76d1ca3b7d9d..24d3d9dcbff4 100644 --- a/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs @@ -32,7 +32,7 @@ namespace QuantConnect.Brokerages.Services.OrderPolling /// Reads orders from the brokerage on an interval and turns what it reads into order events. /// Use it when a brokerage has no order stream, when the stream goes down, or to check an order /// the broker never replied about. This base class holds what both modes share: the loop, the - /// watch registry, the compare and the events. The subclass decides what one poll reads: + /// subscription registry, the compare and the events. The subclass decides what one poll reads: /// or . /// public abstract class BaseBrokerageOrderPollingService : IDisposable @@ -43,9 +43,8 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable private const int MaxFailedPollsBeforeWarning = 3; /// - /// Guards the registry and the polling task against the poll loop, the handler thread inside - /// , the order threads through / - /// / , and the notification-timeout check. + /// One lock for the registry and the polling task. Taken by the poll loop, the handler thread + /// inside , and the order threads through the subscribe methods. /// private readonly Lock _lock = new(); @@ -55,9 +54,9 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable private readonly Dictionary _orderEntries = []; /// - /// The brokerage's message handler, when it has one. The constructor wires it both ways: polled - /// states enqueue here, and is registered as their listener, so - /// polled states queue behind an order request that holds the stream lock. + /// The brokerage's message handler, when it has one. Polled states enqueue here with + /// as their listener, so they queue behind an order request + /// that holds the stream lock. /// private readonly BrokerageConcurrentMessageHandler _messageHandler; @@ -68,8 +67,7 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable private readonly Action _processSnapshot; /// - /// Resolves brokerage order ids to Lean orders on every compare, so the service never drifts from - /// what Lean actually knows. + /// Resolves brokerage order ids to Lean orders on every compare. /// private readonly IOrderProvider _orderProvider; @@ -103,8 +101,8 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable public event EventHandler> OrderEvents; /// - /// A watched order that nothing reported for of polling. Raised once; - /// the id is unwatched with it. The brokerage decides what the silence means. + /// A subscribed order that nothing reported for of polling. Raised once; + /// the id is unsubscribed with it. The brokerage decides what the silence means. /// public event EventHandler BrokerageOrderNeverNotified; @@ -125,7 +123,7 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable public TimeSpan PollInterval { get; } /// - /// How long a watched order may stay completely unreported, in polling time, before + /// How long a subscribed order may stay completely unreported, in polling time, before /// is raised for it. /// public TimeSpan NotificationTimeout { get; } @@ -134,15 +132,14 @@ public abstract class BaseBrokerageOrderPollingService : IDisposable /// Initializes what both modes share: the message handler wiring, the order provider, and the two /// time settings with their defaults. /// - /// The brokerage's message handler. The service wires it both ways - /// itself: it registers and enqueues every polled state, so one - /// handler serializes polled states with everything else the brokerage processes. Null routes each - /// state straight into - only the poll loop calls it then, so the - /// calls are still one at a time. + /// The brokerage's message handler. The service registers + /// and enqueues every polled state, so one handler serializes + /// them with the brokerage's other messages. Null processes each state directly - only the + /// poll loop calls then, so the calls are still one at a time. /// Resolves brokerage order ids to Lean orders. /// How long the loop sleeps between polls. Null falls back to the /// brokerage-order-poll-interval-ms configuration entry, default 3000 ms. - /// How long a watched order may stay unreported before + /// How long a subscribed order may stay unreported before /// is raised. Null takes 60000 ms. protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null) @@ -176,10 +173,10 @@ protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler mes /// protected IEnumerable GetOpenBrokerageIds() { - KeyValuePair[] entries; + var entries = default(KeyValuePair[]); lock (_lock) { - entries = [.. _orderEntries]; + entries = _orderEntries.ToArray(); } foreach (var (brokerageId, entry) in entries) @@ -192,58 +189,45 @@ protected IEnumerable GetOpenBrokerageIds() } /// - /// Watches a brokerage order id, with nothing seen for it yet, so the first state to carry the id - /// acknowledges the order and of silence raises - /// . Idempotent: watching an already-watched id never overwrites - /// its state. + /// Subscribes to a brokerage order id. The first state to carry the id acknowledges the order; + /// of silence raises . + /// Idempotent: a repeat call never overwrites the recorded state. /// - /// The brokerage order id to watch. - public void Watch(string brokerageId) + /// The brokerage order id to subscribe to. + public void Subscribe(string brokerageId) { - Watch(brokerageId, lastSeen: null); + Subscribe(brokerageId, lastSeen: null); } /// - /// Watches a brokerage order id, seeded with what another path already reported, so the next poll - /// does not repeat it. Used for orders adopted at startup, for a submit reported from the request - /// path, and to move state onto the new id of a replace. Idempotent: watching an already-watched - /// id never overwrites its state. + /// Subscribes to a brokerage order id, seeded with what another path already reported, so the + /// next poll does not repeat it. Used for orders adopted at startup and for stream-to-polling + /// handovers. Idempotent: a repeat call never overwrites the recorded state. /// - /// The brokerage order id to watch. + /// The brokerage order id to subscribe to. /// The state another path already reported for the order. - public void Watch(string brokerageId, BrokerageOrderSnapshot lastSeen) + public void Subscribe(string brokerageId, BrokerageOrderSnapshot lastSeen) { lock (_lock) { if (!_orderEntries.TryGetValue(brokerageId, out var entry)) { - entry = new OrderTrackingEntry(); - if (lastSeen != null) - { - entry.LastSnapshot = lastSeen; - entry.ReportedFilledQuantity = lastSeen.FilledQuantity ?? 0m; - // seeded means another path already heard from the broker about this order, - // and a seed carrying the order's end means the end was already reported - entry.Acknowledged = true; - entry.SubmitReported = lastSeen.Status != OrderStatus.New; - entry.TerminalReported = lastSeen.Status == OrderStatus.Canceled || lastSeen.Status == OrderStatus.Invalid; - } + entry = lastSeen == null ? new OrderTrackingEntry() : new OrderTrackingEntry(lastSeen); _orderEntries[brokerageId] = entry; } - entry.Watched = true; + entry.Subscribed = true; } } /// - /// Watches the new brokerage order id of a replace and drops the replaced id in the same step. - /// The first state to carry the new id reports the order as update submitted, which a stream - /// would otherwise do. The new id starts with no fill state, because a replacement that counts - /// its executions from zero must not inherit the old order's numbers; a broker that carries the - /// fills across a replace seeds with instead. + /// Subscribes to the new brokerage order id of a replace and drops the replaced id in one step. + /// The first state to carry the new id reports the update submit. The new id starts with no fill + /// state; a broker that carries fills across a replace seeds with + /// instead. /// /// The brokerage order id the replacement runs under. /// The replaced brokerage order id, or null when it is unknown. - public void WatchReplacement(string brokerageId, string previousBrokerageId) + public void SubscribeReplacement(string brokerageId, string previousBrokerageId) { lock (_lock) { @@ -252,21 +236,17 @@ public void WatchReplacement(string brokerageId, string previousBrokerageId) _orderEntries.Remove(previousBrokerageId); } - if (!_orderEntries.TryGetValue(brokerageId, out var entry)) - { - entry = new OrderTrackingEntry(); - _orderEntries[brokerageId] = entry; - } - entry.Watched = true; + var entry = GetOrCreateEntry(brokerageId); + entry.Subscribed = true; entry.IsReplacement = true; } } /// - /// Stops watching an order and drops its state. + /// Unsubscribes an order and drops its state. /// - /// The brokerage order id to stop watching. - public void Unwatch(string brokerageId) + /// The brokerage order id to unsubscribe. + public void Unsubscribe(string brokerageId) { lock (_lock) { @@ -274,6 +254,21 @@ public void Unwatch(string brokerageId) } } + /// + /// The entry for the id, created empty when the id is not tracked yet. The caller holds the + /// registry lock. + /// + /// The brokerage order id to look up. + private OrderTrackingEntry GetOrCreateEntry(string brokerageId) + { + if (!_orderEntries.TryGetValue(brokerageId, out var entry)) + { + entry = new OrderTrackingEntry(); + _orderEntries[brokerageId] = entry; + } + return entry; + } + /// /// Records what another path already reported for an order, so the next poll does not repeat it. /// Called by the streaming path while the stream lives, after it reports its own event. @@ -284,11 +279,7 @@ public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderSta { lock (_lock) { - if (!_orderEntries.TryGetValue(brokerageId, out var entry)) - { - entry = new OrderTrackingEntry(); - _orderEntries[brokerageId] = entry; - } + var entry = GetOrCreateEntry(brokerageId); entry.LastSnapshot = orderState; entry.Acknowledged = true; @@ -302,7 +293,7 @@ public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderSta // the end was already reported - a later poll must not repeat either if (orderState.Status != OrderStatus.New) { - entry.SubmitReported = true; + entry.SubmittedOrderEventInvoked = true; } if (orderState.Status == OrderStatus.Canceled || orderState.Status == OrderStatus.Invalid) { @@ -333,10 +324,9 @@ public bool TryGetLastOrderState(string brokerageId, out BrokerageOrderSnapshot /// /// Compares a state with the last one seen for the same order and raises - /// with what is new: the submit first, then fills, then a close. The constructor registers it on the - /// message handler, so polled orders queue behind an order request that holds the stream lock. Not - /// safe to run twice at the same time - the handler runs it one call at a time, and without a - /// handler only the poll loop calls it. + /// with what is new: the submit first, then fills, then a close. Not safe to run twice at the + /// same time - the message handler runs it one call at a time, and without a handler only the + /// poll loop calls it. /// /// The state a poll read from the broker. public void ProcessOrderState(BrokerageOrderSnapshot orderState) @@ -369,7 +359,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) { // nothing left to report; dropping the state here is the only safe moment, because Lean // has already applied the end of the order - Unwatch(brokerageId); + Unsubscribe(brokerageId); return; } @@ -380,19 +370,13 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) // through UpdateOrderState can never interleave with the diff's read-then-write bookkeeping lock (_lock) { - if (!_orderEntries.TryGetValue(brokerageId, out var entry)) - { - entry = new OrderTrackingEntry(); - _orderEntries[brokerageId] = entry; - } + var entry = GetOrCreateEntry(brokerageId); entry.Acknowledged = true; - // the submit first, once: when nothing was emitted for the id yet, the Lean order is still New, - // and the state is not a reject. Lean requires it before any fill, and a market order can - // already be Filled the first time a poll sees it. The new id of a replace is the one case - // where the Lean order is already past New: there the state proves the replacement is live, - // so the update submit goes out instead. - if (!entry.SubmitReported + // the submit first, once: Lean requires it before any fill, and a market order can already + // be Filled on its first read. The new id of a replace is past New already, so the update + // submit goes out instead. + if (!entry.SubmittedOrderEventInvoked && (entry.LastSnapshot == null || entry.LastSnapshot.Status == OrderStatus.New) && orderState.Status != OrderStatus.Invalid) { @@ -404,7 +388,7 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) { Status = OrderStatus.Submitted }); - entry.SubmitReported = true; + entry.SubmittedOrderEventInvoked = true; } else if (entry.IsReplacement && !leanOrder.Status.IsClosed()) { @@ -412,14 +396,13 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) { Status = OrderStatus.UpdateSubmitted }); - entry.SubmitReported = true; + entry.SubmittedOrderEventInvoked = true; } } } - // then the fills, so a close can never outrun a fill of the same order. A fill needs both - // numbers: without a price the service would have to invent one, and it never invents a - // number - a read without prices simply reports less. + // then the fills, so a close never comes before a fill of the same order. A fill needs + // both numbers: the service never invents a price, a read without one reports less. if (orderState.FilledQuantity.HasValue && orderState.FillPrice.HasValue) { var cumulativeFilled = orderState.FilledQuantity.Value; @@ -469,9 +452,8 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) } } - // the end of the order last, once. The id leaves the read list, but its state stays until a - // compare sees the Lean order closed - forgetting it here would re-report every fill if the - // next poll lands before Lean applies this event. + // the end of the order last, once. The state stays until a compare sees the Lean order + // closed - dropping it here would re-report every fill on the next poll. if ((orderState.Status == OrderStatus.Canceled || orderState.Status == OrderStatus.Invalid) && !entry.TerminalReported) { foreach (var leanOrder in leanOrders) @@ -497,9 +479,9 @@ public void ProcessOrderState(BrokerageOrderSnapshot orderState) } /// - /// The whole handover from a stream to polling, in the only safe order: process what the stream - /// already delivered, pre-load the registry with one - /// per open Lean order, then start the loop. Does nothing while polling already runs. + /// The handover from a stream to polling: process what the stream already delivered, pre-load one + /// per open Lean order, then start the + /// loop. Does nothing while polling already runs. /// /// /// The stream reported 100 of 233 shares, the pre-load carries 100, so the first poll reports only the other 133. @@ -514,8 +496,8 @@ public void Start(Func preLoadOpenOrders) return; } - // an empty locked block waits for any order request in flight and processes the stream messages - // it buffered, so the pre-load below counts every fill the stream delivered + // an empty locked block waits for any order request in flight and processes its buffered + // stream messages, so the pre-load below counts every fill the stream delivered _messageHandler?.WithLockedStream(() => { }); if (preLoadOpenOrders != null) @@ -530,7 +512,7 @@ public void Start(Func preLoadOpenOrders) var lastSeen = preLoadOpenOrders(openLeanOrder); if (lastSeen != null && !string.IsNullOrEmpty(lastSeen.BrokerageOrderId)) { - Watch(lastSeen.BrokerageOrderId, lastSeen); + Subscribe(lastSeen.BrokerageOrderId, lastSeen); preLoadedCount++; } } @@ -624,7 +606,7 @@ private async Task PollLoop(CancellationToken cancellationToken) { Log.Trace($"{GetType().Name}.{nameof(PollLoop)}(): started, polling every {PollInterval.TotalMilliseconds}ms."); - // per run, so a stopped loop still draining a slow read never shares them with the next run + // per run, so a stopped loop still finishing a slow read never shares them with the next run var consecutiveFailureCount = 0; var isPollingFailureReported = false; @@ -650,7 +632,7 @@ private async Task PollLoop(CancellationToken cancellationToken) isPollingFailureReported = false; // silence only means something after a read that succeeded: a failed poll asked - // the broker nothing, so it must not count against a watched order + // the broker nothing, so it must not count against a subscribed order if (!cancellationToken.IsCancellationRequested) { CheckNotificationTimeouts(); @@ -685,18 +667,18 @@ private async Task PollLoop(CancellationToken cancellationToken) } /// - /// Counts one interval of silence for every watched order the broker never reported, and raises - /// once for each one that reached the notification timeout. Called only - /// after a successful poll, because only a read that succeeded proves the silence is real. + /// Counts one interval of silence per subscribed order nobody reported, and raises + /// once for each that reached the notification timeout. + /// Called only after a successful poll: only a read that succeeded proves the silence. /// private void CheckNotificationTimeouts() { - List<(string BrokerageId, TimeSpan WatchDuration)> expired = null; + List<(string BrokerageId, TimeSpan SubscribedDuration)> expired = null; lock (_lock) { foreach (var (brokerageId, entry) in _orderEntries) { - if (!entry.Watched || entry.Acknowledged) + if (!entry.Subscribed || entry.Acknowledged) { continue; } @@ -719,12 +701,12 @@ private void CheckNotificationTimeouts() if (expired != null) { - foreach (var (brokerageId, watchDuration) in expired) + foreach (var (brokerageId, subscribedDuration) in expired) { - // Resolved outside the registry lock. A placement whose id assignment was itself the - // thing that never happened resolves to no Lean order, so the args carry null then. + // resolved outside the registry lock; a placement whose id was never assigned + // resolves to null var leanOrder = _orderProvider?.GetOrdersByBrokerageId(brokerageId)?.FirstOrDefault(); - BrokerageOrderNeverNotified?.Invoke(this, new BrokerageOrderNeverNotifiedEventArgs(brokerageId, leanOrder, watchDuration)); + BrokerageOrderNeverNotified?.Invoke(this, new BrokerageOrderNeverNotifiedEventArgs(brokerageId, leanOrder, subscribedDuration)); } } } diff --git a/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs b/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs index 7125368affee..f0ddcb629097 100644 --- a/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs +++ b/Brokerages/Services/OrderPolling/BulkOrdersPollingService.cs @@ -21,15 +21,15 @@ namespace QuantConnect.Brokerages.Services.OrderPolling { /// - /// For a broker with only a bulk endpoint. A poll calls the read once, whatever is watched, and the + /// For a broker with only a bulk endpoint. A poll calls the read once, whatever is subscribed, and the /// read returns everything the broker lists. /// /// /// Use it when the broker cannot be asked about one order id, or when one request already returns the - /// whole account cheaply. The poll runs even with nothing watched, so it can be the order path of a + /// whole account cheaply. The poll runs even with nothing subscribed, so it can be the order path of a /// whole run. /// - /// CreateOrderPollingService(() => _api.GetAllOrders().Select(ToOrderSnapshot), _messageHandler, _orderProvider); + /// InitializeOrderPollingService(() => _api.GetAllOrders().Select(ToOrderSnapshot), _messageHandler, _orderProvider); /// /// public class BulkOrdersPollingService : BaseBrokerageOrderPollingService @@ -78,7 +78,7 @@ public BulkOrdersPollingService( /// Resolves brokerage order ids to Lean orders. /// The sleep between polls. Null takes brokerage-order-poll-interval-ms, default 3000 ms. /// The silence that raises - /// for a watched order. Null takes 60000 ms. + /// for a subscribed order. Null takes 60000 ms. public BulkOrdersPollingService( Func> getAllBrokerageOrders, BrokerageConcurrentMessageHandler messageHandler, diff --git a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs index e6596d97a00d..6f59d1f21bff 100644 --- a/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs +++ b/Brokerages/Services/OrderPolling/Models/BrokerageOrderNeverNotifiedEventArgs.cs @@ -19,9 +19,9 @@ namespace QuantConnect.Brokerages.Services.OrderPolling.Models { /// - /// Raised when no read saw a watched brokerage order id for the whole notification timeout. A question, not - /// a verdict: the order may never have reached the broker, or closed before the first poll. The - /// brokerage decides what to do next. + /// Raised when no read saw a subscribed brokerage order id for the whole notification timeout. + /// The order may never have reached the broker, or closed before the first poll. The brokerage + /// decides what to do next. /// public class BrokerageOrderNeverNotifiedEventArgs : EventArgs { @@ -36,30 +36,30 @@ public class BrokerageOrderNeverNotifiedEventArgs : EventArgs public Order Order { get; } /// - /// How long the id was watched, in polling time. + /// How long the id was subscribed, in polling time. /// - public TimeSpan WatchDuration { get; } + public TimeSpan SubscribedDuration { get; } /// /// Creates a new . /// /// The brokerage order id no read ever saw. /// The Lean order behind the id, or null when nothing resolves. - /// How long the id was watched, in polling time. - public BrokerageOrderNeverNotifiedEventArgs(string brokerageOrderId, Order order, TimeSpan watchDuration) + /// How long the id was subscribed, in polling time. + public BrokerageOrderNeverNotifiedEventArgs(string brokerageOrderId, Order order, TimeSpan subscribedDuration) { BrokerageOrderId = brokerageOrderId; Order = order; - WatchDuration = watchDuration; + SubscribedDuration = subscribedDuration; } /// - /// The order and the watch duration, ready for a log line or a warning. + /// The order and the subscribed duration, ready for a log line or a warning. /// public override string ToString() { var order = Order?.ToString() ?? $"brokerage order id '{BrokerageOrderId}'"; - return $"{order}, watched for {WatchDuration.TotalSeconds:F0} seconds"; + return $"{order}, subscribed for {SubscribedDuration.TotalSeconds:F0} seconds"; } } } diff --git a/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs b/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs index db43a283044f..e6f9c952fa6c 100644 --- a/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs +++ b/Brokerages/Services/OrderPolling/Models/OrderTrackingEntry.cs @@ -13,6 +13,7 @@ * limitations under the License. */ +using QuantConnect.Orders; using System; namespace QuantConnect.Brokerages.Services.OrderPolling.Models @@ -36,7 +37,7 @@ internal class OrderTrackingEntry /// /// Set once the submit was reported for the order, by any path, so it goes out exactly once. /// - public bool SubmitReported; + public bool SubmittedOrderEventInvoked; /// /// Set once the order's end was reported, so the id leaves the read list and a later state @@ -45,13 +46,13 @@ internal class OrderTrackingEntry public bool TerminalReported; /// - /// Set by : the notification timeout - /// only applies to explicitly watched orders. + /// Set by : the notification timeout + /// only applies to explicitly subscribed orders. /// - public bool Watched; + public bool Subscribed; /// - /// Set by : the id is the new id + /// Set by : the id is the new id /// of a replace, so the first state to carry it reports the update submit instead of a plain submit. /// public bool IsReplacement; @@ -63,8 +64,48 @@ internal class OrderTrackingEntry public bool Acknowledged; /// - /// How long the order has been watched with nothing reporting it, in polling time. + /// How long the order has been subscribed with nothing reporting it, in polling time. /// public TimeSpan UnacknowledgedDuration; + + /// + /// Creates an empty entry: nothing seen, nothing reported yet. + /// + public OrderTrackingEntry() + { + } + + /// + /// Creates an entry seeded from a snapshot another path already reported: the fill quantity + /// counts as reported, the id as acknowledged, and the snapshot's status decides whether the + /// submit and the end already went out. + /// + /// The snapshot another path already reported for the order. + public OrderTrackingEntry(BrokerageOrderSnapshot lastSnapshot) + : this(lastSnapshot, + lastSnapshot.FilledQuantity ?? 0m, + acknowledged: true, + submittedOrderEventInvoked: lastSnapshot.Status != OrderStatus.New, + terminalReported: lastSnapshot.Status == OrderStatus.Canceled || lastSnapshot.Status == OrderStatus.Invalid) + { + } + + /// + /// Creates an entry seeded with what another path already reported for the order. + /// + /// The last snapshot seen for the order. + /// The cumulative filled quantity already reported to Lean. + /// Whether anything already carried the order's id. + /// Whether the submit was already reported. + /// Whether the order's end was already reported. + public OrderTrackingEntry(BrokerageOrderSnapshot lastSnapshot, decimal reportedFilledQuantity, bool acknowledged, + bool submittedOrderEventInvoked, bool terminalReported) + { + LastSnapshot = lastSnapshot; + ReportedFilledQuantity = reportedFilledQuantity; + Acknowledged = acknowledged; + SubmittedOrderEventInvoked = submittedOrderEventInvoked; + TerminalReported = terminalReported; + } } } diff --git a/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs b/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs index c4d797cc5aaa..088a1506b19b 100644 --- a/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs +++ b/Brokerages/Services/OrderPolling/SingleOrderPollingService.cs @@ -22,15 +22,15 @@ namespace QuantConnect.Brokerages.Services.OrderPolling { /// - /// For a broker with a get-order endpoint. A poll calls the read once per watched brokerage id - - /// no request when nothing is watched. A null return means the broker does not know the id, so the + /// For a broker with a get-order endpoint. A poll calls the read once per subscribed brokerage id - + /// no request when nothing is subscribed. A null return means the broker does not know the id, so the /// notification timeout keeps counting. /// /// - /// Use it when the broker can be asked about one order id. The poll reads only the watched orders, + /// Use it when the broker can be asked about one order id. The poll reads only the subscribed orders, /// so an idle account sends no requests and rate limits stay untouched. /// - /// CreateOrderPollingService(id => ToOrderSnapshot(_api.GetOrderById(id)), _messageHandler, _orderProvider); + /// InitializeOrderPollingService(id => ToOrderSnapshot(_api.GetOrderById(id)), _messageHandler, _orderProvider); /// /// public class SingleOrderPollingService : BaseBrokerageOrderPollingService @@ -79,7 +79,7 @@ public SingleOrderPollingService( /// Resolves brokerage order ids to Lean orders. /// The sleep between polls. Null takes brokerage-order-poll-interval-ms, default 3000 ms. /// The silence that raises - /// for a watched order. Null takes 60000 ms. + /// for a subscribed order. Null takes 60000 ms. public SingleOrderPollingService( Func getBrokerageOrderById, BrokerageConcurrentMessageHandler messageHandler, @@ -92,8 +92,8 @@ public SingleOrderPollingService( } /// - /// Calls the read once per watched brokerage id. One id whose read throws is logged and skipped, - /// so it cannot starve the other watched orders; the poll only counts as failed when every read failed. + /// Calls the read once per subscribed brokerage id. One id whose read throws is logged and skipped, + /// so it cannot block the other subscribed orders; the poll only counts as failed when every read failed. /// protected override IEnumerable GetOrderSnapshots() { diff --git a/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs index a92984dd36fd..ef40d572355c 100644 --- a/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs +++ b/Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs @@ -96,13 +96,13 @@ public void FirstStateAlreadyFilledEmitsSubmitBeforeFill() } [Test] - public void ReplacementWatchReportsUpdateSubmittedOnce() + public void ReplacementSubscriptionReportsUpdateSubmittedOnce() { // a replace moved the order onto a new brokerage id, and the plugin marked the new id var order = AddOrder(100m, "42", OrderStatus.Submitted); order.BrokerId.Clear(); order.BrokerId.Add("43"); - _service.WatchReplacement("43", "42"); + _service.SubscribeReplacement("43", "42"); _service.ProcessOrderState(State("43", OrderStatus.Submitted)); @@ -116,7 +116,7 @@ public void ReplacementWatchReportsUpdateSubmittedOnce() } [Test] - public void ReplacementWatchDropsThePreviousIdAndCountsFillsFromZero() + public void ReplacementSubscriptionDropsThePreviousIdAndCountsFillsFromZero() { // the old id already reported a fill, then the replace re-keys the order. The replacement // counts its executions from zero, so its first fill must not be shrunk by the old total. @@ -126,7 +126,7 @@ public void ReplacementWatchDropsThePreviousIdAndCountsFillsFromZero() order.BrokerId.Clear(); order.BrokerId.Add("43"); - _service.WatchReplacement("43", "42"); + _service.SubscribeReplacement("43", "42"); _orderEvents.Clear(); // the first state of the new id can already carry a fill: the update submit still goes first @@ -322,8 +322,8 @@ public void TerminalSeedIsNotRepeated() { AddOrder(100m, "42", OrderStatus.Submitted); - // another path already reported the cancel; the watch moves the state, e.g. across a replace - _service.Watch("42", State("42", OrderStatus.Canceled, message: "canceled by the broker")); + // another path already reported the cancel; the subscription moves the state, e.g. across a replace + _service.Subscribe("42", State("42", OrderStatus.Canceled, message: "canceled by the broker")); _service.ProcessOrderState(State("42", OrderStatus.Canceled, message: "canceled by the broker")); Assert.IsEmpty(_orderEvents); @@ -384,12 +384,12 @@ public void RejectDoesNotEmitSubmit() } [Test] - public void SeededWatchDoesNotRepeatWhatWasAlreadyReported() + public void SeededSubscriptionDoesNotRepeatWhatWasAlreadyReported() { AddOrder(200m, "42", OrderStatus.PartiallyFilled); // another path already reported the submit and 100 shares - _service.Watch("42", State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); + _service.Subscribe("42", State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); _service.ProcessOrderState(State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); Assert.IsEmpty(_orderEvents); @@ -431,20 +431,20 @@ public void StreamReportedTerminalIsNotRepeatedByThePoll() } [Test] - public void WatchNeverOverwritesExistingState() + public void SubscribeNeverOverwritesExistingState() { AddOrder(200m, "42", OrderStatus.PartiallyFilled); _service.UpdateOrderState("42", State("42", OrderStatus.PartiallyFilled, filled: 100m, price: 310m)); - // a later plain watch keeps the recorded state - _service.Watch("42"); + // a later plain subscribe keeps the recorded state + _service.Subscribe("42"); Assert.IsTrue(_service.TryGetLastOrderState("42", out var lastSeen)); Assert.AreEqual(100m, lastSeen.FilledQuantity); - // and a later seeded watch ignores its seed too: the recorded state wins, so the next poll + // and a later seeded subscribe ignores its seed too: the recorded state wins, so the next poll // cannot re-report fills the stream already delivered - _service.Watch("42", State("42", OrderStatus.Submitted, filled: 0m)); + _service.Subscribe("42", State("42", OrderStatus.Submitted, filled: 0m)); Assert.IsTrue(_service.TryGetLastOrderState("42", out lastSeen)); Assert.AreEqual(100m, lastSeen.FilledQuantity); @@ -453,7 +453,7 @@ public void WatchNeverOverwritesExistingState() } [Test] - public void NotificationTimeoutFiresOnceAndUnwatchesTheId() + public void NotificationTimeoutFiresOnceAndUnsubscribesTheId() { var order = AddOrder(100m, "77"); using var neverNotified = new ManualResetEventSlim(false); @@ -473,12 +473,12 @@ public void NotificationTimeoutFiresOnceAndUnwatchesTheId() neverNotified.Set(); }; - service.Watch("77"); + service.Subscribe("77"); service.Start(); Assert.IsTrue(neverNotified.Wait(TimeSpan.FromSeconds(5)), "the notification timeout never fired"); - // let a few more sweeps run: the id was unwatched with the event, so it fires exactly once + // let a few more sweeps run: the id was unsubscribed with the event, so it fires exactly once Thread.Sleep(200); service.Stop(); @@ -487,27 +487,27 @@ public void NotificationTimeoutFiresOnceAndUnwatchesTheId() Assert.AreEqual(1, raised.Count); Assert.AreEqual("77", raised[0].BrokerageOrderId); Assert.AreSame(order, raised[0].Order); - Assert.GreaterOrEqual(raised[0].WatchDuration, TimeSpan.FromMilliseconds(75)); + Assert.GreaterOrEqual(raised[0].SubscribedDuration, TimeSpan.FromMilliseconds(75)); } Assert.IsFalse(service.TryGetLastOrderState("77", out _)); } [Test] - public void NeverNotifiedArgsPrintTheOrderAndTheWatchDuration() + public void NeverNotifiedArgsPrintTheOrderAndTheSubscribedDuration() { var order = AddOrder(100m, "42"); - // the message the default warning embeds: the Lean order's own ToString plus the watch duration + // the message the default warning embeds: the Lean order's own ToString plus the subscribed duration var withOrder = new BrokerageOrderNeverNotifiedEventArgs("42", order, TimeSpan.FromSeconds(60)); - Assert.AreEqual($"{order}, watched for 60 seconds", withOrder.ToString()); + Assert.AreEqual($"{order}, subscribed for 60 seconds", withOrder.ToString()); // a placement whose id was never assigned resolves to no Lean order: the id carries the identity var withoutOrder = new BrokerageOrderNeverNotifiedEventArgs("42", order: null, TimeSpan.FromSeconds(60)); - Assert.AreEqual("brokerage order id '42', watched for 60 seconds", withoutOrder.ToString()); + Assert.AreEqual("brokerage order id '42', subscribed for 60 seconds", withoutOrder.ToString()); } [Test] - public void AcknowledgedWatchNeverTimesOut() + public void AcknowledgedSubscriptionNeverTimesOut() { var fired = 0; using var service = new SingleOrderPollingService( @@ -518,8 +518,8 @@ public void AcknowledgedWatchNeverTimesOut() notificationTimeout: TimeSpan.FromMilliseconds(75)); service.BrokerageOrderNeverNotified += (_, _) => Interlocked.Increment(ref fired); - // the stream acknowledged the order right after it was watched - service.Watch("77"); + // the stream acknowledged the order right after it was subscribed + service.Subscribe("77"); service.UpdateOrderState("77", State("77", OrderStatus.Submitted)); service.Start(); @@ -575,7 +575,7 @@ public void RepeatedReadFailuresRaiseOneWarningPerOutage() } [Test] - public void PerOrderIdSweepReadsOnlyWatchedIdsAndProcessesTheStates() + public void PerOrderIdSweepReadsOnlySubscribedIdsAndProcessesTheStates() { AddOrder(100m, "42"); var readIds = new List(); @@ -603,7 +603,7 @@ public void PerOrderIdSweepReadsOnlyWatchedIdsAndProcessesTheStates() processed.Set(); }; - // nothing watched: sweeps read nothing + // nothing subscribed: sweeps read nothing service.Start(); Thread.Sleep(100); lock (readIds) @@ -611,8 +611,8 @@ public void PerOrderIdSweepReadsOnlyWatchedIdsAndProcessesTheStates() Assert.IsEmpty(readIds); } - service.Watch("42"); - Assert.IsTrue(processed.Wait(TimeSpan.FromSeconds(5)), "the watched id never produced an event"); + service.Subscribe("42"); + Assert.IsTrue(processed.Wait(TimeSpan.FromSeconds(5)), "the subscribed id never produced an event"); service.Stop(); lock (readIds) From 4bee817afb829efd21cd5bed40449d21806db315 Mon Sep 17 00:00:00 2001 From: Romazes Date: Wed, 19 Aug 2026 18:42:50 +0300 Subject: [PATCH 25/25] docs: remove the adr from the repo - the adr lives outside the repository now --- .../0001-brokerage-order-polling-service.md | 1406 ----------------- 1 file changed, 1406 deletions(-) delete mode 100644 Documentation/ADR/0001-brokerage-order-polling-service.md diff --git a/Documentation/ADR/0001-brokerage-order-polling-service.md b/Documentation/ADR/0001-brokerage-order-polling-service.md deleted file mode 100644 index 6bc3e9fbd153..000000000000 --- a/Documentation/ADR/0001-brokerage-order-polling-service.md +++ /dev/null @@ -1,1406 +0,0 @@ -# ADR 0001: Brokerage order polling service - -## Status - -Proposed - 2026-08-07 - -Pilot implemented - 2026-08-13: CharlesSchwab is the first plugin on the service -(`Lean.Brokerages.CharlesSchwab`, draft PR #107). Its own `OrderUpdatePollingService` and diff are deleted; -what remains in the plugin is what this document says remains - the read with its sweep window, the -model-to-state mapping, the leg id assignment shared with the stream's `OrderAccepted`, and the seeded -stream-to-polling handover through the pre-loading `Start` overload. - -Replacement watch implemented - 2026-08-14: a polled replace goes through `WatchReplacement` and the diff -reports the update submit. The plugin's own replace reporting and its by-position leg id derivation are -deleted. Backed by the replace survey (see "A replace, across the brokers"). - -Pilot verified live - 2026-08-14: place, replace, cancel and fill were reported correctly by the polled -connection running next to a streaming one, on real CharlesSchwab accounts. The diff's documented edges -showed up as designed: a market order already filled on its first listing reports the submit and the fill -in one batch, and the executions of one sweep arrive as one event. - -Second plugin adopted - 2026-08-15: Public.com runs on `SingleOrderPollingService` -(`Lean.Brokerages.Public`, draft PR #6). Its own service class, diff and snapshot model are deleted; the -plugin keeps the get-order read and the model-to-state mapping. Its same-id replace stays plugin-side, so -`WatchReplacement` is not wired, and a get-order 404 maps to a null state - the contract's "the broker -does not know the id". One rollout intention changed: Public kept its change-of-average price recovery, -moved into its mapping (see "The diff", pricing). - -Wiring moved into core - 2026-08-17: the root `Brokerage` class gained the seam the base-class section -describes - two protected `CreateOrderPollingService` overloads whose read-callback signature picks the -mode, the service as a protected `OrderPollingService` property with `IsOrderPolling`, a virtual -`OnBrokerageOrderNeverNotified` for the silence warning, and a `Dispose` that covers the service. -CharlesSchwab and Public.com were moved onto it: the creation, the three event forwards and the dispose -call left both plugins (see "Wiring, per plugin"). - -Third plugin adopted - 2026-08-17: Tradier runs on `SingleOrderPollingService` -(`Lean.Brokerages.Tradier`, draft PR #54). Its fill timer, `CheckForFills` -diff, order cache and unknown-id verification are deleted (~230 lines); the plugin keeps the get-order -read and the mapping. Tradier is the first adopter that splits orders across zero: the legs chain from -the read through the base cross-zero helpers, and the second leg's watch seed carries what the first leg -filled (see "A cross-zero order, two ids"). Two behavior changes are intentional: orders placed outside -Lean are ignored - a per-id read never sees them, where the old code raised a fatal "UnknownOrderId" -error - and the fee attaches once per Lean order instead of once per broker leg. - -Testing process run end to end - 2026-08-17: Public.com is the first plugin tested by this document's own -order - live capture first, offline replay second (`PublicBrokerageOrderPollingTests.cs`, in draft PR #6). -No recorded payloads existed, so two Explicit live tests ran by hand with debug logging on, and three -capture runs recorded the get-order bodies: an equity place-update-cancel and a market fill, an option -update whose fill beat the cancel, and a multi-leg cancel. Eleven offline tests replay those bodies. The -runs confirmed the per-id mode's edges on a real account: a market order filled on its first read reports -the submit and the fill in one batch, a `PENDING_CANCEL` read between the request and the cancel reports -nothing, and one canceled shared-id combo reports one `Canceled` per leg. - -Seeded start folded into Start - 2026-08-18: `SeedAndStart(seed)` became the `Start(preLoadOpenOrders)` -overload, so starting is one method with two shapes: the plain `Start()` resumes the loop, and the overload -runs the handover first. Behavior unchanged; Schwab's call site renamed with it. - -State constructors and wiring traces - 2026-08-18: `BrokerageOrderSnapshot` gained two constructors - the -always-known facts positionally, and an overload taking the message without the fill numbers - and all -three adopters build through them. The wiring now traces the created mode class with its intervals, and -the pre-loading `Start` traces how many open orders it pre-loaded. Schwab moved its mapping into -`CharlesSchwabExtensions.ToLegOrderStates`, called as `brokerageOrder.ToLegOrderStates()` from the read. - -Folders split per service - 2026-08-18: the service moved to `Brokerages/Services/OrderPolling/`, with its -data shapes in `Models/` under it, and the namespaces follow the folders. `Services` holds one subfolder -per plugin service from now on, so the next service gets a sibling folder instead of growing one flat -namespace. All three adopters moved onto the new usings with it (see "Where it lives"). - -Watch alarm renamed and enriched - 2026-08-18: `OrderNotAcknowledged` became `BrokerageOrderNeverNotified` - -the brokerage never notified about the order - with the virtual and the args renamed with it. The args now -resolve the Lean order behind the id (null for a placement whose id was never assigned), carry the watch -duration, and print themselves through `ToString`, which the default warning uses. The message code string -stays `"OrderNotAcknowledged"`, so live logs keep their vocabulary. - -State renamed to snapshot - 2026-08-19: `BrokerOrderState` became `BrokerageOrderSnapshot` - this -document's own word for it, with the `Brokerage` prefix the rest of its family already uses. The -property-fill constructor left with it: every builder goes through the constructors now, and the -constructors default the time to `DateTime.UtcNow` when none is passed. All three adopters and the -tests renamed and simplified with it. - -Mode classes renamed - 2026-08-19: `PerOrderIdPollingService` and `AllOrdersPollingService` became -`SingleOrderPollingService` and `BulkOrdersPollingService` - named for the request shape one sweep -sends: one order per request, or one request for everything the broker lists. The abstract parent took -Lean's `Base` prefix with it: `BaseBrokerageOrderPollingService`. - -Constructor chains and clearer names - 2026-08-19: `watchTimeout` became `notificationTimeout` - the -silence that raises `BrokerageOrderNeverNotified`; the read callbacks became `getBrokerageOrderById` and -`getAllBrokerageOrders`; both mode classes offer three chained constructors instead of optional -parameters; and the defaults read `3 * 1000` and `60 * 1000` milliseconds in code and docs alike. - -Abstract read renamed - 2026-08-19: the subclass hook `Sweep()` became `GetOrderSnapshots()` - the same -get verb as its read callbacks and `Brokerage.GetOpenOrders`, and it names what one pass returns. "Sweep" -stays the prose word for one polling pass. - -Poll vocabulary and registry names - 2026-08-19: "sweep" left every XML doc and code comment - "poll" is -the code word now, this document keeps sweep for prose. The registry record moved into its own file, -`Models/OrderTrackingEntry` with `LastSnapshot` (was a private nested `OrderStateEntry` with `LastSeen`), -and the private fields follow (`_orderEntries`, `_processSnapshot`, `MaxFailedPollsBeforeWarning`). -`GetWatchedBrokerageIds` became `GetOpenBrokerageIds` - it never filtered on the watch flag, and what it -returns is Lean's "open": the end not reported yet - returning `IEnumerable`, copying the entries -under the registry lock and filtering outside it. `Dispose` now always disposes the cancellation source. - -## Purpose - -Three brokerage plugins have already written the same thing, and a fourth one needs it and does not have it: - -- **CharlesSchwab** wrote `Services/OrderUpdatePollingService.cs` so the algorithm keeps working when a second - algorithm takes the single streaming connection away. -- **Public.com** wrote `OrderPollingService.cs` because Public.com has no order stream at all. -- **Tradier** has a smaller version of the same idea written inline in the brokerage. -- **InteractiveBrokers** has the problem and no answer: when the broker does not reply, the algorithm stops. - -Underneath those four there is one problem, not four: **the real-time channel is the only thing telling Lean what -happened to an order, and it is not reliable.** It can be absent, it can be taken away, it can drop for 15 minutes, -and it can stay up while quietly missing an update. In all four shapes the broker still knows the answer over HTTP, -and nobody asks. - -This document proposes one helper class in Lean core, `BaseBrokerageOrderPollingService`, that any brokerage can create -and use. The brokerage picks one of two classes — read one order by its brokerage id, or read all orders — and -hands the service a read callback that converts each order from the broker's own model into one shared snapshot -shape. Every N seconds the service runs the read, and the snapshots travel through the brokerage's message handler -back into the service, which compares each one with the last state it has seen for that order and raises an event -with the order events that are new. The brokerage decides what to do with them. - -This document covers the service only. It does not change any brokerage on its own — each plugin adopts it in its -own pull request. - -## The problem - -### 1. The same loop, written three times - -| Plugin | Class | Size | Mode | Interval | How a result reaches Lean | -| --- | --- | --- | --- | --- | --- | -| CharlesSchwab | `Services/OrderUpdatePollingService.cs` | 245 lines | all orders | `charles-schwab-order-poll-interval-ms`, default `3000` | `_messageHandler.HandleNewMessage` | -| Public.com | `OrderPollingService.cs` | 268 lines | per order id | `OrderPollingInterval` ctor argument | `_messageHandler.HandleNewMessage` | -| Tradier | inline, `TradierBrokerage.cs:1240-1284` | ~45 lines | per order id, one shot | `Task.Delay(2s)` | direct | - -All three copies are deleted by their adoptions; the table records what stood before. The two service -classes were near copies outside their two callbacks. Both had: a background task, a -`CancellationTokenSource` recreated by `Start` and cleared by `Stop`, an idempotent `Start`, a `Stop` that canceled and -waited up to 2 seconds before disposing the source, a `Dispose` that refused to start again, a loop that logged and -retried on a failed read instead of dying, and the same two trace lines. Even the comments matched, because the second -one was written from the first. - -What actually differed was small and none of it was a design decision worth keeping twice: `Task.Run` against -`Task.Factory.StartNew(LongRunning)`, an async fetch against a sync one, `Task.Delay` against -`cancellationToken.WaitHandle.WaitOne`, and a failure counter that only Schwab had. Public had one real extra: a -registry of watched brokerage ids with the last state seen for each (`Models/OrderSnapshot.cs`: status, cumulative -filled quantity, average price). - -Tradier's version was the same idea again in miniature. When a fill arrived for a brokerage id Lean did not know, it -waited 2 seconds, re-checked `_orderProvider.GetOrdersByBrokerageId`, and re-requested the orders from the API -(`TradierBrokerage.cs:1240-1284` before its adoption deleted the path). - -### 2. Two brokerages block the order thread for minutes, then kill the run - -Both CharlesSchwab and InteractiveBrokers place the order, then **block inside the order method** waiting for the -broker to confirm it on the real-time channel. Neither of them ever asked the broker over HTTP instead — -this service is that ask, and Schwab's polled mode now answers the same wait. - -| Brokerage | Waits for | How long | Where the number comes from | When it expires | -| --- | --- | --- | --- | --- | -| CharlesSchwab | `OrderAccepted` on the account activity stream | **3 minutes** | hardcoded `TimeSpan.FromMinutes(3)`, `CharlesSchwabBrokerage.cs:483` | `Error` `MissingWebSocketResponse` (`:485`) | -| InteractiveBrokers | `openOrder` / `orderStatus` / `execDetails` callback | **5 minutes** | `ib-response-timeout`, default `300` seconds, `InteractiveBrokersBrokerage.cs:84` | `Error` `NoBrokerageResponse` (`:1659`) | -| InteractiveBrokers, `MarketOnOpen` / `ComboLegLimit` / `ComboMarket` / `ComboLimit` | same | 10 seconds | `ib-no-submission-orders-response-timeout`, `:90` | Lean **invents** a `Submitted` event (`:1649-1652`) | - -Two costs, and both are paid on every occurrence. - -**The wait itself.** Schwab's is in `PlaceOrder`; IB's is in `IBPlaceOrder` (`:1536`), which serves both `PlaceOrder` -(`:451`) and `UpdateOrder` (`:483`), and `CancelOrder` has its own copy at `:538`. So a single lost message parks an -order thread for three or five minutes while the market moves. - -**The end of the run.** A `BrokerageMessageType.Error` is not a log line. `DefaultBrokerageMessageHandler` turns it -into `SetRuntimeError` (`Common/Brokerages/DefaultBrokerageMessageHandler.cs:92-96`), which ends the deployment. - -The IB plugin already admits the cause in a comment: - -```csharp -// tracks pending brokerage order responses. In some cases we've seen orders been placed and they never get through to IB -private readonly ConcurrentDictionary _pendingOrderResponse = new(); -``` -`InteractiveBrokersBrokerage.cs:160-161` - -And the third row is the worst one, because it is not even a failure the operator can see. IB is known not to send a -submission event for those order types, so after 10 seconds Lean writes the event itself: - -```csharp -Status = OrderStatus.Submitted, -Message = "Lean Generated Interactive Brokers Order Event" -``` - -That event is a guess. The broker is never asked whether the order is there. - -So today the answer to "the broker did not reply" is: block for minutes, then stop the algorithm, or guess. - -### 3. A real-time stream can be down, or just quiet - -Section 2 was about one lost message: the order confirmation never arrives, the algorithm stops or Lean invents the -event itself. But the stream can also fail in two quieter ways: the connection drops, or the connection is fine and -a single update is just never sent. Today nothing recovers from either. - -**The socket drops.** When a brokerage reports `Disconnect` and any exchange is open, Lean does not stop immediately: -it waits `DefaultInitialDelay`, **15 minutes**, for the connection to come back -(`Common/Brokerages/DefaultBrokerageMessageHandler.cs:38` and `:117-122`). That is 15 minutes in which the algorithm -is alive, orders may fill, and no order update can arrive. Worse, nothing re-reads the orders once the socket comes -back: a search across the CharlesSchwab, Tastytrade, Alpaca and TradeStation plugins finds no path that calls -`GetOpenOrders` or re-syncs order state on reconnect. Whatever happened during the gap is lost for the rest of the -run. - -Schwab has the sharper version of this: the account activity stream can be taken away permanently, because Schwab -allows one streaming connection per user and a second algorithm on the same account takes the slot. That is the -reason its polling fallback exists at all. - -**The socket is up but incomplete.** A live connection is not proof that every update arrived. IB -[documents this about its own API](https://interactivebrokers.github.io/tws-api/order_submission.html#order_status#:~:text=There%20are%20not%20guaranteed%20to%20be%20orderStatus%20callbacks%20for%20every%20change%20in%20order%20status.), -and the plugin copies the warning into the code: - -```csharp -// There are not guaranteed to be orderStatus callbacks for every change in order status. For example with market orders -// when the order is accepted and executes immediately, there commonly will not be any corresponding orderStatus callbacks. -// For that reason it is recommended to monitor the IBApi.EWrapper.execDetails function in addition to -// IBApi.EWrapper.orderStatus. From IB API docs -``` -`InteractiveBrokersBrokerage.cs:2553-2555` - -A dropped or never-sent update leaves a Lean order open forever while the broker has closed it. The algorithm then -sizes its next trade against holdings that are wrong, and any cancel or update it sends on that order is rejected. - -A poll answers all three cases with the same question — *does the broker still have this order, and at what -status?* — which is why they belong in one service and not three: - -| Case | Mode that covers it | -| --- | --- | -| No order stream at all (Public.com) | all orders, running for the whole session | -| Stream lost or taken away (Schwab, any disconnect) | all orders, started when the stream goes down and stopped when it returns | -| Stream up but an update never arrived (IB, Schwab acks) | watched ids only, running while something is unconfirmed | - -## The shared input: a snapshot, not a Lean order - -The obvious shared input would be `IBrokerage.GetOpenOrders()` (`Common/Interfaces/IBrokerage.cs:94`) — the one -order read every brokerage already implements. It returns Lean `Order` objects, and that is exactly why it does not -work. - -An `Order` carries `BrokerId`, `Symbol`, `Quantity`, `Price`, `Time`, `LastFillTime`, `LastUpdateTime`, -`CanceledTime`, `Type`, `Status` and `Tag` (`Common/Orders/Order.cs`). **There is no filled quantity and no fill -price on it.** Lean core already says this out loud in the startup path that adopts these orders: - -> Beware that this order ticket may not accurately reflect the quantity of the order if the open order is partially -> filled. - -`Engine/Setup/BrokerageSetupHandler.cs:504` - -The plugins prove the gap themselves. Each one sets `Status` on the orders it returns, and each one does it -differently: IB maps the IB order state (`InteractiveBrokersBrokerage.cs:3318`), Alpaca writes `Submitted`, or -`PartiallyFilled` when `brokerageOrder.FilledQuantity` is between zero and the order quantity -(`AlpacaBrokerage.cs:386-390`), TradeStation does the same from `leg.ExecQuantity` -(`TradeStationExtensions.cs:378`). Two of them **read the broker's filled quantity and then throw the number -away**, because a Lean `Order` has nowhere to put it. All that survives is the word `PartiallyFilled`. - -So the plugins already have these numbers. There is just no field to store them in. That is the whole -fix: instead of the service reading Lean `Order` objects, **the brokerage converts its -own order model into one small shared snapshot and passes that to the service.** The snapshot has fields for the -status, the total filled quantity and the fill price — the three numbers Public's replaced poller tracked -per order (its since-deleted `Models/OrderSnapshot.cs`) — so one compare in Lean core works for every brokerage and can report real -fills, not only that the order exists. - -Two rules shape what the service does with a snapshot: - -1. **The service acts only on what a snapshot says.** The plugin fills the snapshot's status by mapping its - broker's own status, the same way it already does for stream messages. So a `Canceled` in a snapshot is a fact, - not a guess. -2. **A missing order proves nothing.** The service never emits an event because an order stopped appearing in the - reads. An order can be missing because it was filled, canceled, rejected, replaced, or never reached the broker - at all — and each of those needs a different event. If the service guessed `Canceled`, a fill would be lost and - the holdings would be wrong. So when an order stays missing, the service does not decide anything: after a - timeout it tells the brokerage the order was never seen, and the brokerage checks with the broker what happened. - -One more fact, this one about the modes: IB **cannot ask the broker about one order id**. Its API only returns all -open orders (`reqOpenOrders` / `reqAllOpenOrders`, with a 15 second wait — `InteractiveBrokersBrokerage.cs:626`). -So the service cannot require per-order reads from every brokerage. Which orders one read covers is the brokerage's -choice, made when it picks the class. - -## Design - -### Where it lives - -`Lean/Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs` — the base class — with -`SingleOrderPollingService.cs` and `BulkOrdersPollingService.cs` next to it, in namespace -`QuantConnect.Brokerages.Services.OrderPolling`. The data shapes live one level deeper, in `Models/` with -the matching namespace suffix: `BrokerageOrderSnapshot.cs` and `BrokerageOrderNeverNotifiedEventArgs.cs`. - -`Services` is a new folder under `Brokerages`, and it follows the convention the other subfolders there already use: -`Authentication`, `CrossZero` and `LevelOneOrderBook` each take the matching namespace suffix. Each service -takes its own subfolder inside it — `OrderPolling` is the first — so the next plugin service gets a sibling -folder instead of growing one flat namespace. - -Tests go to `Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs`. - -### How a poll flows - -``` -poll thread — one sweep every PollInterval - service calls the read: once per watched id, or once for all orders (by class) - brokerage the read asks the broker and converts each order to a BrokerageOrderSnapshot - service sends each state to the brokerage's message handler - (waits in the queue while an order request holds the lock) - service counts failed reads: three sweeps in a row -> one Warning - service checks the watched orders: one the broker never reported is flagged after a timeout - -handler thread — when the message handler takes a state from the queue - handler dispatches it to what the brokerage registered: service.ProcessOrderState(orderState) - service compares it with the last state seen and keeps only what is new - service raises OrderEvents: the submit first, then fills, then a close - brokerage forwards them: OnOrderEvents(events) -> Lean applies them -``` - -The service does the start and the end: it runs the loop and it compares the states. The brokerage does the middle: -it reads the broker, converts the orders, and says where the results go. That split is what makes the class -generic: nothing broker-specific ever enters it, because the brokerage translated everything into the shared shape -before the service looks at it. And because the service is the one calling the read, a read that throws is caught -and counted by the service itself — the repeated-failure warning below works without any extra code in the plugin. - -Both replaced pollers had exactly this flow: Schwab's loop handed each polled order to -`_messageHandler.HandleNewMessage` and the diff ran when the handler dequeued it, and Public wired its -poller the same way. The service's own loop does the enqueue now. - -### The brokerage order snapshot - -```csharp -namespace QuantConnect.Brokerages.Services.OrderPolling.Models; - -/// -/// One order, as the brokerage last saw it. The brokerage converts its own order model into this shape -/// and passes it to the service, which compares it with the last snapshot seen for the same order and -/// reports only what is new. -/// -public class BrokerageOrderSnapshot -{ - /// The brokerage order id. Some brokers give every combo leg its own id, some give the - /// whole combo one id; the snapshot carries whatever the broker uses. - public string BrokerageOrderId { get; set; } - - /// The Lean status the brokerage maps its broker's own status to. - public OrderStatus Status { get; set; } - - /// The total absolute quantity filled so far. Null when the read does not carry it. - public decimal? FilledQuantity { get; set; } - - /// The price the broker reports for the fills. Null when the read does not carry it. - public decimal? FillPrice { get; set; } - - /// When the brokerage reported this snapshot, in UTC. - public DateTime TimeUtc { get; set; } - - /// The broker's own words for a closing status, e.g. the reject reason. - public string Message { get; set; } - - /// The always-known facts positionally - id and status - with the time defaulting to - /// and the fill numbers and message to null; an overload takes the - /// message without the fill numbers. - public BrokerageOrderSnapshot(string brokerageOrderId, OrderStatus status, DateTime? timeUtc = null, - decimal? filledQuantity = null, decimal? fillPrice = null, string message = null); - public BrokerageOrderSnapshot(string brokerageOrderId, OrderStatus status, DateTime? timeUtc, string message); -} -``` - -Every field except the id and the status is optional, and null means "my read does not know", never "zero". A -brokerage whose read is `GetOpenOrders()` fills only the id and the status, and the service will emit only what -those two can prove. A brokerage whose endpoint returns fill numbers fills `FilledQuantity` and `FillPrice` too and gets fill events. -A brokerage whose endpoint returns a full execution history, like Schwab, reduces it in the mapping: the quantities -sum into `FilledQuantity`, and the newest execution's price becomes `FillPrice`. The service never invents a number -to cover a null. - -The state does not carry an average price or a list of executions, and the service does no price math. -`FillPrice` is used as the broker reported it, and Lean's portfolio averages the fills on its own, like it does -for every other fill event. - -Fees never travel through a poll: both existing pollers report `OrderFee.Zero`, and the snapshot keeps that rule -instead of carrying a fee field nobody fills. - -Combos come in two shapes, and the state carries both without any extra field. Schwab gives every leg its own -brokerage id (`mainId + legId - 1`), so the plugin passes one state per leg. Public.com gives the whole combo -**one** id (`PublicBrokerage.Brokerage.cs:431-442`), so the plugin passes one state and the service fans it out: -`GetOrdersByBrokerageId` returns every Lean leg order behind the id, and each leg's share of a new fill is - -``` -legFill = leanOrder.Quantity * newPart / abs(leanOrder.GroupOrderManager.Quantity) -``` - -The state has no quantity field because the service does not need one: it already knows the brokerage id, so it -reads the group quantity from the Lean orders themselves. That number equals the broker's own order quantity — the -group quantity is exactly what the combo was placed with. Public's replaced diff split fills this way, and -the rule moved into the service's fan-out (`Brokerages/Services/OrderPolling/BaseBrokerageOrderPollingService.cs`). One rule for the mapping follows: -`FilledQuantity` of a shared-id combo is in strategy units, the same units the group quantity counts in. - -A worked example, from a real Public.com order — 5 AAPL strangles, one brokerage id, a put leg and a call leg with -ratio 1 each: - -``` -Lean orders behind the id: put leg Quantity 5 (ratio 1 x group quantity 5) - call leg Quantity 5 (ratio 1 x group quantity 5) -group quantity: 5 (abs(GroupOrderManager.Quantity)) - -broker reports FilledQuantity 2 -> newPart = 2 - 0 = 2 strangles - -put leg: 5 * 2 / 5 = 2 contracts filled -call leg: 5 * 2 / 5 = 2 contracts filled -``` - -The proportion is the point. If the call leg had ratio 2 (Lean quantity 10), the same order-level "2 filled" would -give it 10 * 2 / 5 = 4 contracts — each leg fills at its own ratio, all from one broker number. - -And the other shape, from a real Schwab recording — one combo `OrderResponse`, main id `1002667707949`, three legs, -each leg with its own brokerage id counting up from the main one (`WS_ACCT_ACTIVITY_COMBO_MARKET_FILLED.json` in -the Schwab repo): - -``` -one OrderResponse -> three states, one per leg: - leg 1 -> BrokerageOrderId "1002667707949" (main id + 1 - 1) - leg 2 -> BrokerageOrderId "1002667707950" (main id + 2 - 1) - leg 3 -> BrokerageOrderId "1002667707951" (main id + 3 - 1) - -per state: FilledQuantity = the sum of that leg's execution legs, - FillPrice = that leg's newest execution price -``` - -Here every state finds exactly one Lean order, so there is no split at all — the fan-out and the group quantity -never come into play. The two shapes meet the same diff; only the mapping differs. - -The shape is not guessed — it is what a survey of eight plugins' order reads actually returns. Each column maps to -one field or rule of the state: **filled qty** feeds `FilledQuantity`, **fill price** feeds `FillPrice`, **reason -text** feeds `Message`, and **one id, many Lean orders** is the case the fan-out exists for. A "no" in a cell is -what the nullable fields are for — that broker's state simply carries less, and the service emits less. - -| Broker read | filled qty | fill price | reason text | one id, many Lean orders | -| --- | --- | --- | --- | --- | -| InteractiveBrokers (`reqAllOpenOrders`) | yes — captured but never read | on the paired `orderStatus` callback, not hooked today | no — error callback only | yes, combo legs share the id | -| CharlesSchwab (`GetAllOrders`) | yes | from its execution legs | yes | yes, `mainId + legId - 1` | -| Public.com (`GetOrderById`) | yes | yes | rejects only | yes, combo legs share the id | -| Webull (`GetOpenOrders`) | yes | yes | no | no | -| TradeStation (`GetOrders`) | per leg | yes, as a string | yes | yes, combo legs share the id | -| Alpaca (`ListOrdersAsync`) | yes | yes | no | no | -| Binance (`GetOpenOrders`) | yes | no | no | no | -| Tradier (`GetOrder`) | yes | yes | yes | no | - -The IB row deserves its footnote: an open-orders request is answered with an `openOrder` **and** an `orderStatus` -callback per order -([TWS API docs](https://interactivebrokers.github.io/tws-api/open_orders.html)). The plugin already keeps the whole -`openOrder` payload — `orders.Add((args.Order, args.Contract, args.OrderState))` -(`InteractiveBrokersBrokerage.cs:598`) — and that `IBApi.Order` carries a `FilledQuantity` field. The paired -`orderStatus` callback adds the filled quantity and the average fill price (`Client/OrderStatusEventArgs.cs:38-50`). -Only the last step is missing today: the conversion never reads `FilledQuantity` (no reference anywhere in the -plugin), and `GetOpenOrdersInternal` hooks only `OpenOrder`/`OpenOrderEnd` (`:611-612`), not `OrderStatus`. So the -numbers are already in hand when IB's mapping wants them. - -Three regularities fall out. Every read fills the id and a broker status — the two required fields. Almost every -read fills cumulative quantities, while a full execution list exists at exactly one broker (Schwab) — which is why -the state carries cumulative numbers only and Schwab reduces its legs to them in the mapping. And half the brokers -map one wire order to several Lean orders, so the fan-out is not an edge case. -Nothing common enough to add is missing; the nullable fields cover every "my read does not have it" hole in the -table. Even the ordered quantity needs no field — for the split, the service reads the group quantity from the -Lean orders it already looks up. - -### The class - -```csharp -namespace QuantConnect.Brokerages.Services.OrderPolling; - -/// -/// Reads orders from the brokerage on an interval and turns the returned snapshots into order events. -/// Used when a brokerage has no order stream, when the stream is unavailable, or to resolve an order -/// the broker never replied about. The base class owns everything both modes share — the loop, the -/// watch registry, the compare and the events. What one sweep reads is the subclass: -/// or . -/// -public abstract class BaseBrokerageOrderPollingService : IDisposable -{ - /// Initializes what both modes share: the message handler wiring, the order provider, and - /// the two time settings with their defaults. The service wires the handler both ways itself: it - /// registers and enqueues every polled snapshot, so one handler - /// serializes polled snapshots with everything else the brokerage processes. A null handler routes - /// each snapshot straight into . - protected BaseBrokerageOrderPollingService(BrokerageConcurrentMessageHandler messageHandler, IOrderProvider orderProvider, - TimeSpan? pollInterval = null, TimeSpan? notificationTimeout = null); - - /// One read of the broker, giving the current order snapshots. The loop calls it every - /// poll interval, hands each snapshot to the message handler, and counts a throw as one failed poll. - protected abstract IEnumerable GetOrderSnapshots(); - - /// A copy of the ids a poll still has to read: everything tracked whose end was not - /// reported yet. - protected IEnumerable GetOpenBrokerageIds(); - - /// The order events one snapshot produced. Raised inside , never empty. - public event EventHandler> OrderEvents; - - /// A watched order that nothing reported for notificationTimeout of polling. - /// Raised once; the id is unwatched with it. - public event EventHandler BrokerageOrderNeverNotified; - - /// Several reads in a row failed, so the run currently has no order updates. - public event EventHandler Message; - - /// True while the polling task is running. - public bool IsPolling { get; } - - /// How long the loop sleeps between sweeps. - public TimeSpan PollInterval { get; } - - /// How long a watched order may stay completely unreported, in polling time. - public TimeSpan NotificationTimeout { get; } - - /// Watches a brokerage order id, with nothing seen for it yet. Idempotent: watching an - /// already-watched id never overwrites its state. - public void Watch(string brokerageId); - - /// Watches a brokerage order id, seeded with what another path already reported, so the - /// next poll does not repeat it. Used for orders adopted at startup, for a submit reported from - /// the request path, and to move state onto the new id of a replace. - public void Watch(string brokerageId, BrokerageOrderSnapshot lastSeen); - - /// Watches the new brokerage order id of a replace and drops the replaced id in the same - /// step, so the first state to carry the new id reports the update submit. The new id starts with - /// no fill state; a broker that carries fills across a replace seeds with Watch instead. - public void WatchReplacement(string brokerageId, string previousBrokerageId); - - /// Stops watching an order and drops its state. - public void Unwatch(string brokerageId); - - /// Records what another path already reported for an order, so the next poll does not - /// repeat it. Called by the streaming path while the stream lives. - public void UpdateOrderState(string brokerageId, BrokerageOrderSnapshot orderState); - - /// The last state seen for an order, from any path. The streaming path reads it for its - /// own duplicate check, and a replace reads it to move the state to the new id. - public bool TryGetLastOrderState(string brokerageId, out BrokerageOrderSnapshot lastSeen); - - /// - /// Compares a snapshot with the last state seen for the same order and raises - /// with what is new. The constructor registers it on the message - /// handler, so polled orders queue behind an order request that holds the stream lock. - /// - public void ProcessOrderState(BrokerageOrderSnapshot orderState); - - /// The whole handover from a stream to polling, in the only safe order: process what the - /// stream already delivered, pre-load one watch per open Lean order, then start the loop. A null - /// callback pre-loads nothing. See "Seed before Start". - public void Start(Func preLoadOpenOrders); - - public void Start(); - public void Stop(); - public void Dispose(); -} - -/// -/// For a broker with a get-order endpoint. A sweep calls the read once per watched brokerage id — -/// no request when nothing is watched. A null return means the broker does not know the id, so the -/// notification timeout keeps counting. A read that throws is logged and skipped, so one bad id cannot -/// starve the others; the sweep only counts as failed when every read of the sweep failed. -/// -public class SingleOrderPollingService : BaseBrokerageOrderPollingService -{ - public SingleOrderPollingService(Func getBrokerageOrderById, BrokerageConcurrentMessageHandler messageHandler, - IOrderProvider orderProvider, TimeSpan? pollInterval, TimeSpan? notificationTimeout); - // two shorter overloads chain to it: (read, handler, provider) and (read, handler, provider, pollInterval) -} - -/// -/// For a broker with only a bulk endpoint. A sweep calls the read once, whatever is watched. -/// -public class BulkOrdersPollingService : BaseBrokerageOrderPollingService -{ - public BulkOrdersPollingService(Func> getAllBrokerageOrders, BrokerageConcurrentMessageHandler messageHandler, - IOrderProvider orderProvider, TimeSpan? pollInterval, TimeSpan? notificationTimeout); - // two shorter overloads chain to it: (read, handler, provider) and (read, handler, provider, pollInterval) -} -``` - -The loop, `Start`, `Stop`, `Dispose` and the failure counter come from Schwab's service, the more complete one. -The watch registry, the compare with the last state seen and the seeding of orders already open at startup come from Public's. - -### The two modes - -The mode is the class. Both run the same diff — it lives in the base — and a subclass is only its `GetOrderSnapshots`: - -- **`SingleOrderPollingService`** — `Func`: the sweep loops the watched ids and calls the - read once per id. Nothing watched, nothing requested. Public.com's replaced service had exactly this - constructor — `new OrderPollingService(_apiClient.GetOrderById, _messageHandler.HandleNewMessage, interval)` — - and its adoption passes the same read to `CreateOrderPollingService`. -- **`BulkOrdersPollingService`** — `Func>`: the sweep calls the read once, and the - read returns everything the broker lists. - -Two classes instead of one class with both reads, because a bulk read does not fit a per-id shape: called once -per watched id, it would repeat the full-account request for every id in the same sweep, and with nothing watched -it would never run at all (Schwab's fallback watches nothing — it is the whole order path). The split also keeps -each class clean: one class with both reads would hold a null field for the unused mode and a branch in the loop -to pick the right read; a subclass holds only its own read. - -The watch registry serves both modes. In per-id mode it is also the read list. In all-orders mode it only feeds -the notification timeout: the service still checks that every watched id shows up in the snapshots sooner or later. - -| Plugin | The sweep reads | Scope | Why | -| --- | --- | --- | --- | -| CharlesSchwab | bulk | all | one request returns the whole account, and Schwab is the order path for the run | -| Public.com | per id | watched | Public.com has a get-order endpoint and only cares about its own orders | -| InteractiveBrokers | bulk | watched | IB has **no** per-order request — `reqOpenOrders` always returns everything | -| Tradier | per id | watched | adopted: `GetOrder` asks about one id, so a sweep reads only the watched orders and an idle account sends no request at all (the plan here said bulk; the per-order endpoint made per id the better fit) | -| Webull | bulk | watched | its only order read is `GetOpenOrders` (`Api/ApiClient.cs:666`); order updates come over a gRPC stream, and the poll covers its drops | -| TradeStation | bulk | watched | one `GetOrders` request returns the account's orders (`Api/TradeStationApiClient.cs:185`); the stream stays the first path | -| Alpaca | per id | watched | the SDK has `GetOrderAsync` and the plugin already calls it (`AlpacaBrokerage.cs:546`), so a watch asks only about its own orders | -| Binance | bulk | watched | a single-order request needs the symbol next to the id and the plugin never built one; `GetOpenOrders` sits on the shared REST client base, so every market variant has it | - -### The service never raises Lean order events itself - -The service raises its own `OrderEvents` event, from inside `ProcessOrderState`. It never calls `OnOrderEvents`. The -brokerage routes snapshots through its `BrokerageConcurrentMessageHandler` and forwards the events. - -This is not style. The poll runs on its own thread, so it can see an order already filled while `PlaceOrder` is -still reporting `Submitted` for the same order. If the fill goes out first, the late submit flips a filled order -back to open, and Lean then accepts a cancel or update on it that the broker rejects. Schwab hit exactly this and -fixed it by pushing polled orders through the same message handler as stream messages, so they wait in the queue -while an order request holds the stream lock (`WithLockedStream`, -`Brokerages/BrokerageConcurrentMessageHandler.cs:99`). That is why `ProcessOrderState` is a separate public method -instead of something the loop calls itself: the message handler sits between the read and the diff, -and the queue keeps the order right. The handler's message type also has to cover both the stream's model and the -snapshot — the next section is how it does. - -One requirement travels with the queue: the order request, the record the sweep needs to assign the ids, and -the watch must all happen inside the same `WithLockedStream` block that the snapshots queue behind. The leg ids -themselves are assigned by the first sweep that sees the order, from the snapshot: only the broker says which -leg id belongs to which symbol, so an id derived from the request's leg order would be a guess. The -placement's assignment is the same code the stream's `OrderAccepted` runs, and it is what releases the -plugin's place wait. A replace has a mirror of it, fed from its own pending record: its assignment moves -each Lean order onto its new id and marks it through `WatchReplacement`, so the diff reports the update -submit instead of a plain submit. Nothing waits on a replacement - the replace reply already confirmed it, -and the watch raises `BrokerageOrderNeverNotified` if the broker never lists it. Schwab's fallback path works this -way today. - -### One lock for two message types - -`BrokerageConcurrentMessageHandler` knows exactly one message type, so a plugin whose stream model and snapshot -differ had no way to push both through one lock. Schwab works around it with a marker interface: its stream model -and its polled model both implement `IOrderUpdateMessage`, and the handler is typed to that. This works inside one -plugin that owns both models, and it cannot be the shared answer — every plugin that adopts the service would -have to add the interface to its own wire models, and the core `BrokerageOrderSnapshot` cannot implement a per-plugin -interface. - -So a second, non-generic `BrokerageConcurrentMessageHandler` ships with the service, in the same file -(`Brokerages/BrokerageConcurrentMessageHandler.cs`). It wraps a `BrokerageConcurrentMessageHandler` -inside, so the lock, the buffer and the drain loop are the exact same code, not a copy. Any number of listeners -register, one per message type, and every source enqueues through one `HandleNewMessage(object)`: - -```csharp -_messageHandler = new BrokerageConcurrentMessageHandler(concurrencyEnabled); -_messageHandler.Register(OnAccountContent); // the stream's own type -// the polling service registers its own BrokerageOrderSnapshot listener itself, in its constructor - -// one method for both sources -_messageHandler.HandleNewMessage(accountContent); -_messageHandler.HandleNewMessage(orderState); -``` - -`Register` subscribes a filter: a dequeued message runs one `is` check per registered type and lands in every -listener that matches, in registration order; a message no listener matches is dropped. A plugin that adopts the -service switches its handler field to the non-generic class and turns the constructor callback into one -`Register` call — the stream's hot path pays one type check per registered type, nanoseconds next to the lock the -path already takes. The shared lock across two types has its own tests -(`Tests/Brokerages/BrokerageConcurrentMessageHandlerMultiSourceTests.cs`). - -One rule is absolute: **the generic `BrokerageConcurrentMessageHandler` stays byte-identical to master.** -Thirteen live plugins hold a field of it today — CharlesSchwab, Public.com, Alpaca, Binance, TradeStation, -Tastytrade, Webull, ByBit, Eze, OANDA, IG, dYdX and TerminalLink — plus the Template scaffold. It is not edited, -not even additively: the non-generic class reuses it by composition, and its tests -(`Tests/Brokerages/BrokerageConcurrentMessageHandlerTests.cs`) stay untouched with it. Only a plugin that adopts -the polling service moves to the non-generic class, one plugin at a time. - -### The diff - -`ProcessOrderState` compares the snapshot with the last state it has seen for that brokerage id: - -``` -record the id as seen, for the notification timeout -find the Lean orders by brokerage id (IOrderProvider.GetOrdersByBrokerageId — a list, -because combo legs can share one id and each leg is its own Lean order) - none found -> skip and write nothing: not ours, or ours with the id not on the - Lean order yet — the next sweep sees it again - closed in Lean -> skip, unwatch, drop its state - -the submit first, once: Submitted is emitted when nothing was emitted for the id yet, -the Lean order is still New, and the snapshot is not a reject. Lean requires it before -any fill, and a market order can already be Filled the first time a poll sees it. The -second gate matters in bulk mode beside a live stream: orders the stream already -confirmed are not New in Lean, so a sweep seeing them for the first time stays quiet. -The marked exception is the new id of a replace: WatchReplacement flagged it, its Lean -order is already past New, and the first state reports UpdateSubmitted instead. - -then the fills, so a close can never outrun a fill of the same order: - a fill needs both numbers: without a FillPrice nothing is emitted and - alreadyReported does not move - the service never invents a price - the new part is FilledQuantity - alreadyReported; nothing at zero or below - one fill event, priced at the state's FillPrice - alreadyReported never shrinks, and it moves only when a part was actually emitted - legs sharing one id split the new part by each leg's share of the group quantity, - read from the Lean order's GroupOrderManager - fill quantities are signed by the Lean order's direction; the state stays absolute - an order is Filled once its total covers abs(leanOrder.Quantity), else PartiallyFilled - -the end of the order last: - Canceled / Invalid -> emitted once, with the snapshot's Message; the id leaves the - read list, but its state stays until a compare sees the Lean - order closed - anything already emitted -> skipped -``` - -The cumulative compare is the rule both existing pollers already share: everything at or below `alreadyReported` -was seen before, so a re-read of the same history reports nothing, and a fill the stream already delivered is not -repeated. It only works because `alreadyReported` never moves backwards. - -A worked example — long 1000 AAPL, two 100-share fills at the same price: - -| Broker reports (cumulative) | alreadyReported | New part | Event | -| --- | --- | --- | --- | -| FilledQuantity 100, FillPrice 310 | 0 | 100 | PartiallyFilled +100 at 310 | -| FilledQuantity 200, FillPrice 310 | 100 | 100 | PartiallyFilled +100 at 310 | -| FilledQuantity 200 again, next sweep | 200 | 0 | nothing | - -Two fills at the same price never look alike to the service, because `FilledQuantity` is the running total and -totals only grow. This is also the field's contract for the mapping: the plugin fills in the broker's cumulative -number, never the size of the last fill — with the increment in that field, the second fill above would look -identical to the first and be lost. - -Pricing is deliberately simple: the new part takes the state's `FillPrice`, as the broker reported it. When several -fills land inside one sweep, the quantity is still exact and the price is the broker's reported price at sweep -time, not each fill's own — Tradier's replaced poller shipped exactly this trade-off, and its adoption -keeps it, documented in the mapping (`TradierBrokerage.cs:1170-1172`). Public recovers the exact increment price from the change of the average; -the service refuses that arithmetic on purpose — it amplifies broker rounding and can even go negative on a -tiny part, while the simple price needs no guard at all. Public's adoption kept the recovery anyway, but inside -its own mapping: the state's `FillPrice` arrives already recovered, with a guard that falls back to the plain -average when the new part is not positive. The service still only copies the price it is given. - -The recovery also cannot move into the service today. The state has no average field, so the service never -sees the broker's raw average — the mapping turns it into a fill price first. Keeping it in Public costs one -small map (the previous read's cumulative and average) and one fallback guard only the plugin can judge. It -moves into the service the day a second average-only broker adopts — IB's `orderStatus` reports an average -fill price — as one additive change: a nullable `AveragePrice` on the state and the arithmetic in the service. - -One more detail is load-bearing. **State outlives the terminal event.** Forgetting an order the moment its -`Canceled` goes out re-reports every fill if the next sweep lands before Lean applies the event — Schwab's own ADR -documents exactly this race. So state is dropped only when a compare sees the order closed **in Lean**, never at -emission. - -### Later: orders placed outside Lean - -The "none found" branch skips today, but it is also an opening. An id the order provider keeps not knowing is most -likely an order the user placed outside Lean, in the broker's own app — and Lean already has a door for those: -`OnNewBrokerageOrderNotification` (`Brokerages/Brokerage.cs:258`). The transaction handler picks it up -(`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:190`, `:1674`), asks the algorithm's brokerage message -handler whether to accept the order, and adopts it with `AddOpenOrder`. TradeStation already raises it from its -stream (`TradeStationBrokerage.cs:1088`); a poll can feed the same door. - -Not in this PR, for one reason: telling "placed outside Lean" apart from "ours, id not assigned yet" needs care — -the 2-second recheck Tradier's replaced poll carried existed exactly because an unknown id can turn out to be -ours a moment later. The safe -shape is: only an id that stays unknown across several sweeps, and is not inside any order request, gets raised as -a new brokerage-side order. That rule can be added to the diff later without changing the snapshot or the API, so -it is future work, not part of this design. - -This is the piece the snapshot buys. In the first draft of this document the diff could only ever emit `Submitted`, -because a Lean `Order` carries no fill numbers, and Schwab and Public had to subclass the service to keep their own -diffs. With the snapshot the diff is shared, whole, and the plugins keep only their mapping. - -### Watch mode and the give-up rule - -`Watch(brokerageId)` is called by `PlaceOrder` right after the request returns an id. From then on: - -- A snapshot arrives for the id, or the stream records one through `UpdateOrderState` → the order was seen, - and the id stays watched until the order closes. -- The order is closed in Lean → `Unwatch`, and its state is dropped. -- `notificationTimeout` of polling passes and nothing ever carried the id → `BrokerageOrderNeverNotified` is raised once, with - the id, the Lean order behind it when one resolves (null for a placement whose id assignment never - happened), and how long it was watched; the args print all of it through `ToString`, and the id is - unwatched. - -The timeout only counts while the service is polling, and only on sweeps whose read succeeded — a failed read -asked the broker nothing, so it proves no silence. A watch set while the service is stopped does not count — -otherwise every healthy order would hit the timeout the moment polling starts. So a plugin in watch mode calls -`Start` together with `Watch` (calling `Start` twice is fine) and may `Stop` once nothing is watched. - -`BrokerageOrderNeverNotified` is a question, not a verdict — this is rule 2 again, a missing order proves nothing. The service does -not know whether the order never arrived or filled instantly, so it does not decide. The brokerage handles it: -Public can call its get-order endpoint, Schwab can read the order with its executions, IB can use `reqExecutions`. -A brokerage that handles nothing raises a `Warning` and the run keeps going, which is already better than today's -`Error` that stops it. - -### One registry for the stream and the poll - -In watch mode the stream is alive **while** the service polls, so both paths see the same fills, and each must know -what the other already reported. The registry is that shared memory, in both directions: - -- **The stream writes what it reports.** After reporting a stream fill, the plugin calls `UpdateOrderState` with the - new cumulative state. The next poll compare starts from it and repeats nothing. -- **The stream reads before it reports.** The stream's own duplicate check is not enough in watch mode, because it - does not know what the poll already reported. `TryGetLastOrderState` fills that gap: a stream update at or below - the registry's quantity was already reported — by either path — and is dropped. - -Without that write, watch mode reports a fill twice: the poll reports it, the stream reports the same fill a -moment later, and Lean applies both. This is the one wiring rule a plugin must follow when it polls while its -stream is alive. - -A fallback-mode plugin needs none of that wiring, because its stream never comes back once polling starts. Its -stream handler makes no registry calls — Schwab's keeps only its own cumulative-quantity dictionary, untouched. -Everything it owes the service is one seed at the moment the stream dies (see "Seed before Start"). The registry -complements the plugin's existing logic, it never replaces it — and in fallback mode the stream and the service -never even touch while the stream lives. - -A place and a replace are reported the same way: not by the plugin. For a place, Schwab watches the main id -and keeps the placement's Lean orders by symbol; the first sweep to see the order assigns each leg id from -the snapshot — one shared method with the stream's `OrderAccepted`, because only the broker says which leg id -belongs to which symbol — and reports the submit through the diff, one poll interval later at most. For a -replace, its own pending record goes in under the new main id, and a mirror of the assignment handles it: it -moves each Lean order onto its new leg id and marks it with `WatchReplacement`, which drops the replaced id -in the same call — so the dead order's last snapshot reports nothing — and makes the diff report the update -submit instead of a plain submit. The new id starts with no fill state, because a Charles Schwab replacement counts -its executions from zero; a broker that carries fills across a replace moves the state instead: -`TryGetLastOrderState(oldId, out var lastSeen)`, then `Watch(newId, lastSeen)`, then `Unwatch(oldId)`. - -The seed rule stays for what a plugin still reports itself: it seeds in the same locked block, so the next -sweep does not repeat it. The assignment also releases the three minute place wait, so the same -`MissingWebSocketResponse` error guards a placement nothing ever confirms, on the stream and on the poll -alike. A replacement has no wait of its own - the replace reply already confirmed it. The watch doubles as -the alarm: an id the broker never lists raises `BrokerageOrderNeverNotified` instead of staying silent. - -### A replace, across the brokers - -A replace is the one order action that can change the key the registry lives on. Whether it does is the -broker's design, not the plugin's choice — so before deciding what the service should own here, the update -path of nine sibling plugins was read, twice (a survey pass and a line-by-line check pass), for the four -facts that matter to a poll: does the id change, where does the new id come from, who reports -`UpdateSubmitted` today, and what happens to the fill count. - -| Plugin | Update | Id after a replace | New id from | `UpdateSubmitted` reported by | Fills after | -| --- | --- | --- | --- | --- | --- | -| CharlesSchwab | yes, combos too | new — one per leg | REST reply, legs from snapshot/stream | plugin after REST (poll mode); stream `ChangeAccepted` | reset | -| Tastytrade | yes, combos too | new — one for all legs | REST reply | stream `Routed`/`Live`, waited 100 s | unknown | -| Alpaca | yes, no combos | new | REST reply | stream `Replaced` only | unknown | -| InteractiveBrokers | yes, combos too | same — modified in place | — | stream `orderStatus` + an updated flag | carry over | -| TradeStation | yes, combos too | same | — (the reply's `OrderID` is never read) | plugin after REST; the stream echo is swallowed | carry over | -| Tradier | price and type only | same | — | plugin after REST | carry over | -| Public.com | single-leg only | same | — (the reply echoes the id) | plugin after REST | carry over | -| Webull | single-leg only | same (`client_order_id` kept) | — | stream `MODIFY_SUCCESS` | carry over | -| ByBit | futures amend only | same | — | plugin after REST | carry over | -| Binance | no — cancel and re-create | — | — | never emitted | — | - -The rows in code: IB re-sends the same broker id (`InteractiveBrokersBrokerage.cs:1599,1626`), TradeStation -PUTs to the existing id and never reads the reply's `OrderID` (`Api/TradeStationApiClient.cs:329-335`), -Tradier's PUT has no quantity parameter at all (`TradierBrokerage.cs:476-497`), Public replaces in place -with the id echoed back (`Api/ApiClient.cs:351-362`), Webull keeps the `client_order_id` that is the Lean -`BrokerId` (`Api/ApiClient.cs:595-605`), ByBit amends futures under the same id -(`Api/BybitTradeApiEndpoint.cs:94-100`) and cannot amend spot (`BybitBrokerage.Brokerage.cs:200-204`), and -Binance's `UpdateOrder` throws (`BinanceBrokerage.cs:346-349`). - -**The same-id half needs nothing new from the service.** The watched id survives the replace, and the fill -count continues on it — all six carry-over cells are verified in code, e.g. TradeStation's cumulative -`ExecQuantity` delta is never reset by an update (`TradeStationBrokerage.cs:1187-1194`). And the report -itself can never come from a sweep: the state carries a status and two fill numbers, no price and no -quantity, so a modified order polls exactly like an unmodified one. The rule for a same-id plugin is -Tradier's and Public's, already shipping: report `UpdateSubmitted` right after the REST reply -(`TradierBrokerage.cs:948-949`, `PublicBrokerage.Brokerage.cs:693`) and leave the registry alone — Public -runs on this service today and its `UpdateOrder` makes no service call. The two that report from the stream -instead (IB `InteractiveBrokersBrokerage.cs:2359,2380`, Webull `WebullBrokerage.Brokerage.cs:448-453`) -simply have no update report while their channel is down; on adoption they move the report next to the REST -reply, like the other four. - -**The cancel-replace half is where the report can move into the service.** All three learn the new id -synchronously, from the REST reply itself: Schwab's `UpdateOrder` result, Alpaca's `PatchOrderAsync` -response (`AlpacaBrokerage.cs:782-793`), Tastytrade's `ReplaceOrderById` return -(`Api/TastytradeApiClient.cs:172-186`). So the plugin can always watch the new id inside the same locked -block as the request — the same rule the place already follows. What no REST reply gives is the -confirmation that the replacement is live: Tastytrade holds its `UpdateSubmitted` until the stream says the -new order is `Routed`/`Live` and waits 100 seconds for that (`TastytradeBrokerage.Brokerage.cs:431,530-533`), -and Alpaca reports it only from the stream's `Replaced` event (`AlpacaBrokerage.cs:677,1058-1059`), so with -the stream down it is never reported. The first sweep that lists the new id is exactly that confirmation. -So the general shape is one addition, the mirror of the place rule: a **replacement watch** — -`WatchReplacement(newBrokerageId, previousBrokerageId)` marks the new id and drops the old one in one -locked step, and the diff's first state for a marked id whose Lean order is open but past `New` reports -`UpdateSubmitted`, "Update submitted by polling". Combos fit both shapes: Tastytrade's whole combo takes -one new id (`TastytradeBrokerage.Brokerage.cs:416`), and Schwab's per-leg ids go through the same by-symbol -assignment its place already runs. - -**Dropping the old id is a correctness step, not housekeeping.** A cancel-replace ends the old order at the -broker, and the old order's last snapshot says so — Schwab lists it `Replaced`, Tastytrade sends -`Cancelled` for it, Alpaca `Replaced`. A registry still holding the old id could read that as the Lean -order's end. The streaming plugins prove the hazard is real: Tastytrade swallows exactly this `Cancelled` -today (`TastytradeBrokerage.Brokerage.cs:575-581`), and Schwab's status mapping keeps `Replaced` -non-terminal on purpose. The replacement watch removes the old entry in the same call, and once the plugin -re-keys the Lean order the old id no longer resolves — both guards, one step. - -**The fill count restarts with the id.** Schwab's replacement counts its executions from zero, so the new -entry starts at zero reported. Alpaca and Tastytrade leave no evidence either way in code or tests — -"unknown" is the honest cell. A broker that turns out to carry fills across a replace seeds instead of -starting fresh: `TryGetLastOrderState(oldId)` then `Watch(newId, lastSeen)`, two calls that already exist. - -**Telling Lean about the new id stays in the plugin.** Three plugins, three conventions: Schwab swaps the -whole `BrokerId` list through `OnOrderIdChangedEvent`, Tastytrade does the same right after the REST reply -(`TastytradeBrokerage.Brokerage.cs:416`), and Alpaca appends the new id and reads `BrokerId.Last()` from -then on (`AlpacaBrokerage.cs:789-793`). The service never touches `Order.BrokerId` — the plugin tells -Lean, the service only watches. - -So the seed rule above stands for the same-id half — what a plugin reports itself, it seeds — and the -cancel-replace half gets the replacement watch instead: one additive method and one diff branch. Schwab, -the pilot, runs on it: its polled replace goes through a mirror of its place's by-symbol assignment and the -diff reports the update submit. The by-position leg id derivation its first pilot shipped with is deleted. - -### A cross-zero order, two ids - -A broker that cannot cross a position through zero gets two brokerage orders for one Lean order: a closing -leg, then an opening leg the base class places when the first one fills -(`TryHandleRemainingCrossZeroOrder`, `Brokerages/Brokerage.cs:892`). That breaks the diff's frame twice. -The diff calls a fill final only when the cumulative reaches the Lean order's whole quantity, and neither -leg reaches it alone. And the trigger for the second leg is the first leg's *broker-side* fill — a state -the diff never surfaces, because at that point the Lean order is only half done. - -Tradier, the first adopter with this split, keeps both jobs in the read, on the base helpers that already -own the pending leg: - -- The read sees the closing leg filled at the broker and hands `TryHandleRemainingCrossZeroOrder` a - `Filled` event carrying the unreported part. The helper knows whether a remaining leg is pending — for - every other order it declines and nothing happens. When it takes the event, it reports the fill as - `PartiallyFilled` and places the second leg itself; the read then records the state through - `UpdateOrderState`, so the sweep's own diff stays silent. -- The second leg's watch seed carries what the first leg filled, and its reads add the same offset to the - leg's own cumulative. The diff's frame is whole again: the leg's last fill reaches the Lean quantity and - reports `Filled`. A second leg the order provider has not indexed yet resolves through the base - cross-zero map instead, and the hook reports its closing fill itself, marking it reported so the diff - stays silent. - -The service itself needs nothing new for this — the seed, `UpdateOrderState` and `TryGetLastOrderState` -were already there. What it costs the plugin is two small maps holding the first leg's filled quantity — -one keyed by the Lean order id between the second leg's request and its watch, one keyed by the second -leg's brokerage id for the reads — and one closed-order hook in the read. - -### Seed before Start - -Polling never starts first: the stream reported orders before it, and the registry must know what was already -reported before the first sweep runs. So every `Start` that follows stream time begins with a handover, and the -service owns its order — `Start(preLoadOpenOrders)` does nothing while polling already runs, and otherwise runs three -steps: - -1. Drain the message handler — the service runs an empty `WithLockedStream` block on the handler it was built - with, so every fill the stream already delivered is counted. Nothing slips in after it, because the switch - runs on the stream's own thread — the only thread that delivers stream messages. -2. One `preLoadOpenOrders(openLeanOrder)` call per open Lean order, each becoming a `Watch(id, lastSeen)`: the - plugin returns the brokerage id, the order's status and the cumulative filled quantity from its own - bookkeeping. Orders the stream already closed need no seed — the diff skips every order Lean has closed, - and a null return skips the order. -3. Start the loop. - -A plugin with nothing to hand over — its stream reported nothing, or it has no stream — calls the plain -`Start()` instead, and a null callback pre-loads nothing. The first sweep then continues from what the -stream reported instead of repeating it. A fallback-mode plugin makes this one call, because its stream -never comes back — Schwab's `ToPreLoadState` is the working example of the callback. A gap-mode plugin repeats -the call on every drop, and `Stop`s when the stream returns. - -The seed source already exists in most streaming plugins, because they keep the same bookkeeping Schwab does — the -cumulative quantity already reported, per Lean order: Webull's `_orderIdToPreviousCumulativeQuantity` -(`WebullBrokerage.cs:78`), ByBit's `_cumulativeFillQuantity` (`BybitBrokerage.Messaging.cs:42`), Alpaca's and -TradeStation's `_orderIdToFillQuantity` (`AlpacaBrokerage.cs:59`, `TradeStationBrokerage.cs:132` — both signed, so -their seed takes the absolute value). Webull is the clearest next adopter: a dropped stream is a blind gap today, -nothing replays the missed events, and its `StreamDisconnected`/`StreamReconnected` events are ready-made -`Start`/`Stop` triggers. TradeStation needs the seed only for the outage window itself, because its server replays -an order snapshot on every reconnect. Two would add the dictionary first: Tastytrade tracks processed fill ids -instead of quantities, and Binance keeps nothing — its stream reports the per-fill delta the wire sends. - -### Start and Stop follow the stream - -`Start` and `Stop` are the hook for the disconnect case. The brokerage calls `Start` when the real-time channel goes -down and `Stop` when it comes back, so polling covers exactly the window in which the stream cannot deliver. Both are -idempotent, and `Stop` does not dispose the service, so a run can switch back and forth as many times as the socket -does. `Dispose` is the one-way door. - -Three rules for a plugin that uses this: - -- **Seed before every `Start`.** While the socket was up the stream was reporting and the registry was not - listening. The pre-loading `Start` overload hands over what was reported, in the right order — see "Seed before Start". -- **Sweep once after the stream returns, before stopping.** The socket coming back does not replay what it missed, - and no plugin re-reads orders on reconnect today. One last sweep closes the gap. -- **Coming back is not always allowed.** Schwab must stay on polling for the rest of the run, because reconnecting - takes the single streaming slot back from the other algorithm. That decision belongs to the plugin, not to this - service, which is why the service only offers `Start` and `Stop` and never reconnects anything itself. - -What that sweep recovers depends on what the plugin's read returns. A read that only lists open orders brings back -missed submissions. A read with fill numbers brings back the missed fills too — Schwab's does. The service reports -what the states can prove, nothing more. - -### Repeated read failures - -A single failed read is logged and retried on the next sweep. Three failures in a row raise one -`BrokerageMessageType.Warning` through the `Message` event, and a later successful read arms the warning again. This -is Schwab's rule, kept as is: while the sweeps are failing the run may have no order path at all, and a log line -alone leaves the algorithm looking idle for no visible reason. - -Never an `Error`. An `Error` ends the run, which is the outcome this service exists to avoid. - -### What the service keeps, and what it does not - -The service owns the watch registry, the last snapshot seen and the already-reported quantity per order, the failure -counter, and the polling task with its cancellation source. The streaming path shares the snapshot registry through -`UpdateOrderState` and `TryGetLastOrderState`, so the poll and the stream never report the same fill twice. The service -does **not** own the Lean order state: that is read from `IOrderProvider` on every compare, so the service never -drifts from what Lean actually knows. - -Four kinds of thread touch that state: the poll loop, the handler thread inside `ProcessOrderState`, the order -threads through `Watch`/`Unwatch`/`UpdateOrderState`, and the notification-timeout check. One internal lock protects the -registry from all of them. A per-id sweep copies the watched ids under that lock before it reads the broker, so -`PlaceOrder` can watch a new id while a sweep runs. `ProcessOrderState` must not run twice at the same time, and -the service does not guard that itself: the message handler already runs it one call at a time, and a plugin -without a handler must make its calls run one at a time too. - -### Configuration - -Both time settings are optional constructor arguments. A plugin that has its own configuration key — Schwab keeps -`charles-schwab-order-poll-interval-ms` — reads it and passes the value in, so no existing deployment changes. A -plugin that passes nothing gets the shared defaults, resolved once in the base class constructor both subclasses -chain to: - -```csharp -PollInterval = pollInterval ?? TimeSpan.FromMilliseconds(Config.GetInt("brokerage-order-poll-interval-ms", 3 * 1000)); -NotificationTimeout = notificationTimeout ?? TimeSpan.FromMilliseconds(60 * 1000); -``` - -`brokerage-order-poll-interval-ms` is one new generic config entry, so the interval can be tuned once for every -brokerage that uses the default. Core helpers already work this way: the message handler reads -`brokerage-concurrent-message-handler-buffer-size` in its constructor -(`Brokerages/BrokerageConcurrentMessageHandler.cs:56`). The 3000 ms default is the value Schwab runs today. - -## Wiring, per plugin - -The general shape, for a streaming brokerage with a bulk endpoint. The root `Brokerage` class owns the -wiring: one protected create call builds the service, forwards its events onto the brokerage events, -stores it in the protected `OrderPollingService` property and hands it to `Dispose`. The read callback's -signature picks the mode — a bulk read can only build an `BulkOrdersPollingService`: - -```csharp -// bulk broker: one request per sweep reads the whole account. The service wires itself onto the -// handler: it registers its diff as a listener and enqueues every snapshot, so the plugin never -// touches that relationship again. -CreateOrderPollingService( - () => _apiClient.GetAllOrders().Select(ToOrderState), // read: model -> snapshot - _messageHandler, - _orderProvider, pollInterval: TimeSpan.FromSeconds(3), notificationTimeout: TimeSpan.FromMinutes(1)); -``` - -A per-id broker only changes the read — one call replaces Public.com's construction, its three event -forwards and its dispose line: - -```csharp -CreateOrderPollingService(ReadOrderState, _messageHandler, _orderProvider, pollInterval: OrderPollingInterval); -// BrokerageOrderSnapshot ReadOrderState(string brokerageId): get-order by id, a 404 maps to null. -// Public keeps its own key, public-order-poll-interval-ms, default 1000 ms. A plugin that passes no -// interval gets the shared brokerage-order-poll-interval-ms default, 3000 ms. -``` - -What silence means is one override per plugin: `OnBrokerageOrderNeverNotified` defaults to a single -warning built from the brokerage name, and Schwab overrides it to keep the wording its live pilot -verified. - -InteractiveBrokers, the plugin with no answer today, adopts `BulkOrdersPollingService` (it has no per-id request) with -the thinnest possible mapping — id and status from the orders `reqAllOpenOrders` returns, fill fields left null. -The fill numbers are not out of reach: the captured `IBApi.Order` already carries a `FilledQuantity` field, and the -paired `orderStatus` callback adds the average fill price, so the mapping can grow later without a new endpoint. -The first step stays thin: - -```csharp -// IBPlaceOrder, instead of blocking up to 5 minutes and then killing the run -OrderPollingService.Watch(ibOrderId.ToStringInvariant()); -OrderPollingService.Start(); // idempotent; Stop once nothing is watched -``` - -One honest cost first: IB has no `BrokerageConcurrentMessageHandler` today — only Schwab and Public do — so its -adoption starts by putting one, or an equivalent lock, between its order path and `ProcessOrderState`. Without that, -the fill-before-submit ordering this design depends on does not exist in IB. - -The 5 minute block goes away: `IBPlaceOrder` returns as soon as the request is out, and the watch resolves the order -in the background. The `_noSubmissionOrderTypes` guess (`MarketOnOpen`, `ComboLegLimit`, `ComboMarket`, `ComboLimit`) -becomes a real check: the broker either lists the order, and `Submitted` is true, or it does not, and the brokerage -is asked instead of Lean inventing an event. - -The same two lines cover a dropped socket, in any streaming plugin: - -```csharp -// where the brokerage already handles the connection going down; see "Seed before Start" -OrderPollingService.Start(ToPreLoadState); - -// on reconnect: one last sweep for the gap, then hand the job back to the stream -OrderPollingService.Stop(); -``` - -CharlesSchwab and Public.com delete their own service class **and their diff**, and keep only the mapping from -their models to the snapshot. Schwab keeps its own rule of never going back to the stream. Tradier deletes its -fill timer and `CheckForFills` diff the same way; its cross-zero split is "A cross-zero order, two ids". - -## What stays in the plugin - -- Reading the broker: the read callback and the class choice between per-id and all orders, plus how far - back a bulk read reaches (Schwab reads from its oldest open Lean order). -- Converting model to state: the status mapping, combo leg ids (Schwab's `mainId + legId - 1`), reducing an - execution history to the two numbers (Schwab sums its execution legs and takes the newest leg's price). A plugin - whose leg ids are derived rather than returned by the broker should verify they resolve and warn when they do - not — the service skips silently. -- Routing: handing the create call its message handler, or null when the poll loop is the only caller of - the diff. The event forwards and the dispose are the base class's job. -- Reporting without the stream: the plugin reports neither the place nor the replace itself. It watches the - main id, the first sweep assigns the leg ids by symbol — a replacement's through `WatchReplacement` — and - the diff reports the submit or the update submit. -- Deciding what a never-notified order means — the `OnBrokerageOrderNeverNotified` override; the default - is one warning built from the brokerage name. - -## Testing the polling in a plugin - -The service's own behavior — the diff, the watch, the registry — is covered once, in Lean's -`Tests/Brokerages/Services/OrderPolling/BrokerageOrderPollingServiceTests.cs`. A plugin does not test the service -again: it tests its read, its mapping and its wiring **through** the service. The offline tests carry -the everyday coverage; the live tests prove the whole path against the real broker — and for a plugin -starting from nothing they come first, because their runs record the data the offline tests replay. The reference is the Schwab pilot's fixture, -`Lean.Brokerages.CharlesSchwab/QuantConnect.CharlesSchwabBrokerage.Tests/CharlesSchwabBrokerageOrderUpdatePollingTests.cs`; -its mock classes sit next to it in `Tests/Models`. For a polling-primary, per-id plugin the reference is -Public.com's fixture, `Lean.Brokerages.Public/QuantConnect.PublicBrokerage.Tests/PublicBrokerageOrderPollingTests.cs` — -the first one built in this section's order, live capture first. - -### The test doubles - -Two subclasses, and no reflection: - -- A mock brokerage deriving from the real one. It runs the normal initialization and overrides exactly - the edges: the API client factory returns a mock, the socket is replaced with one a test can feed - captured messages into, and the protected polling trigger gets a public wrapper (Schwab's - `SwitchToOrderUpdatePolling` calls its stream-loss handler; a polling-primary plugin just calls - `Connect`). It also plays the transaction handler's part where a test has none: its - `OnOrderIdChangedEvent` override moves the new broker id onto the order, so a sweep can find a - replacement. -- A mock API client deriving from the real one. The read's source is a settable property (Schwab: the - order collection its bulk read returns; a per-id broker sets the get-order response), and an - `AutoResetEvent` fires when the read runs, so a test waits for the next sweep instead of sleeping. - -A test stages broker time by swapping that property between waits: set the working snapshot, wait for -the events, set the half-filled snapshot, wait, set the filled one. One config line per test shortens -the poll interval (Schwab sets `charles-schwab-order-poll-interval-ms` to 100 ms), and the fixture's -setup puts it back, so no test changes the pace of the next one. - -And the test plays the transaction handler: every fixture subscribes to `OrdersStatusChanged` and -writes each reported status back onto the Lean order before anything else (Schwab's `ApplyOrderStatus` -helper) — the same closing of the loop Lean does live. The diff decides from the Lean order: the -submit goes out only while the order is still `New`, and an id leaves the registry only once its order -closed. A fixture that skips the write-back gets duplicate submits and orders that never stop being -read. The asserts then follow the live trading cycle, on the collected events in the order it produces -them: the submit, the fills, the close. - -### The mock data is recorded, not invented - -Every snapshot and stream message in the offline tests is a copy of something the broker really sent. -That sets the build order for a new plugin: one with no `TestData/` folder and no captured payloads -cannot start with the offline tests, because there is nothing to feed them. It starts with one live -test — write it, and run it by hand against a real account. The debug log of that run holds the -broker's real JSON responses and raw socket messages; copying them out is what creates the mock data, -and the offline tests come second, replaying it. Getting a payload is a manual, one-time step: - -- REST order JSON comes from one live run: run the plugin (or one of the live tests below) with debug - logging and copy the orders-endpoint response from the log, or save it to a `TestData/*.json` file - (Schwab keeps `REST_GET_OPEN_ORDERS.json`). Paste it with its real ids, and write a comment naming - the recorded order and its story — "order 1002449136610: two option legs, ended EXPIRED". -- Raw socket messages are copied from the same live log (Schwab's sits at `bin/Debug/log.txt` after one - Explicit run) and fed through the mock socket, so the wrapper handles them exactly like live bytes. -- A staged sequence interpolates only what the stage changes into one recorded skeleton — Schwab's - `BuildRigMarketOrderJson(status, filledQuantity, executionActivities)` replays a real market order's - five executions one sweep at a time. - -Copy the nulls too. A real Public.com order carries `"filledQuantity": null` until something fills, -`"legs": null` outside a combo, and `"limitPrice": null` on a market order; a builder that writes `"0"` or `[]` -instead feeds the code a shape the broker never sends, and the test passes on the wrong parse path. The -staged skeleton keeps every field exactly as recorded and changes only what the stage changes. - -A note for the AI model driving an adoption: the log only records what the code writes to it. Before -the capture run, go through every request path the polling touches — the order read, place, replace, -cancel, and the socket's message handling — and check that each one traces the broker's raw response -(`Log.Trace`). Where a path is silent, add the trace and warn the developer which paths were silent: -the capture run only records payloads on a build that has the logging. - -Short payloads stay inline in the test as verbatim strings; only large or shared ones go to -`TestData/*.json`. - -### Offline tests - -These run without credentials, through the real order methods of the mock brokerage, and no request -leaves the test. What they must show, per plugin — the list is the checklist for the next adopter: - -- **Place while polling**: the sweep reports the submit; a recorded terminal snapshot reports the close - once; a rejected snapshot reports `Invalid` carrying the broker's words; the fills of one sweep - arrive as one event; a staged sequence reports every partial fill; a working order without fill data - does not stop the sweep. A combo plugin adds the leg id assignment from the snapshot by symbol. -- **Cancel and its races**: the cancel request stays quiet and the poll reports the `Canceled` once; an - intermediate `CancelPending` read reports nothing; a fill that beats the cancel ends the order `Filled` - with no `Canceled` at all; a canceled shared-id combo reports one `Canceled` per leg. -- **An id the broker does not know yet**: the read returns null (Public.com's get-order 404) and the - sweep asks again until the order appears — the submit is reported then, not before. -- **Stream-to-polling handover**, for a hybrid plugin: the stream reports part of a fill and the seeded - poll reports only the rest; the same fill split differently by the two paths stays consistent; a leg - the stream already closed is not repeated by the poll. A polling-primary plugin proves the seed with - the orders adopted at startup: one adopted half-filled reports only the part that fills after. -- **Event ordering**: a fill arriving within the first sweep still reports `Submitted` before `Filled`, - and `UpdateSubmitted` before `Filled` after a replace — the proof the message-handler lock holds. -- **The whole lifecycle**: place, update, cancel while polling, and the same on the stream, so the two - paths can be compared event for event. -- **The plugin's own edges**: Schwab adds its sweep window (a read reaches back to the oldest open - order) and its closed-socket subscription guards; Tradier adds its cross-zero legs. - -### Live tests - -The live half is `[Explicit]`, run by hand during market hours against a real account, with the cost -spelled out in the reason string — "takes the single streamer connection", "places real orders", "buys -real shares" — so nobody runs it by accident. Schwab's `WebSocketVsPolling` category runs two brokerage -connections side by side: the first loses the stream and falls back to polling, the second keeps -streaming, and every test asserts the same lifecycle on both. What proved worth copying: - -- Place-update-cancel first: the full lifecycle on a limit order resting far from the market costs - nothing and checks the submit, the update submit and the cancel on both connections. -- Fills on a cheap liquid stock next: market and limit orders sized to a few dollars, run until - `Filled`. -- Account hygiene inside the test: a flat-account pre-check that sells leftovers before asserting, a - sell-back after every real buy — seeding the holdings first, so the sell maps to the right - instruction — and `cleanup:` log lines separating the hygiene from the behavior under test. -- Partial-fill hunting is its own test and it logs instead of asserting: a bid resting inside the - spread for the whole balance may catch a partial fill, but nothing can force one, so a hard assert - would only make the test flaky. The developer adjusts the price to the live quote before running. -- One live run doubles as the recorder: its debug log is where the offline tests' REST and socket - payloads come from. A polling-primary plugin turns the logging on inside the test itself - (`Log.DebuggingEnabled = true`), so every hand run records — and every extra asset class is one more - capture run, the way Public.com recorded its option and its multi-leg bodies after the equity ones. - -## Alternatives not taken - -- **Hand the service Lean `Order` objects from `GetOpenOrders()` and let it diff those.** The first draft of this - document. It dies at the boundary: a Lean `Order` has no filled quantity and no fill price, so the shared diff - could only emit `Submitted`, and every brokerage with richer data had to subclass the service and override the - diff. The snapshot carries the same numbers the plugins already read and today throw away, so the subclass and - the override are gone. -- **A core interface the wire model implements** (`OrderResponse : IBrokerageOrderSnapshot`), so the plugin passes its - API model straight in with no conversion. Checked against all eight surveyed plugins and rejected on the - evidence. Two cannot implement it at all: IB's order model is compiled into the vendor `CSharpAPI.dll`, and - Alpaca's `JsonOrder` is `internal sealed` inside the SDK — both would need a wrapper class, which is the same - work as filling the snapshot. Tradier's model exposes public **fields**, which cannot implement interface - properties, so its whole serialization surface would have to change. TradeStation's model is a struct, boxed on - every interface use. And for the four brokers where one wire order becomes several Lean orders, one object - cannot be several states — the per-leg conversion survives anyway. The interface would also pull the - broker-to-Lean status mapping inside the wire DTOs as computed properties, and the service registry would hold - whole wire objects alive as stored state (Schwab's `OrderResponse` carries the full execution history). A plain - class the plugin fills is one pattern that works for all eight. -- **A marker interface as the message handler's type** — Schwab's current answer to two message types in one - handler (`IOrderUpdateMessage`). As the shared answer it fails the same way the core interface does: every - plugin edits its wire models, and the core `BrokerageOrderSnapshot` cannot implement a per-plugin interface. - Replaced by the non-generic multi-source handler (see "One lock for two message types"). -- **A dual-generic handler**, `BrokerageConcurrentMessageHandler` with the stream type and the polled type. - Rejected: every existing plugin migrates to the new shape even with no polling, a plugin with no stream (IB) - has no honest `T`, and a third source would need ``. The non-generic handler adds a `Register` call - instead of a type parameter. -- **A handler base class that queues work items (`Action`) instead of messages.** The typed wrapper would then - wrap every stream message in a new closure — one allocation per message on the hottest path a brokerage has. - The non-generic handler keeps the message itself in the buffer and allocates only at registration. -- **Put it on the `Brokerage` base class, driven by the engine, like the cash sync.** - `ShouldPerformCashSync` / `PerformCashSync` (`Brokerages/Brokerage.cs:578`, called from - `Engine/TransactionHandlers/BrokerageTransactionHandler.cs:731`) is the existing shape for core-driven periodic - work, and it was the obvious candidate. Rejected: cash sync runs once a day on a schedule core can decide, while - the right poll interval here is a property of the broker's rate limits and of whether its stream is alive. It also - has to be off by default — most plugins must not poll — and a base-class hook that nearly everyone turns off is - worse than a class you create when you need it. -- **Let the service call `OnOrderEvents` directly.** Simpler to wire and reintroduces the fill-before-submit bug in - every plugin that uses it. See "The service never raises Lean order events itself". -- **Emit `Canceled` when an order leaves the open list.** This is the tempting one, and it is wrong: an order that - filled also leaves the open list, so this drops fills and desynchronizes holdings. It is why rule 2 exists — the - service acts on what a snapshot says, never on an order going missing. -- **Carry the execution history in the state** — a per-execution list of quantity, price and time, so every - execution becomes its own event at its own price. The survey killed it: exactly one of the eight brokers (Schwab) - returns such a list, and the cumulative compare already keeps quantities exact. The list would buy per-execution - price precision for one broker at the cost of a second diff branch every plugin has to reason about. If it is - ever wanted, it comes back as one additive nullable field without breaking anyone — the same goes for a - `GetOrderExecutions` API on `IBrokerage`. -- **Arm the replacement watch from the `OrderIdChanged` event instead of `WatchReplacement`.** The plugin - already raises `OnOrderIdChangedEvent` when a replace re-keys a Lean order, so the service could subscribe - to it instead of offering a method. Rejected three times over. The event does not mean "replace": Lean - core raises it on a plain place — the cross-zero flow assigns the second part's id through it - (`Brokerages/Brokerage.cs:940`) — and Binance assigns every initial id with it, so the service would - report an update submit for a placement. The event does not carry the previous id, and the service cannot - recover it: the transaction handler subscribes first and swaps `order.BrokerId` before a later subscriber - runs (`Engine/TransactionHandlers/BrokerageTransactionHandler.cs:1497`), so the old registry entries could - not be dropped. And a stream-alive plugin raises the event while reporting `UpdateSubmitted` itself, so an - event-armed service would report it a second time. The reuse that works is locality, not wiring: the - plugin calls `WatchReplacement` on the same line where it builds the id-changed event, with the old and - new id it already holds. -- **Leave it in the plugins.** It is already written three times, and the fourth copy would be IB's. - -### A base brokerage class that owns the wiring - -Every adopter wires the service the same way: create the mode class with the read callback, forward -`OrderEvents` to `OnOrderEvents` and `Message` to `OnMessage`, decide what `BrokerageOrderNeverNotified` means, and -dispose the service with the brokerage. So the tempting next step is to move that wiring into an inheritance -layer — one abstract brokerage class that owns the service and asks the plugin only for overrides: - -```csharp -public abstract class BaseWebSocketsAndPollingServiceBrokerage : BaseWebsocketsBrokerage -{ - // owns the service, forwards its events, disposes it with the brokerage - protected abstract IEnumerable ReadOrderStates(); // or a per-id read - protected virtual BrokerageOrderSnapshot ToSeedState(Order order) => null; -} -``` - -The attraction is real: a developer who picks the base class is told by the compiler what to implement, -instead of reading this document. Considered and not taken, on three facts. - -**The class cannot reach the plugins that need it.** A C# class inherits one class, so a layer under -`BaseWebsocketsBrokerage` serves only the plugins that already sit there — and most do not: - -| Plugin | Base class today | Order updates today | Could inherit it | -| --- | --- | --- | --- | -| CharlesSchwab | `BaseWebsocketsBrokerage` | account activity stream, this service as the fallback | yes | -| Tradier | `BaseWebsocketsBrokerage` | its own inline poll | yes | -| Binance (+ US and futures variants) | `BaseWebsocketsBrokerage` | user data stream | only through `BinanceBrokerage` — the variants subclass it | -| ByBit | `BaseWebsocketsBrokerage` | user data stream | yes, as a fallback only | -| dYdX | `BaseWebsocketsBrokerage` | subaccounts channel | yes, as a fallback only | -| Eze | `BaseWebsocketsBrokerage` | websocket protobuf push | yes, as a fallback only | -| Public.com | `Brokerage` | none — this service is the order path | **no** | -| InteractiveBrokers | `Brokerage`, `sealed` | vendor TCP callback SDK | **no** | -| TradeStation | `Brokerage` | HTTP stream, not a websocket | **no** | -| Alpaca | `Brokerage` | vendor SDK streaming client | **no** | -| Tastytrade | `Brokerage` | its own two-socket wrapper | no — a base class migration first | -| WeBull | `Brokerage` | its own websocket order events | no — a base class migration first | -| IG | `Brokerage` | Lightstreamer vendor SDK | no | -| OANDA | `Brokerage`, through its own `OandaRestApiBase` | HTTP transaction stream | no — its own hierarchy holds the slot | -| TerminalLink | `Brokerage` | Bloomberg session API | no | -| TradingTechnologies, Fix.Bloomberg, Fix.InteractiveBrokers | `Brokerage` / `FixBrokerage` | FIX execution reports | no | - -Only the first six rows sit on `BaseWebsocketsBrokerage`, and only two of those poll. The two plugins this -document opens with as the ones that need polling most — Public.com, with no order stream at all -(`PublicBrokerage.cs:42` extends plain `Brokerage`), and InteractiveBrokers, whose lost reply ends the run -(`InteractiveBrokersBrokerage.cs:68`, a `sealed` class on the vendor SDK) — are exactly the two the class -can never serve. - -**Serving both sides means writing the class twice.** The only way around the table is a second abstract -class with the same body under `Brokerage`, next to the websocket one. The bodies cannot be shared: a class -inherits one class, and a default interface method can neither hold the service field nor call the protected -`OnOrderEvents`. That is the same wiring twice in core, edited in pairs forever, and a fix applied to one -copy and not the other quietly makes the two halves behave differently. - -**The overrides are not where the adoption work is.** Counted on the two adopters: the wiring a base class -could absorb is about 30 lines in Schwab and about 45 in Public — the creation, three event forwards, one -dispose call. What it cannot absorb is everything else the plugin writes: the read and the mapping (~200 -lines in Schwab, ~45 in Public), the `Watch` calls inside the order methods, and the start trigger, which is -broker policy — Public starts polling in `Connect` because the service is its connection, Schwab starts it -when the stream is taken away. The class cannot even own the stop: `Disconnect` is abstract on `Brokerage` -(`Brokerages/Brokerage.cs:151`), so `Stop` stays a line the plugin writes either way. An abstract read would -guard the one step nobody gets wrong, and guard it for a third of the plugins. - -The place that reaches every row of the table is the root `Brokerage` class, the way the cross-zero -helpers and `CreateOAuthTokenHandler` already sit there as protected members only some plugins use -(`Brokerages/Brokerage.cs:702`, `:344`): one protected creation helper per mode, whose read-callback -signature picks the class, the service as a protected property, and a `Dispose` that covers it. Every -brokerage derives from `Brokerage`, the sealed IB class included, and constructing the service directly -stays possible — the helper is additive. Implemented that way on 2026-08-17: the seam is the two -`CreateOrderPollingService` overloads, `OrderPollingService`, `IsOrderPolling` and the virtual -`OnBrokerageOrderNeverNotified`, and Schwab, Public.com and Tradier all create their service through it -(see "Wiring, per plugin"). The abstract class this section declines stays declined. - -## Risks - -| Risk | What we do about it | -| --- | --- | -| A plugin maps a broker status to the wrong Lean status | The mapping is the same one its streaming path already needs, written once per plugin and covered by its own tests. The service only emits transitions, so a wrong mapping surfaces once, not as a flood. | -| A state without fill numbers cannot produce fill events | By design: null means unknown, and the service never invents a number. The watch still confirms submission, and the notification timeout still fires. | -| Several fills inside one sweep share one price | Quantities stay exact; the price is the broker's reported price at sweep time. Tradier kept this trade-off through its adoption (`TradierBrokerage.cs:1170-1172`); a shorter poll interval narrows it. | -| Polling adds requests on brokers with tight rate limits | Watch mode reads only while an order still waits for its first report, and the interval is a constructor argument the plugin picks. | -| A bulk read is expensive on some plugins — `GetOpenOrders()` rebuilds Lean orders and maps symbols on every call | The action converts straight from the broker's wire model to the snapshot, skipping the Lean `Order` build entirely; and in watch mode a sweep only runs while something is pending. | -| A plugin wires the handler wrong and gets fill-before-submit | The misuse is gone: the service takes the handler in its constructor and wires both directions itself. A plugin either hands over its handler, or passes null and the poll loop is the only caller of the diff. Schwab and Public have a handler; Tradier adopted with null — its submit event goes out before the watch begins, so the poll cannot outrun it — and IB's rollout step still names adding one. | -| The stream and the poll both see the same fill in watch mode | The registry works in both directions: the stream writes what it reports (`UpdateOrderState`) and checks before reporting (`TryGetLastOrderState`). Named as the one non-optional wiring rule for polling beside a live stream. | -| The notification timeout cannot tell "never arrived" from "filled instantly" | It does not try. `BrokerageOrderNeverNotified` hands the question to the brokerage, which has the endpoints to answer it. | -| Polling while the stream is down misses the fills that happened during the gap | Only when the read carries no fill data. A read with fill numbers recovers them — the state has the fields, so this is a property of the broker's endpoint, not of the service. | -| A plugin starts polling on disconnect and forgets to stop on reconnect | Both paths are one line and sit next to the connection handling the plugin already has. The poll side repeats nothing the registry already holds, so the cost of forgetting is extra requests, not extra events. | - -## Rollout - -1. This PR: the non-generic multi-source message handler, the service, the snapshot, their unit tests, - and the protected create seam on `Brokerage`. The generic handler is untouched, so every plugin - compiles as before, and no plugin is forced onto the seam. -2. InteractiveBrokers: add the message handler it does not have, then replace the `NoBrokerageResponse` error and - the invented `Submitted` with a watch. This is the proof that the abstraction holds for a plugin that did not - write it. -3. CharlesSchwab and Public.com (done): delete their service class and their fill/close diff. What stays is real - and named: the read and its sweep window, the model-to-state mapping, Schwab's stream-unavailable switch and - its by-symbol leg id assignment. One behavior change is intentional: a Public poll that shows a - new fill and the cancel together now emits both — the old code dropped the cancel. Schwab's per-execution - prices become per-sweep prices while the quantities stay exact; Public kept its change-of-average price - recovery inside its mapping, so its part prices stay exact too. -4. Tradier (done, further than planned here): the step was a watch for submissions with fills staying on its own - path. The adoption moved the fill path itself onto `SingleOrderPollingService` and handles the cross-zero split - as "A cross-zero order, two ids" describes. Two costs are accepted: a sweep is one gated request per watched - order instead of one bulk request — the one-order-per-symbol rule keeps that count small, and an idle account - now polls nothing at all — and orders placed outside Lean are ignored, where the old code raised a fatal - "UnknownOrderId" error.