Distributed Execution with Microsoft Orleans

The actor model treats concurrency as a mailbox problem, not a lock problem. Each unit of work is an isolated actor: it owns its state, processes one message at a time, and talks to peers only by sending messages. No shared mutable heap between actors. Races shrink because there is a single logical owner for each piece of live state.

Orleans takes that philosophy into a distributed runtime with virtual actors, called grains. You address a grain by identity, not by host. The cluster activates it when needed, places it on a silo, deactivates it under pressure, and rehydrates it later. You write as if the actor always exists; Orleans owns the lifecycle.

That idea was battle-tested at Microsoft scale. Orleans powered cloud services for the Halo franchise (presence, game sessions, player state), where millions of independent actors had to stay addressable without the app managing server topology by hand. Managing uses the same ownership trick at a much smaller scale: one live bot, one grain, one decision path.

Ownership has to be explicit. If “whoever hit the API last” owns the bot, you will spend forever debugging races. On Managing the answer is concrete: one live-bot grain per bot GUID. The API and SignalR edge only resolve that grain by id. They do not own the live decision path.

Persistence is a snapshot model, not event sourcing and not a SAGA log. Live bot grains write persistent state into PostgreSQL via ADO.NET grain storage. Supporting grains (registry, candles, feed) use Redis when available, otherwise the same Postgres path. Cluster membership and reminders also sit on ADO.NET with Npgsql. On activate, Orleans reloads the last snapshot; if that snapshot is empty after a wipe, the grain hydrates from the application database before resuming. The strategy cycle sits in a composed trading engine. Peers stay outside the boundary. Exchange calls leave through the adapter behind that engine, not through controllers.

Activation and placement stay with Orleans. Accidental concurrency drops because there is a single logical owner for each live bot. The ownership rule is simple: one GUID, one grain, one decision path.

Heavy work must not starve a live bot, so callers never pick a host. They ask Orleans for the grain by key, and custom placement directors decide where it runs.

The Trading director filters silos whose names contain -Trading, then uses a power-of-two pick: sample up to two candidates and choose one at random. That softens hot spots by spreading activations instead of always hitting the same silo. If no Trading silo matches, it falls back to the same two-choice pick across every compatible silo rather than failing closed. The Compute director prefers -Compute names with a random pick among matches, otherwise random among all compatible. There is no hard “Trading is full” reject: when the preferred role is missing, fallback keeps activation possible and load spreads by chance across compatible hosts.

Three Orleans primitives carry different failure modes in Managing. The interesting part is not the textbook definition. It is what breaks when you put money behind them.

Turn-based concurrency is a feature until an exchange call blocks the turn. Managing keeps venue I/O asynchronous: grain methods await HTTP/SDK calls so the scheduler can run other grains while a Binance or Bybit request is in flight. A sync-over-async wrapper around that path would stall the grain and hurt neighbors on the same silo. That discipline is product code, not something Orleans enforces.

The live-bot grain is not marked reentrant. Default Orleans queues one turn at a time. If SignalR asks for bot state while a timer cycle is mid-flight, the request waits for the turn to finish instead of interleaving into half-updated positions. Safer for trading state; slower for chatty UI. That tradeoff is intentional.

GC jitter is real on .NET. Managing is not a microsecond HFT stack; it is medium-frequency execution where a short pause is annoying, not always fatal. Still, Server GC, allocation discipline, and keeping grain turns short matter more than Orleans slogans. I treat latency budgets as an ops concern alongside silo sizing, not as solved by virtual actors.

Orleans removes a class of concurrency bugs. It does not invent idempotency, durable state, retries, backpressure, or exchange-specific recovery. Those policies are product requirements: partial fills, stale websockets, and rate limits must be inspectable states, not “retry until it works.”

Crash recovery made ownership concrete. A Trading silo died mid-cycle while an exchange order was still in flight. The grain went with the host. The reboot reminder later woke the same bot GUID on another silo, restored the PostgreSQL snapshot, and resumed. Availability came back.

The nastier case is a network timeout after the venue already accepted the order. The next timer tick must not open a second position. Managing’s guard is reconcile-first: before a new entry, the cycle inspects open broker positions and open orders for that account and ticker, and skips a duplicate intent when exchange truth already shows the fill or working order. A deterministic client order id derived from bot GUID plus signal id makes a venue-level retry a no-op. Orleans will never invent that key for you.

Passing unit tests alone do not prove bots survive a dead host. Silo death is exercised by killing a Trading process under paper or staging load and watching the reboot reminder bring bots back on another host. Timeout-after-fill is reviewed as a path test: inject a hung or dropped response after a successful venue accept, then assert the next cycle reconciles instead of doubling size. Duplicate prevention is checked the same way: open-order and open-position fixtures before a new entry. Those drills catch the failure modes this article claims.

In production today the whole Managing stack runs on one small KVM2 server: API, proxies, databases, and the Orleans runtime (about 20% CPU and 6 GB RAM on average). Five users, more than fifty grain ups without failure, ten active reminders, and about 9,977 hours of cumulative bot runtime (roughly 416 days). That is not Halo scale. It is proof that virtual actors buy isolation and recovery even on a tiny footprint.

Isolation is what keeps scale from becoming shared fate. A bot stuck in a sync-over-async exchange call or a long GC pause only blocks its own grain turn; other grains on other silos keep running. A non-reentrant live-bot grain means a slow cycle does not interleave with SignalR state reads or another timer tick for that same bot. Trading versus Compute placement keeps backtests off trading silos, so heavy compute cannot jitter live exchange cycles. Paper, copy, and live modes can share the same trading-engine shape, but they use separate grain IDs, storage, and adapters, so a bad live strategy does not corrupt paper state. Operationally that lets me quarantine, throttle, or restart one bot without a global lock or a shared queue for every tenant.

Exchanges change and venue quirks multiply. The durable move was keeping ownership, policy, and adapters separable, not chasing a fashionable runtime. That boundary comes from domain-driven design already in place: execution owns order intent, position lifecycle, risk, and reconciliation; market data owns candles and feeds; venue integrations are an anti-corruption layer that translates exchange-specific symbols, statuses, and failures into domain contracts. Adding a venue means a new adapter behind the trading engine; ownership and reconcile policy stay put. Adding an execution mode means extending an edge service, reusing the grain model, and leaving the domain brain intact. Evolving live state means versioning persistent-state schemas, config, and grain interfaces carefully so a deploy does not brick bots mid-cycle. .NET Aspire runs the local topology (silos, placement, reminders, storage wiring) in a shape close to production, so those seams show up on a laptop instead of only after ops. Orleans holds ownership. Domain policy holds business rules. Adapters hold venue specifics. That is what lets the system outlive any one framework or venue.

Orleans solved Managing’s ownership and orchestration problem: one grain per live bot, placement across silos, timers for the hot path, and reminders that wake bots after a host dies. That is the runtime win.

I am especially satisfied with how much the Orleans team provides out of the box: virtual-actor lifecycle, placement, serialized turns, timers, and durable reminders fit together as one coherent runtime. That lets the work stay focused on trading policy and product safeguards instead of rebuilding distributed-runtime plumbing.

Reliability still depends entirely on product guardrails: snapshot persistence you trust, async venue I/O, non-reentrant trading turns, reconcile-before-retry, and deterministic order identity where the exchange supports it. Virtual actors hold the mailbox. You still own the money path.