Scaling the research loop: durable orchestration with Temporal.io and Optuna

Distributed Systems

Temporal.io / asyncio

Multi-Objective Optimization

Data Engineering

A single-process research framework has a hard ceiling: one machine, one failure away from losing a multi-day sweep, and no way to put the CPU-hungry part somewhere different from the network-hungry part.

So I rebuilt the research loop as a distributed, durable pipeline: Temporal.io for orchestration, Optuna for multi-objective parameter search, the Hummingbot v2 engine as the backtest core, InfluxDB 3 as the candle cache, and Ansible to put all of it onto a VPS from bare metal.

~16 100 lines across 195 files — Python workers, TypeScript/Electron analytics UI, Ansible roles and Jinja templates — deployed and running on a live host.

Architecture: nodes that don’t know about each other

The system is split into autonomous nodes, each its own Python package with its own pyproject.toml, its own virtualenv and its own dependency set. They communicate only through Temporal task queues.

Node	Hosted workflow	Dependencies	Responsibility
`main_orchestrator`	`CoinSelectWorkflow`	`temporalio` + shared core lib — nothing else	Top of the sweep: walks the exchange’s pair list and launches windows in batches of four behind an `asyncio.Semaphore`
`optimization_heavy_engine`	`WalkForwards`	Optuna, Hummingbot v2, pandas	CPU-bound: slices history into IS/OOS windows, optimizes in-sample, retests the best trials out-of-sample
`klines_downloader_VPS_worker`	`DataBackfillService`	aiohttp, ccxt, influxdb3	Network-bound and strategy-agnostic: detects gaps in the candle cache and downloads only what is missing

Each row starts the one below it as a child workflow, so a full sweep is a single durable tree — and each row is a separate process that can live on a separate machine.

The key decision: activities are invoked by string name with an explicit task queue, never by importing the function. The consequence is that the orchestration node’s dependency list is two entries long — it does not need Optuna, pandas or the backtest engine to schedule work that uses them. The nodes can be moved to separate machines by editing an inventory file, not code.

The downloader is deliberately kept ignorant of strategies. Indicator warm-up is computed on the strategy-aware side by a dedicated activity, and the downloader simply receives an already left-extended range. Adding a new strategy means adding one module with a config generator, a constraints function and a warm-up declaration — picked up through importlib. The engine and the orchestration never change.

Durability where it actually matters

A parameter sweep is a long-running, expensive, failure-prone job. Temporal is the right tool, but only if it is configured with intent:

Retry policies with exponential backoff (1 s initial, ×2, 30 s ceiling, 5 attempts) on network-facing activities — transient exchange errors resolve themselves and shouldn’t kill a window.
Multi-hour timeouts with heartbeats. start_to_close_timeout = 2h paired with heartbeat_timeout = 2h, and the optimization activity heartbeats progress on every trial — so a stuck worker is distinguishable from a slow one in the Temporal UI, which is the whole point.
Deliberate backpressure. max_concurrent_activities=1 on the heavy worker, workflow cache disabled on the downloader, a semaphore capping parallel child workflows, and N worker processes spawned per CPU core by a set -euo pipefail bash launcher that skips already-running PIDs.
Deterministic child workflow IDs derived from the parent’s ID, with explicit ID-reuse policies (ALLOW_DUPLICATE for the backfill service, TERMINATE_IF_RUNNING for re-launched windows).

Multi-objective optimization

Trading strategy selection is genuinely multi-objective — profit, drawdown and risk-adjusted return trade off against each other and collapsing them into one scalar throws away the information you need to choose. The study optimizes three objectives simultaneously:

maximize   net_pnl_quote
maximize   max_drawdown_pct
maximize   sharpe_ratio

and reports the Pareto front of non-dominated trials rather than a single “best” configuration.

The sampler moved from NSGA-III to a multivariate TPE with a warm-up of random startup trials.

Anti-overfitting as explicit constraints

Optuna’s constrained domination carries the guardrails, with each constraint normalised against its threshold:

Constraint	Purpose
minimum 40 trades	statistical significance — no conclusions from a handful of fills
maximum 25 % drawdown	a return path that is actually survivable
minimum profit factor 1.1	a real edge, not one rounding error above break-even

The sampler seed and the objective list are stored as study user_attrs, so any run can be reproduced and audited after the fact.

Warm-up decoupling

A subtle but decisive methodological point: trials use different indicator lengths, so a naive setup evaluates a 50-period EMA over a shorter effective window than a 10-period one — and then compares their metrics as if they were equivalent. Here the maximum warm-up across the entire search space is computed up front and pre-loaded, so every trial is scored over an identical evaluation window.

Data engineering

Incremental, gap-driven ingestion. Instead of re-downloading ranges, the pipeline finds what is actually missing with a SQL window function:

WITH d AS (
  SELECT time, LAG(time) OVER (ORDER BY time) AS prev_time
  FROM klines WHERE pair = ? AND timeframe = ?
)
SELECT prev_time, time FROM d
WHERE time - prev_time > INTERVAL '<timeframe>'

plus a separate MIN/MAX query for the edge gaps before the earliest and after the latest stored candle. Only those gaps are fetched.

Non-blocking writes. The download activity is a producer–consumer over asyncio.Queue with a dedicated writer task in a ThreadPoolExecutor and a sentinel for shutdown, so database writes never stall the event loop that is pulling from the exchange. The same pattern moves blocking study.ask / study.tell calls off the loop.

Columnar all the way down. InfluxDB 3 is queried over Arrow Flight SQL, and timestamp normalisation happens in pyarrow.compute without ever materialising through pandas. A Feather backend implements the identical query signature, so the storage backend is a parameter threaded through the workflows rather than a fork in the code.

Optuna storage on the right side of the hot path. Trials run against in-memory storage; results are flushed to Postgres once at the end via a batched ON CONFLICT DO UPDATE upsert (5 000 rows per batch, exponential backoff on operational errors). Writing every trial straight to a remote study was measured, rejected, and the reasoning left in a comment.

Infrastructure as code

The Ansible layer provisions a bare VPS from nothing: bootstrapping Python through the raw module on a host that doesn’t have it yet, then base setup, git checkout, Postgres, Temporal CLI, uv, the Hummingbot environment and per-node Python environments — 7 plays across 8 roles.

The convention I’m most pleased with: The roles discover nodes by finding pyproject.toml files and intersecting the result with group_names, so the list of nodes exists in exactly one place. Spreading the system across more VPSes is a change to inventory.ini and nothing else.

Two assert invariants — “this node has no group in the inventory” and “this host has no node assigned” — turn the two most likely misconfigurations into loud failures instead of silent no-ops. Secrets are rendered from Ansible Vault into Jinja templates with | mandatory, mode 0600 and no_log: true; the git token is passed via http.extraHeader so it never settles into .git/config.

Analytics interface

Pareto fronts are not readable as tables. The desktop UI (Electron + React 18 + TypeScript) renders the front as a 3D scatter across the three backtest objectives via SciChart, alongside a 2D robustness chart pairing each in-sample study with its out-of-sample counterpart. Studies are loaded from Postgres in batches with an in-memory cache, concurrency limiting and cancellation — with the front computation itself implemented locally rather than taken on faith from the storage layer.