Architecture

Pipeline overview#

Stream-Ingest → Feature Validators → Persistence
                                       ↓
                                  Event Bus (PG LISTEN/NOTIFY)
                                       ↓
                            ┌─────────────────┐
                            │ Decision Daemon │ (per-tenant config)
                            └────────┬────────┘
                                     ↓
                              Validated Action
                                     ↓
                              Outcome Feedback
                                     ↓
                            Model Training / Tuning

The platform turns raw data streams into convergent decisions through a deliberately staged pipeline. Each stage has a single responsibility and emits durable events for the next.

Event-driven core#

• Event bus. PostgreSQL LISTEN/NOTIFY carries events between daemons. Events are persisted to a durable table, not just broadcast — consumers can replay from a cursor after a reconnect.
• At-least-once delivery. Consumers acknowledge processed events. Unacknowledged events are re-delivered after timeout. Idempotency is the consumer’s responsibility.
• Cursor-tracked replay. Every consumer maintains its own cursor. After a disconnect, it resumes from where it stopped — no events are lost, no events are double-processed beyond the configured retry semantics.

ML pipeline#

• Models. XGBoost trained against historical features, with ONNX Runtime used for inference (2–5× faster than the training framework).
• Outputs. Models classify regimes across three dimensions — direction, volatility, quality. No speculative point forecasts.
• Hyperparameter tuning. Optuna with walk-forward holdout, champion/challenger waves, quality gates.
• Feedback loop. Each decision’s outcome is recorded and fed back into the next tuning round.

Multi-tenant model#

Tenant isolation is enforced at two levels:

• Stored procedures. All tenant-facing operations go through stored procedures that enforce ownership. Direct table writes by tenant code are not permitted.
• Row-level security. Postgres RLS as the hard backstop. Even if application logic has a bug, RLS prevents cross-tenant data access.

Per-tenant configuration covers:

• Keys and rate limits.
• Kill-switch. Operator override to disable a tenant immediately without redeployment.
• Risk profile. Threshold parameters per decision class.

Validators#

Three places where validation happens:

1. Before persistence. Stream events that fail validation are marked, not silently dropped. Operators see them in the dashboard.
2. Before model training. Features used for training are re-validated against the schema. Drift triggers an alert before models are retrained.
3. At runtime. Feature validators run continuously with DATAMISSING auto-recovery — when an expected feature is absent, the daemon falls back to a documented default and raises a structured event.

Observability#

• 21 live dashboards. Heartbeat, latency, drift detection, decision distribution, per-tenant volume.
• Structured logging. Console, file, and database — same log events readable from all three with the same query.
• Health monitoring. Each daemon emits health events at a fixed cadence. Missing events trigger pages.

Production-readiness mechanisms#

• Circuit breakers per data source — failures degrade gracefully rather than cascading.
• Email and XMPP alerting for operator-visible events.
• Universal config-driven defaults. Data source mappings, fallback chains, unit conversions, thresholds — all in JSON. No magic numbers in code.
• Process isolation via tmux + Devuan — daemons can be restarted individually without coordinating.

What this is not#

• Not a Kubernetes-style autoscaling platform. This runs on a small number of well-understood machines with deliberate daemon placement.
• Not “set up and forget.” Token sets, model versions, tenant configs, and dashboards require operator attention.
• Not generic. The platform reflects a specific class of high-frequency decision problems — adapting it to a very different workload is the Development service.