# ArchitectureEvent-driven, multi-tenant, ML-augmented decision platform — how it is structured.

## Pipeline overview

```
Stream-Ingest → Feature Validators → Persistence
                                       ↓
                                  Event Bus (PG LISTEN/NOTIFY)
                                       ↓
                            ┌─────────────────┐
                            │ Decision Daemon │ (per-tenant config)
                            └────────┬────────┘
                                     ↓
                              Validated Action
                                     ↓
                              Outcome Feedback
                                     ↓
                            Model Training / Tuning
```

The platform turns raw data streams into convergent decisions
through a deliberately staged pipeline. Each stage has a single
responsibility and emits durable events for the next.

## Event-driven core

- **Event bus.** PostgreSQL `LISTEN/NOTIFY` carries events
  between daemons. Events are persisted to a durable table, not
  just broadcast — consumers can replay from a cursor after a
  reconnect.
- **At-least-once delivery.** Consumers acknowledge processed
  events. Unacknowledged events are re-delivered after timeout.
  Idempotency is the consumer's responsibility.
- **Cursor-tracked replay.** Every consumer maintains its own
  cursor. After a disconnect, it resumes from where it stopped —
  no events are lost, no events are double-processed beyond the
  configured retry semantics.

## ML pipeline

- **Models.** XGBoost trained against historical features, with
  ONNX Runtime used for inference (2–5× faster than the training
  framework).
- **Outputs.** Models classify regimes across three dimensions —
  direction, volatility, quality. No speculative point forecasts.
- **Hyperparameter tuning.** Optuna with walk-forward holdout,
  champion/challenger waves, quality gates.
- **Feedback loop.** Each decision's outcome is recorded and
  fed back into the next tuning round.

## Multi-tenant model

Tenant isolation is enforced at two levels:

- **Stored procedures.** All tenant-facing operations go through
  stored procedures that enforce ownership. Direct table writes
  by tenant code are not permitted.
- **Row-level security.** Postgres RLS as the hard backstop. Even
  if application logic has a bug, RLS prevents cross-tenant data
  access.

Per-tenant configuration covers:

- **Keys and rate limits.**
- **Kill-switch.** Operator override to disable a tenant
  immediately without redeployment.
- **Risk profile.** Threshold parameters per decision class.

## Validators

Three places where validation happens:

1. **Before persistence.** Stream events that fail validation are
   marked, not silently dropped. Operators see them in the
   dashboard.
2. **Before model training.** Features used for training are
   re-validated against the schema. Drift triggers an alert
   before models are retrained.
3. **At runtime.** Feature validators run continuously with
   `DATAMISSING` auto-recovery — when an expected feature is
   absent, the daemon falls back to a documented default and
   raises a structured event.

## Observability

- **21 live dashboards.** Heartbeat, latency, drift detection,
  decision distribution, per-tenant volume.
- **Structured logging.** Console, file, and database — same log
  events readable from all three with the same query.
- **Health monitoring.** Each daemon emits health events at a
  fixed cadence. Missing events trigger pages.

## Production-readiness mechanisms

- **Circuit breakers** per data source — failures degrade
  gracefully rather than cascading.
- **Email and XMPP alerting** for operator-visible events.
- **Universal config-driven defaults.** Data source mappings,
  fallback chains, unit conversions, thresholds — all in JSON.
  No magic numbers in code.
- **Process isolation** via tmux + Devuan — daemons can be
  restarted individually without coordinating.

## What this is not

- Not a Kubernetes-style autoscaling platform. This runs on a
  small number of well-understood machines with deliberate
  daemon placement.
- Not "set up and forget." Token sets, model versions, tenant
  configs, and dashboards require operator attention.
- Not generic. The platform reflects a specific class of
  high-frequency decision problems — adapting it to a very
  different workload is the [Development](/services/development/)
  service.