Skip to main content
Operational infrastructure that runs alongside the oversight loop. These subsystems handle notifications, accounting, strategy promotion, and automatic recovery from failures.

Alerts and Notifications

Rate-limited alert system with multi-channel delivery. Alerts fire during the oversight loop for risk events, kill switch activations, strategy failures, and settlements.

Alert Levels and Categories

Rate Limiting

Each category is limited to 5 alerts per 60 seconds. Excess alerts are silently dropped to prevent notification floods during volatile periods.

Channels

Webhook delivery runs in a background thread to avoid blocking the oversight loop. Failed deliveries are logged but do not retry.

MCP Tool

Returns recent alerts with counts by category. Use this to check what happened while the LLM was idle.

Double-Entry Ledger

SQLite-backed accounting ledger that records every financial event as a debit/credit journal entry. Provides balance sheet and income statement views.

Accounts

Journal Entries

Every financial event creates a balanced journal entry:

Configuration

When enabled, the oversight loop automatically records:
  • Settlement P&L when markets resolve
  • Management and performance fee accruals each tick
  • All amounts validated (must be finite, positive)

MCP Tool

Returns recent journal entries, balance sheet (assets vs. liabilities), and income statement.

Reports

Paper-to-Live Promotion

Structured lifecycle that gates strategy promotion behind performance criteria. Strategies must prove themselves in paper trading before progressing to shadow mode and then live trading.

Promotion Stages

Promotion Criteria

Each transition requires meeting minimum thresholds:

Configuration

When enabled, the oversight loop checks promotion eligibility every 60 ticks. Promotions are recorded with a full audit trail in SQLite (strategy name, from/to stage, timestamp, metrics at promotion time).

MCP Tool

Returns current promotion stage for each strategy and promotion history.

Persistence

Two SQLite tables track state:
  • promotions: current stage per strategy
  • promotion_history: every promotion event with timestamp and reason
Data is stored in ~/.horizon/promotion.db with WAL mode and 0o600 permissions.

Self-Healing Recovery

Automatic strategy restart with exponential backoff and circuit breaker. Detects failed strategies and stale feeds, attempts recovery without human intervention.

How It Works

  1. The oversight loop calls recovery.check_strategies() every 10 ticks
  2. For each failed strategy, it attempts a restart via controller.restart()
  3. If the restart fails, backoff doubles: 60s, 120s, 240s…
  4. After 3 failed attempts, the circuit breaker opens and the strategy is marked permanently failed

Circuit Breaker

To re-enable recovery for a permanently failed strategy:

Feed Monitoring

The recovery manager also detects stale feeds (no update for 300+ seconds) and logs warnings. Feed staleness is informational; automatic feed reconnection is handled by the feed manager.

Configuration

Recovery Events

Every recovery attempt is logged:

Configuration Summary

All four subsystems are off by default. Enable them individually in FundConfig. They integrate with the oversight loop automatically when enabled.