Alerts and Notifications
Rate-limited alert system with multi-channel delivery. Alerts fire during the oversight loop for risk events, kill switch activations, strategy failures, and settlements.Alert Levels and Categories
| Level | Use |
|---|---|
INFO | Routine events (settlement detected, strategy promoted) |
WARNING | Approaching limits (80% drawdown, strategy paused) |
CRITICAL | Immediate action needed (kill switch, multiple failures) |
| Category | Triggered By |
|---|---|
KILL_SWITCH | Fund drawdown exceeds limit |
STRATEGY_FAILURE | Strategy thread crashes |
DRAWDOWN | Fund or strategy drawdown approaching limit |
SETTLEMENT | Market resolved, P&L booked |
CORRELATION | Cross-strategy correlation exceeds threshold |
RISK_LIMIT | VaR budget breach |
RECOVERY | Self-healing restart attempt |
PROMOTION | Strategy promoted to next stage |
REBALANCE | Capital reallocation triggered |
GENERAL | Everything else |
Rate Limiting
Each category is limited to 5 alerts per 60 seconds. Excess alerts are silently dropped to prevent notification floods during volatile periods.Channels
MCP Tool
Double-Entry Ledger
SQLite-backed accounting ledger that records every financial event as a debit/credit journal entry. Provides balance sheet and income statement views.Accounts
| Account | Type | Tracks |
|---|---|---|
CASH | Asset | Available capital |
POSITIONS | Asset | Capital deployed in positions |
FEES_RECEIVABLE | Asset | Fees owed (internal) |
MANAGEMENT_FEES | Revenue | Annual management fee accruals |
PERFORMANCE_FEES | Revenue | Performance fee accruals |
REALIZED_PNL | Revenue | Booked profit and loss |
UNREALIZED_PNL | Revenue | Mark-to-market P&L |
Journal Entries
Every financial event creates a balanced journal entry:| Event | Debit | Credit | Amount |
|---|---|---|---|
| Deposit | CASH | (external) | Deposit amount |
| Withdrawal | (external) | CASH | Withdrawal amount |
| Profitable trade | CASH | REALIZED_PNL | Profit |
| Losing trade | REALIZED_PNL | CASH | Loss |
| Management fee | MANAGEMENT_FEES | FEES_RECEIVABLE | Fee amount |
| Performance fee | PERFORMANCE_FEES | FEES_RECEIVABLE | Fee amount |
Configuration
- Settlement P&L when markets resolve
- Management and performance fee accruals each tick
- All amounts validated (must be finite, positive)
MCP Tool
Reports
Paper-to-Live Promotion
Structured lifecycle that gates strategy promotion behind performance criteria. Strategies must prove themselves in paper trading before progressing to shadow mode and then live trading.Promotion Stages
| Stage | Description |
|---|---|
| PAPER | Simulated execution, no real capital |
| SHADOW | Runs alongside live but orders not submitted |
| LIVE | Real execution with real capital |
Promotion Criteria
Each transition requires meeting minimum thresholds:| Criteria | Default | Description |
|---|---|---|
min_paper_days | 14 | Minimum days in current stage |
min_trades | 50 | Minimum number of trades |
min_sharpe | 1.0 | Minimum Sharpe ratio |
max_drawdown_pct | 20.0 | Maximum drawdown percentage |
min_win_rate | 0.45 | Minimum win rate |
Configuration
MCP Tool
Persistence
Two SQLite tables track state:promotions: current stage per strategypromotion_history: every promotion event with timestamp and reason
~/.horizon/promotion.db with WAL mode and 0o600 permissions.
Self-Healing Recovery
Automatic strategy restart with exponential backoff and circuit breaker. Detects failed strategies and stale feeds, attempts recovery without human intervention.How It Works
- The oversight loop calls
recovery.check_strategies()every 10 ticks - For each failed strategy, it attempts a restart via
controller.restart() - If the restart fails, backoff doubles: 60s, 120s, 240s…
- After 3 failed attempts, the circuit breaker opens and the strategy is marked permanently failed
Circuit Breaker
| Attempt | Cooldown | Action |
|---|---|---|
| 1 | 60s | Restart strategy |
| 2 | 120s | Restart strategy |
| 3 | 240s | Restart strategy |
| 4+ | — | Circuit breaker open, no more retries |
Feed Monitoring
The recovery manager also detects stale feeds (no update for 300+ seconds) and logs warnings. Feed staleness is informational; automatic feed reconnection is handled by the feed manager.Configuration
Recovery Events
Every recovery attempt is logged:Configuration Summary
| Parameter | Default | Description |
|---|---|---|
alert_webhook_url | None | Webhook URL for alert delivery |
ledger_enabled | False | Enable double-entry accounting |
promotion_enabled | False | Enable paper-to-live promotion |
self_healing_enabled | False | Enable automatic recovery |
FundConfig. They integrate with the oversight loop automatically when enabled.