Skip to main content
Operational infrastructure that runs alongside the oversight loop. These subsystems handle notifications, accounting, strategy promotion, and automatic recovery from failures.

Alerts and Notifications

Rate-limited alert system with multi-channel delivery. Alerts fire during the oversight loop for risk events, kill switch activations, strategy failures, and settlements.

Alert Levels and Categories

LevelUse
INFORoutine events (settlement detected, strategy promoted)
WARNINGApproaching limits (80% drawdown, strategy paused)
CRITICALImmediate action needed (kill switch, multiple failures)
CategoryTriggered By
KILL_SWITCHFund drawdown exceeds limit
STRATEGY_FAILUREStrategy thread crashes
DRAWDOWNFund or strategy drawdown approaching limit
SETTLEMENTMarket resolved, P&L booked
CORRELATIONCross-strategy correlation exceeds threshold
RISK_LIMITVaR budget breach
RECOVERYSelf-healing restart attempt
PROMOTIONStrategy promoted to next stage
REBALANCECapital reallocation triggered
GENERALEverything else

Rate Limiting

Each category is limited to 5 alerts per 60 seconds. Excess alerts are silently dropped to prevent notification floods during volatile periods.

Channels

from horizon.fund import FundConfig

# Webhook channel -- delivers alerts via HTTP POST
fund = FundManager(FundConfig(
    total_capital=100_000,
    alert_webhook_url="https://hooks.slack.com/services/...",
))

# The webhook sends JSON:
# {
#     "level": "CRITICAL",
#     "category": "KILL_SWITCH",
#     "title": "Kill switch activated",
#     "message": "Fund drawdown 15.2% exceeded limit 15.0%",
#     "timestamp": "2026-03-13T14:30:00.000000"
# }
Webhook delivery runs in a background thread to avoid blocking the oversight loop. Failed deliveries are logged but do not retry.

MCP Tool

fund_alerts(limit=50)
Returns recent alerts with counts by category. Use this to check what happened while the LLM was idle.

Double-Entry Ledger

SQLite-backed accounting ledger that records every financial event as a debit/credit journal entry. Provides balance sheet and income statement views.

Accounts

AccountTypeTracks
CASHAssetAvailable capital
POSITIONSAssetCapital deployed in positions
FEES_RECEIVABLEAssetFees owed (internal)
MANAGEMENT_FEESRevenueAnnual management fee accruals
PERFORMANCE_FEESRevenuePerformance fee accruals
REALIZED_PNLRevenueBooked profit and loss
UNREALIZED_PNLRevenueMark-to-market P&L

Journal Entries

Every financial event creates a balanced journal entry:
EventDebitCreditAmount
DepositCASH(external)Deposit amount
Withdrawal(external)CASHWithdrawal amount
Profitable tradeCASHREALIZED_PNLProfit
Losing tradeREALIZED_PNLCASHLoss
Management feeMANAGEMENT_FEESFEES_RECEIVABLEFee amount
Performance feePERFORMANCE_FEESFEES_RECEIVABLEFee amount

Configuration

fund = FundManager(FundConfig(
    total_capital=100_000,
    ledger_enabled=True,  # Enable double-entry ledger
))
When enabled, the oversight loop automatically records:
  • Settlement P&L when markets resolve
  • Management and performance fee accruals each tick
  • All amounts validated (must be finite, positive)

MCP Tool

fund_ledger(limit=50)
Returns recent journal entries, balance sheet (assets vs. liabilities), and income statement.

Reports

# Balance sheet
bs = fund.ledger.balance_sheet()
# {"CASH": 95000.0, "POSITIONS": 5000.0, "MANAGEMENT_FEES": 200.0, ...}

# Income statement
inc = fund.ledger.income_statement()
# {"REALIZED_PNL": 1500.0, "MANAGEMENT_FEES": -200.0, "PERFORMANCE_FEES": -100.0}

# Recent entries
entries = fund.ledger.recent_entries(limit=20)

Paper-to-Live Promotion

Structured lifecycle that gates strategy promotion behind performance criteria. Strategies must prove themselves in paper trading before progressing to shadow mode and then live trading.

Promotion Stages

PAPER --> SHADOW --> LIVE
StageDescription
PAPERSimulated execution, no real capital
SHADOWRuns alongside live but orders not submitted
LIVEReal execution with real capital

Promotion Criteria

Each transition requires meeting minimum thresholds:
CriteriaDefaultDescription
min_paper_days14Minimum days in current stage
min_trades50Minimum number of trades
min_sharpe1.0Minimum Sharpe ratio
max_drawdown_pct20.0Maximum drawdown percentage
min_win_rate0.45Minimum win rate

Configuration

fund = FundManager(FundConfig(
    total_capital=100_000,
    promotion_enabled=True,  # Enable promotion lifecycle
))
When enabled, the oversight loop checks promotion eligibility every 60 ticks. Promotions are recorded with a full audit trail in SQLite (strategy name, from/to stage, timestamp, metrics at promotion time).

MCP Tool

fund_promotion_status()
Returns current promotion stage for each strategy and promotion history.

Persistence

Two SQLite tables track state:
  • promotions: current stage per strategy
  • promotion_history: every promotion event with timestamp and reason
Data is stored in ~/.horizon/promotion.db with WAL mode and 0o600 permissions.

Self-Healing Recovery

Automatic strategy restart with exponential backoff and circuit breaker. Detects failed strategies and stale feeds, attempts recovery without human intervention.

How It Works

  1. The oversight loop calls recovery.check_strategies() every 10 ticks
  2. For each failed strategy, it attempts a restart via controller.restart()
  3. If the restart fails, backoff doubles: 60s, 120s, 240s…
  4. After 3 failed attempts, the circuit breaker opens and the strategy is marked permanently failed

Circuit Breaker

AttemptCooldownAction
160sRestart strategy
2120sRestart strategy
3240sRestart strategy
4+Circuit breaker open, no more retries
To re-enable recovery for a permanently failed strategy:
fund.recovery.reset("strategy_name")

Feed Monitoring

The recovery manager also detects stale feeds (no update for 300+ seconds) and logs warnings. Feed staleness is informational; automatic feed reconnection is handled by the feed manager.

Configuration

fund = FundManager(FundConfig(
    total_capital=100_000,
    self_healing_enabled=True,  # Enable self-healing
))

Recovery Events

Every recovery attempt is logged:
events = fund.recovery.event_history(limit=20)
# [
#     RecoveryEvent(
#         action="restart_strategy",
#         target="political_mm",
#         success=True,
#         attempt=1,
#         error=None,
#     ),
# ]

stats = fund.recovery.stats()
# {"total_attempts": 5, "successes": 3, "failures": 2, "circuit_breakers_open": 1}

Configuration Summary

ParameterDefaultDescription
alert_webhook_urlNoneWebhook URL for alert delivery
ledger_enabledFalseEnable double-entry accounting
promotion_enabledFalseEnable paper-to-live promotion
self_healing_enabledFalseEnable automatic recovery
All four subsystems are off by default. Enable them individually in FundConfig. They integrate with the oversight loop automatically when enabled.