Data Sources

Seventeen external feeds drive the platform. Each is registered as a DataSource subclass in app/sources/ and pulled by engine.py:fetch_all_sources() on the cadence dictated by the active collection profile.

The full inventory

Three auxiliary fetches run alongside the registered sources and stash results under underscore-prefixed keys (_gamma_levels, _fed_watch, _correlations) so format_report can find them without colliding with per-source dicts:

• Gamma profile — Schwab option-chain pull for the per-strike gamma_levels API path. Requires Schwab auth; logs and continues on token failure.
• Fed Watch — CME rate-cut probability surface for the next FOMC meeting.
• Correlations — rolling SPY/VIX, SPY/DXY, SPY/TNX, SPY/Oil pair correlations computed post-hoc from the persisted daily_signals history. Not an external fetch — it’s a SQL query against our own history.

Provider mix and fallback posture

• market tries Schwab first, falls back to yfinance when Schwab returns None/empty. GEX/ZGL/PCR require Schwab option chains and have no yfinance fallback. VIX uses volume-weighting since Schwab reports no OI for index options. TNX and DXY have no option chains.
• Schwab is optional. Missing or expired tokens log CRITICAL but the platform continues on yfinance / FRED / other sources. See source health for how token status surfaces on /api/v1/status.
• Alpha Vantage is rate-limited at 25 requests/day; /api/v1/source-health surfaces remaining budget.
• News fan-out — the news source merges three feeds: FXStreet financial headlines, Iran International liveblog (for geopolitical risk), and the Hormuz Strait tanker monitor (for oil-supply tail risk).
• ebp pulls the Federal Reserve’s monthly FEDS Notes Excess Bond Premium CSV. EBP is the residual of the Gilchrist-Zakrajšek corporate credit spread after stripping out compensation for expected default risk — the credit-market literature’s premier forward leading indicator, with a 50-year multi-regime history back to 1973. The source persists three fields per cycle: ebp (the excess premium, % points, can be negative), gz_spread (the parent GZ spread, % points), and ebp_recession_prob (the model’s recession probability, a 0-1 fraction; renamed from the upstream est_prob column). Cadence is monthly — the Fed refreshes the file roughly once a month, last row dated YYYY-MM-01, so only the low-frequency profiles (overnight, extended, full) earn a fresh row. Ingested + persisted only: the three fields land on daily_signals for future scoring / matcher use but are not yet wired into the health score or alignment — that is a later, separately-gated step.

Execution order

Fast HTTP sources run first (FRED, liquidity, darkpool, news, sentiment) so the heavy yfinance sources (market, eps, breadth) don’t gate the report on slow network paths. Breadth specifically downloads in batches of 50 tickers with 2-second delays to stay under yfinance’s rate limits.

Every fetch routes through DataSource.safe_fetch() in app/sources/base.py — uniform retry with exponential backoff, WARNING on transient failure, logger.exception() on fatal, returns a default on terminal failure. New sources must use it.

See also

• Schedule — when each profile fires, which sources each profile pulls.
• Lifecycle — what happens to fetched data inside a report cycle.
• Source health — how source_runs tracks ok/fail and the 95%/80% escalation rules.
• DIX and GEX — the two signals that flow through the darkpool source.
• Source code: app/sources/ (one file per source). Inventory + provider table also in app/sources/README.md.