Data Sources

Eighteen external feeds drive the platform. Each is registered as a DataSource subclass in app/sources/ and pulled by engine.py:fetch_all_sources() on the cadence dictated by the active collection profile.

The full inventory

Three auxiliary fetches run alongside the registered sources and stash results under underscore-prefixed keys (_gamma_levels, _fed_watch, _correlations) so format_report can find them without colliding with per-source dicts:

• Gamma profile — Brokerage option-chain pull for the per-strike gamma_levels API path. Requires brokerage auth; logs and continues on token failure.
• Fed Watch — CME rate-cut probability surface for the next FOMC meeting.
• Correlations — rolling SPY/VIX, SPY/DXY, SPY/TNX, SPY/Oil pair correlations computed post-hoc from the persisted daily_signals history. Not an external fetch — it’s a SQL query against our own history.

Provider mix and fallback posture

• market tries the brokerage market-data feed first, falls back to the public market-data fallback when the brokerage returns None/empty. The chain-computed quantities (chain net-Γ, gamma walls, ZGL, PCR — not the dealer-gamma analytics GEX aggregate, which arrives via darkpool) require brokerage option chains and have no fallback. VIX uses volume-weighting since the brokerage reports no OI for index options. TNX and DXY have no option chains.
• The brokerage feed is optional. Missing or expired tokens log CRITICAL but the platform continues on the public market-data fallback / FRED / other sources. See source health for how token status surfaces on /api/v1/status.
• The third-party news-sentiment feed was rate-limited at 25 requests/day and retired 2026-06 (DOCTRINE D22); /api/v1/source-health surfaces remaining budget for active rate-limited sources.
• News fan-out — the news source merges three feeds: FXStreet financial headlines, Iran International liveblog (for geopolitical risk), and the Hormuz Strait tanker monitor (for oil-supply tail risk).
• ebp pulls the Federal Reserve’s monthly FEDS Notes Excess Bond Premium CSV. EBP is the residual of the Gilchrist-Zakrajšek corporate credit spread after stripping out compensation for expected default risk — the credit-market literature’s premier forward leading indicator, with a 50-year multi-regime history back to 1973. The source persists three fields per cycle: ebp (the excess premium, % points, can be negative), gz_spread (the parent GZ spread, % points), and ebp_recession_prob (the model’s recession probability, a 0-1 fraction; renamed from the upstream est_prob column). Cadence is monthly — the Fed refreshes the file roughly once a month, last row dated YYYY-MM-01, so only the low-frequency profiles (overnight, extended, full) earn a fresh row. Ingested + persisted only: the three fields land on daily_signals for future scoring / matcher use but are not yet wired into the health score or alignment — that is a later, separately-gated step.
• cboe pulls free, no-key CBOE CDN CSVs (daily cadence, full / extended profiles). Five data_only carriers for the drawdown-risk / state lane: three options put/call ratios — cboe_equity_pc (equity-only), cboe_total_pc (equity + index), cboe_index_pc (index-only) — and two VIX term-structure points, vix9d_close (9-day, front of the curve) and vix6m_close (6-month, back). The intermediate VIX3M and the derived VIX term slope / ratio are already persisted elsewhere (Law 3 — not re-emitted here). Each CSV leg is fetched independently through safe_fetch, so a single dead file degrades that one field to null and leaves the others intact. Ingested + persisted only — not yet wired into the health score or alignment.
• ofr pulls the OFR (US Treasury Office of Financial Research) Financial Stress Index from a free, no-key daily CSV (financialresearch.gov, daily T-2, full / extended profiles). Six data_only carriers for the drawdown-risk / state lane: the composite ofr_fsi (>0 = above-average stress) plus five subindices — ofr_fsi_credit, ofr_fsi_equity_valuation, ofr_fsi_safe_assets, ofr_fsi_funding, ofr_fsi_volatility. The three geographic split columns (US / other-advanced / emerging) are deliberately dropped. It is a second, model-independent stress read alongside the FRED STLFSI4 / KCFSI legs. The single CSV routes through safe_fetch; a dead URL or unparseable file degrades the whole dict to empty without raising. Ingested + persisted only — not yet wired into the health score or alignment.
• naaim pulls the NAAIM Exposure Index via a free, two-step weekly page-scrape (naaim.org). The data-file XLSX href rotates each week, so the source scrapes the program page for the current link, then parses the workbook (via openpyxl) and reads the latest row’s “NAAIM Number” into naaim_exposure — the average equity exposure NAAIM-member active managers report running (~-200 leveraged short to +200 leveraged long). A data_only positioning / contrarian carrier for the squeeze-setup and drawdown-risk lane. The source is fragile by design: a non-matching href regex, an empty page, or a missing openpyxl all degrade to empty without raising, and a latest row older than ~14 days still returns the value but logs a staleness warning. Browser-like request headers are sent to avoid bot-filtering. Ingested + persisted only — not yet wired into the health score or alignment.
• options-analytics-live pulls the live options-analytics dealer-risk + levels archive for SPY from two REST endpoints. It persists a fa_*-prefixed family: settled vs live aggregate net dealer gamma + delta (normalized to billions to share the dealer-gamma analytics GEX scale), the intraday flow-shift fractions, a categorical fa_flow_direction (amplifying / dampening / neutral / no_flow / regime_flip), the live gamma-flip / call-wall / put-wall / max-pain price levels, and an fa_as_of vintage stamp. This is a SECOND vendor’s positioning read, deliberately kept distinct from the dealer-gamma analytics GEX and the option-chain gamma walls. Each endpoint is fetched independently through safe_fetch, so a 5xx / timeout / null on one endpoint degrades only its fields and never poisons the other or the cycle; with no API key the source returns empty and logs a warning. Captured forward as a hedge — all data_only carriers, not yet wired into the health score or alignment (forward-accruing until graded).
• edgar_xbrl pulls SEC EDGAR’s free companyfacts API for a fixed six-name entity set — no key, no rate concern (six requests per cycle against SEC’s ~10 req/s ceiling). Unlike every other source in this table it does NOT feed daily_signals: it persists directly to its own append-only table, because the data is per-company audited history, not a daily macro reading. The interesting engineering is the extraction, not the fetch — EDGAR reports cash-flow and income figures as fiscal-year-to-date cumulative totals, not clean quarterly numbers, so the source derives genuine quarterly deltas (and correctly handles a company’s underlying reporting tag changing over the years) before writing anything. See the AI capital cycle framework page for how the derived reads feed the T13/T10 triggers.

Macro calendar — what’s filled, what isn’t

The platform seeds a forward macro-event calendar (calendar_events) that powers the air-pocket flag’s catalyst leg and the AI brief’s calendar_lookahead. The three high-impact monthly prints — Employment Situation (NFP), CPI, PPI — are seeded from FRED’s free release/dates endpoint, plus FOMC decisions and the weekly FRED release projections.

Each macro-event row carries consensus and actual columns. The two behave differently:

• consensus is NULL by constraint. The market’s prior-expectation survey for these releases has no free, terms-of-service-clean source. FRED’s release endpoint serves the schedule only — the published date, never the street’s forecast. Consensus figures live behind paid econ-calendar vendors, which the platform deliberately avoids. So the print-vs-consensus surprise read (the thing the calendar would unlock) stays blocked until a free consensus source appears or a paid one is sanctioned. This is an honest data gap, not a bug — the column exists, waiting for a source that does not yet exist.
• actual is wired for the unambiguous prints. The realized headline figure is free from FRED (the underlying series themselves), so after a print lands the backfill fills actual for the releases whose series → quoted-headline mapping is unambiguous:
  • NFP → PAYEMS (Total Nonfarm Payrolls); headline = month-over-month change in the level, in thousands (e.g. +139K).
  • CPI → CPIAUCSL (CPI-U, All Items, SA); headline = month-over-month percent change (e.g. +0.2% m/m).
  • PPI is deliberately left NULL. PPI has two competing “headline” series in common quotation (final-demand vs all-commodities) and the quoted figure isn’t unambiguously one of them — so per the rule that a wrong actual is worse than an empty one, PPI stays unfilled until a canonical mapping is settled.

Constraint: no free consensus survey for NFP/CPI/PPI; no canonical headline mapping for PPI. Falsifier: a free, ToS-clean consensus feed (or a sanctioned paid one) becomes available, or PPI’s quoted-headline series is settled. Unblock: wire consensus into _emit_fred_release_calendar (consensus=) and the surprise framing follows; add the PPI entry to RELEASE_ACTUAL_MAP and its actual wires automatically.

Execution order

Fast HTTP sources run first (FRED, liquidity, darkpool, news, sentiment) so the heavy market-data fallback sources (market, eps, breadth) don’t gate the report on slow network paths. Breadth specifically downloads in batches of 50 tickers with 2-second delays to stay under the market-data fallback feed’s rate limits.

Every fetch routes through DataSource.safe_fetch() in app/sources/base.py — uniform retry with exponential backoff, WARNING on transient failure, logger.exception() on fatal, returns a default on terminal failure. New sources must use it.

See also

• Schedule — when each profile fires, which sources each profile pulls.
• Lifecycle — what happens to fetched data inside a report cycle.
• Source health — how source_runs tracks ok/fail and the 95%/80% escalation rules.
• DIX and GEX — the two signals that flow through the darkpool source.
• Source code: app/sources/ (one file per source). Inventory + provider table also in app/sources/README.md.