Data Sources

Eighteen external feeds drive the platform. Each is registered as a DataSource subclass in app/sources/ and pulled by engine.py:fetch_all_sources() on the cadence dictated by the active collection profile.

The full inventory

Three auxiliary fetches run alongside the registered sources and stash results under underscore-prefixed keys (_gamma_levels, _fed_watch, _correlations) so format_report can find them without colliding with per-source dicts:

• Gamma profile — Brokerage option-chain pull for the per-strike gamma_levels API path. Requires brokerage auth; logs and continues on token failure.
• Fed Watch — CME rate-cut probability surface for the next FOMC meeting.
• Correlations — rolling SPY/VIX, SPY/DXY, SPY/TNX, SPY/Oil pair correlations computed post-hoc from the persisted daily_signals history. Not an external fetch — it’s a SQL query against our own history.

Provider mix and fallback posture

• market tries the brokerage market-data feed first, falls back to the public market-data fallback when the brokerage returns None/empty. The chain-computed quantities (chain net-Γ, gamma walls, ZGL, PCR — not the dealer-gamma analytics GEX aggregate, which arrives via darkpool) require brokerage option chains and have no fallback. VIX uses volume-weighting since the brokerage reports no OI for index options. TNX and DXY have no option chains.
• The brokerage feed is optional. Missing or expired tokens log CRITICAL but the platform continues on the public market-data fallback / FRED / other sources. See source health for how token status surfaces on /api/v1/status.
• The third-party news-sentiment feed was rate-limited at 25 requests/day and retired 2026-06 (DOCTRINE D22); /api/v1/source-health surfaces remaining budget for active rate-limited sources.
• News fan-out — the news source merges three feeds: FXStreet financial headlines, Iran International liveblog (for geopolitical risk), and the Hormuz Strait tanker monitor (for oil-supply tail risk).
• ebp pulls the Federal Reserve’s monthly FEDS Notes Excess Bond Premium CSV. EBP is the residual of the Gilchrist-Zakrajšek corporate credit spread after stripping out compensation for expected default risk — the credit-market literature’s premier forward leading indicator, with a 50-year multi-regime history back to 1973. The source persists three fields per cycle: ebp (the excess premium, % points, can be negative), gz_spread (the parent GZ spread, % points), and ebp_recession_prob (the model’s recession probability, a 0-1 fraction; renamed from the upstream est_prob column). Cadence is monthly — the Fed refreshes the file roughly once a month, last row dated YYYY-MM-01, so only the low-frequency profiles (overnight, extended, full) earn a fresh row. Ingested + persisted only: the three fields land on daily_signals for future scoring / matcher use but are not yet wired into the health score or alignment — that is a later, separately-gated step.
• cboe pulls free, no-key CBOE CDN CSVs (daily cadence, full / extended profiles). Five data_only carriers for the drawdown-risk / state lane: three options put/call ratios — cboe_equity_pc (equity-only), cboe_total_pc (equity + index), cboe_index_pc (index-only) — and two VIX term-structure points, vix9d_close (9-day, front of the curve) and vix6m_close (6-month, back). The intermediate VIX3M and the derived VIX term slope / ratio are already persisted elsewhere (Law 3 — not re-emitted here). Each CSV leg is fetched independently through safe_fetch, so a single dead file degrades that one field to null and leaves the others intact. Ingested + persisted only — not yet wired into the health score or alignment.
• ofr pulls the OFR (US Treasury Office of Financial Research) Financial Stress Index from a free, no-key daily CSV (financialresearch.gov, daily T-2, full / extended profiles). Six data_only carriers for the drawdown-risk / state lane: the composite ofr_fsi (>0 = above-average stress) plus five subindices — ofr_fsi_credit, ofr_fsi_equity_valuation, ofr_fsi_safe_assets, ofr_fsi_funding, ofr_fsi_volatility. The three geographic split columns (US / other-advanced / emerging) are deliberately dropped. It is a second, model-independent stress read alongside the FRED STLFSI4 / KCFSI legs. The single CSV routes through safe_fetch; a dead URL or unparseable file degrades the whole dict to empty without raising. Ingested + persisted only — not yet wired into the health score or alignment.
• naaim — RETIRED 2026-09-01, history only. The NAAIM Exposure Index measured the average equity exposure NAAIM-member active managers report running (~-200 leveraged short to +200 leveraged long) — a data_only positioning / contrarian carrier for the squeeze-setup and drawdown-risk lane, pulled via a free two-step weekly page-scrape of naaim.org. NAAIM moved the index behind a paid subscription (index.naaim.org) effective 2026-08-01, so no compliant free path remains and the series can never advance. It is no longer polled; the persisted history is retained and still served on /signals/history, /signals/sparklines, and positioning.naaim_exposure, badged with its frozen weekly as-of date. The source module + parser are kept dormant for a future paid revival. It was never wired into the health score or alignment.
• options-analytics-live pulls the live options-analytics dealer-risk + levels archive for SPY from two REST endpoints. It persists a fa_*-prefixed family: settled vs live aggregate net dealer gamma + delta (normalized to billions to share the dealer-gamma analytics GEX scale), the intraday flow-shift fractions, a categorical fa_flow_direction (amplifying / dampening / neutral / no_flow / regime_flip), the live gamma-flip / call-wall / put-wall / max-pain price levels, and an fa_as_of vintage stamp. This is a SECOND vendor’s positioning read, deliberately kept distinct from the dealer-gamma analytics GEX and the option-chain gamma walls. Each endpoint is fetched independently through safe_fetch, so a 5xx / timeout / null on one endpoint degrades only its fields and never poisons the other or the cycle; with no API key the source returns empty and logs a warning. Captured forward as a hedge — all data_only carriers, not yet wired into the health score or alignment (forward-accruing until graded).
• edgar_xbrl pulls SEC EDGAR’s free companyfacts API for a fixed six-name entity set — no key, no rate concern (six requests per cycle against SEC’s ~10 req/s ceiling). Unlike every other source in this table it does NOT feed daily_signals: it persists directly to its own append-only table, because the data is per-company audited history, not a daily macro reading. The interesting engineering is the extraction, not the fetch — EDGAR reports cash-flow and income figures as fiscal-year-to-date cumulative totals, not clean quarterly numbers, so the source derives genuine quarterly deltas (and correctly handles a company’s underlying reporting tag changing over the years) before writing anything. See the AI capital cycle framework page for how the derived reads feed the T13/T10 triggers.

Macro calendar — what’s filled, what isn’t

The platform seeds a forward macro-event calendar (calendar_events) that powers the air-pocket flag’s catalyst leg and the AI brief’s calendar_lookahead. The three high-impact monthly prints — Employment Situation (NFP), CPI, PPI — are seeded from FRED’s free release/dates endpoint, plus FOMC decisions and the weekly FRED release projections.

The long-end-stress Phase 3B rates-event calendar adds two more emitters, both calendar_events rows on the same forward-lookahead substrate: treasury_auctions seeds announced Treasury coupon auctions (10Y / 20Y / 30Y) from the free TreasuryDirect TA_WS securities API (auction date at the 13:00 ET coupon close; 20Y/30Y high-importance, 10Y medium), and rates_calendar seeds hand-entered dated structural events from a committed file — Bank of Japan meeting dates, US quarterly-refunding announcements, the Dutch pension transition wave, the UK DMO remit, the OBR fiscal risks report, the OECD Global Debt Report. Both are idempotent (one row per event, re-emit is a no-op) and gated to once per ET day. The calendar records and reacts to these dates; it never forecasts an embargoed policy outcome.

Each macro-event row carries consensus and actual columns. The two behave differently:

• consensus is NULL by constraint. The market’s prior-expectation survey for these releases has no free, terms-of-service-clean source. FRED’s release endpoint serves the schedule only — the published date, never the street’s forecast. Consensus figures live behind paid econ-calendar vendors, which the platform deliberately avoids. So the print-vs-consensus surprise read (the thing the calendar would unlock) stays blocked until a free consensus source appears or a paid one is sanctioned. This is an honest data gap, not a bug — the column exists, waiting for a source that does not yet exist.
• actual is wired for the unambiguous prints. The realized headline figure is free from FRED (the underlying series themselves), so after a print lands the backfill fills actual for the releases whose series → quoted-headline mapping is unambiguous:
  • NFP → PAYEMS (Total Nonfarm Payrolls); headline = month-over-month change in the level, in thousands (e.g. +139K).
  • CPI → CPIAUCSL (CPI-U, All Items, SA); headline = month-over-month percent change (e.g. +0.2% m/m).
  • PPI is deliberately left NULL. PPI has two competing “headline” series in common quotation (final-demand vs all-commodities) and the quoted figure isn’t unambiguously one of them — so per the rule that a wrong actual is worse than an empty one, PPI stays unfilled until a canonical mapping is settled.

Constraint: no free consensus survey for NFP/CPI/PPI; no canonical headline mapping for PPI. Falsifier: a free, ToS-clean consensus feed (or a sanctioned paid one) becomes available, or PPI’s quoted-headline series is settled. Unblock: wire consensus into _emit_fred_release_calendar (consensus=) and the surprise framing follows; add the PPI entry to RELEASE_ACTUAL_MAP and its actual wires automatically.

Execution order

Fast HTTP sources run first (FRED, liquidity, darkpool, news, sentiment) so the heavy market-data fallback sources (market, eps, breadth) don’t gate the report on slow network paths. Breadth specifically downloads in batches of 50 tickers with 2-second delays to stay under the market-data fallback feed’s rate limits.

Every fetch routes through DataSource.safe_fetch() in app/sources/base.py — uniform retry with exponential backoff, WARNING on transient failure, logger.exception() on fatal, returns a default on terminal failure. New sources must use it.

See also

• Schedule — when each profile fires, which sources each profile pulls.
• Lifecycle — what happens to fetched data inside a report cycle.
• Source health — how source_runs tracks ok/fail and the 95%/80% escalation rules.
• DIX and GEX — the two signals that flow through the darkpool source.
• Source code: app/sources/ (one file per source). Inventory + provider table also in app/sources/README.md.