Skip to content

KB / operations

Report Lifecycle

Last verified

Every report cycle — whether hot (5 min), market (15 min), full (anchor), extended, or overnight — flows through the same four-stage orchestrator in app/engine.py:generate_report_logic(). The orchestrator is thin; the substance lives in the four helpers it delegates to.

Stage 1 — Fetch

fetch_all_sources(profile) instantiates the full source registry, filters to the active profile, and runs each source on the thread pool (run_in_executor). For every source:

Three auxiliary fetches run alongside: gamma profile (Schwab option chain), Fed Watch (CME rate probabilities), and rolling correlations (a SQL query against daily_signals history, not an external call). Stashed under underscore-prefixed keys so format_report finds them without collision.

Diagnostic pruning fires here: source_runs older than RETENTION_SOURCE_RUNS_DAYS (default 30) are deleted. signal_transitions and news_items retention is indefinite unless RETENTION_TRANSITIONS_DAYS / RETENTION_NEWS_DAYS are set explicitly.

Stage 2 — Format

format_report(template, aggregated_data, prev_ctx) is the deterministic, side-effect-free part of the cycle. It:

  1. Builds the template context dict from raw source payloads — column ordering, derived signals, change markers (vs prev_ctx), bridge text, regime classification, alignment categories, and the health score.
  2. Renders app/templates/report.md via Jinja2 ({{ field }}, num filter — see Law 6 in the repo CLAUDE.md). The num filter returns "N/A" for missing or non-numeric values so failed sources don’t leak $0.00 / 0.00% into the rendered report.
  3. Stashes the canonical signals dict on the in-process _engine_sig_input handoff for the next stage to consume. This is the Law 4 contract — capture reads from the source_raw parallel store with the in-process handoff as fallback, never re-parses the rendered markdown. (Stage 3 sunset, 2026-06-17 — the legacy .signals_snapshot.json sidecar was retired; the canonical priority chain is now _READ_PRIORITY = ("source_raw", "handoff").)

Stage 3 — Persist (report files)

write_report_files(report_content, timestamp) writes the rendered markdown twice:

Failures here log CRITICAL but never raise — a report file write failing must not kill the next stage’s database update.

Stage 4 — Capture

capture_signals_hook(timestamp, profile) is the database-write stage:

  1. Read the latest source_raw rows (or fall back to the in-process handoff dict) and persist the canonical signals into daily_signals (one row per trading date, upserted) and intraday_signals (one row per cycle).
  2. Compute and write the health score via a post-hoc UPDATE so the persisted score reflects the full sig_input.
  3. Emit any new signal_transitions rows by diffing the current classifier state against the previous snapshot. See event feed for the classifier inventory and dedup rules.
  4. Evaluate the 63 alert definitions in alert_thresholds.json and write any newly-fired or newly-resolved rows into alerts.
  5. Record predictions into base_rate_predictions for the calibration loop.

The whole stage is wrapped in a broad try/except — a capture failure logs CRITICAL but never raises into the looper. The next cycle gets a clean attempt.

Failure modes

FailureBehavior
Single source raisessource_runs records ok=False, cache holds previous value, cycle continues
Single source returns {}source_runs records ok=False with error="empty result", cache holds previous value
Format raisesCycle fails loudly; no report file written; next cycle retries
Both source_raw and handoff missCapture logs CRITICAL and aborts the cycle (P0 — no silent degradation); next cycle retries
Capture raisesLogs CRITICAL; report file is on disk but DB row is from previous cycle
Report file write failsLogs CRITICAL but capture still runs — DB row stays consistent with the in-memory render

See also