Collection Schedule

Data collection runs on seven named profiles. Six of them (overnight, extended, full, market, screener, darkpool_late) are driven by a static 42-slot list in app/config.py:SCHEDULE. The seventh (hot) runs on its own 5-minute timer, only while the market session is actually open.

The seven profiles

The static SCHEDULE carries 42 time slots across the six background-looper profiles. The hot profile is not in SCHEDULE — it runs on its own interval and only during the live session.

Weekend & holiday behavior

Market data does not move when the market is closed, and a closed-day cycle used to persist carry-forward signal rows — including genuinely drifted weekend vendor quotes — that poisoned every trading-day computation downstream. Since 2026-07 the platform guards closed days at two layers:

• Scheduler layer — background_looper skips the market-data profiles (extended, full, market, screener, darkpool_late) when a slot fires on a non-trading day (weekend or full NYSE holiday). overnight keeps running: its sources (news, SEC filings, the AI-capital-cycle web collectors) are calendar-agnostic and persist to their own tables. hot_looper gates on the holiday-aware session check.
• Capture layer — the signal-capture stage is a no-op for any non-session trade date, no matter which cycle reached it: no daily or intraday signal rows, no transitions, no alerts. This is the invariant; the scheduler skips are the optimization.

Early-close half-days are trading days. The 13:00 ET close still carries real bars, so every slot fires and rows persist normally on days like the Friday after Thanksgiving. (The AI-brief scheduler throttles separately on those days — premium AI calls are a cost decision, not a data decision.)

Why seven profiles

Each profile is a tradeoff between freshness and load. The shapes:

• overnight — the markets are closed but news, Fed liquidity, and the low-cadence collectors keep moving. A light pull every ~2h keeps the news feed warm, catches overnight Fed-balance-sheet operations, and gives the once-a-day filing/web collectors their slot without touching the market-data budget.
• extended — pre-market (08:30, 09:00, 09:15) and post-market (16:30, 17:00, 18:00) windows. Pull the macro skeleton plus weekly positioning (cot, aaii) so the open and close both ship with fresh data.
• full — the four anchor cycles. Open (09:31), midday (12:00), pre-close (15:45), and close (16:01). Every source fires, including the slower breadth batch.
• market — the 15-minute intraday cadence during RTH. Pulls only news, the one genuinely intraday-fast source. credit, liquidity, and fred were dropped from this tick (2026-06): they publish on daily/weekly cadence, so refetching them every 15 min was wasted work — they stay covered by extended (6 slots) and full (4 slots) through the trading day. Note: the 10:30 slot is occupied by the screener; the flanking 10:15 and 10:45 market slots maintain the news cadence.
• hot — the 5-minute “fast pulse” during the live session. Three sources only (market, vol_structure, energy) so the dashboard’s health score and intraday sparklines stay near-live without overloading the brokerage data-session budget.
• screener — the single-name screener. Runs twice daily by operator decision (2026-06-05): once at 10:30 ET (RTH morning snapshot — catches post-earnings gaps and opening momentum ~1 h into the session) and once at 17:30 ET (post-close settled-bar run). The 10:30 board is a mid-session snapshot — today’s Close is the current market price, not the EOD print; the API response carries is_intraday_snapshot=true to signal this. Request-spreading (batch size + inter-batch delay, configurable via SCREENER_FETCH_BATCH_SIZE / SCREENER_FETCH_BATCH_DELAY_S env vars, default 50 tickers / 2 s) keeps the ~903-name market-data pull from hitting the rate budget harder than the existing breadth batch.
• darkpool_late — a dark-pool-only pass at 21:45 ET. The vendor posts the same-day print in the ~20:00–22:00 ET window, after the last full cycle; this slot lands it before midnight so the day’s row carries its own print instead of the prior day’s.

Cycle locking

generate_report_logic() is wrapped by a module-level asyncio.Lock (_report_lock). The startup invocation and the two loopers all queue on the same lock — a full cycle that runs long won’t get clobbered by the next market tick; the market cycle just waits its turn. Cycle-shared globals (_source_cache, _prev_context) are mutated inside the lock; any code that reads or writes them from outside the cycle must take the lock first. C5 (2026-06): the slow post-capture re-render now runs outside _report_lock (its report.md write is serialized by a dedicated _render_lock), so a long full cycle no longer delays a queued hot cycle by its re-render duration.

Release-aware scheduling

The signal registry’s release_calendar slot (declared in signal_definitions.json) carries {publishes, weekday, approximate_time_et, lag_days} for every metric. The release-aware scheduler (app/release_scheduler.py, live in production since 2026-06-04) reads these for its scheduled-source set (cot, aaii) — firing each on its upstream publish window and letting the static cycle reuse the cached payload when it is fresh relative to the last release.

See also

• Data sources — what each source pulls and from where.
• Lifecycle — what happens inside one cycle, end-to-end.
• Source health — how cycle outcomes get tracked.
• Code: app/config.py:SCHEDULE / PROFILES, app/main.py:background_looper / hot_looper.