Screener Pipeline & Lifecycle (Operations)

This is the how. For the vision behind it — why a floating funnel, the two-axis model, the predictive payoff — see The Screener System (vision). For what each pattern looks for, see the archetype pages.

Status convention. Sections tagged LIVE describe shipped behavior; BUILDING describes the target this doc is the spec for, with code converging to it. The distinction is kept honest on purpose — the doc is the contract, not a press release.

The funnel, stage by stage

A screener pass (app/screener/pipeline.py:run_screener) walks five steps:

1. Universe selection LIVE — resolved from screener_config.json → universe.tier (sp500 | sp900, default sp900 = ~903 names) by app/screener/companies.py:universe_tickers(), sourced from app/data/sp900_companies.json (membership-tagged sp500 / sp400). S1 is OHLCV-only — the S&P 900 is index-curated liquid, so gate_pass=True by membership; no per-name .info quality fetch in the sweep.
2. Feature gathering LIVE — build_features_for_universe() downloads OHLCV for every universe member and persists to screener_features_daily (so historical replay isn’t starved).
3. Archetype scoring LIVE — score_universe() runs the registered evaluators, emitting one row per (ticker, archetype) that reaches partial or primed.
4. Ranking & confluence LIVE — rank_candidates() dedupes by ticker (consolidating multi-archetype fires into one row with confluence_count) and sorts by confluence_count desc, then blended_score_norm desc.
5. Persistence LIVE — ranked rows land in screener_candidates.

The gate dial (target: ~20 survivors, no hard cap) BUILDING

The shortlist size is governed by the gates, not a truncation. The S1→S2 funnel config lives in screener_config.json → funnel.s1_to_s2 (max_promote_count, min_confluence_score, include_watchlist). The target is to tune these so a normal pass naturally leaves ~20 names — intense enough that only genuine setups clear, loose enough that the board is rarely empty.

• No [:N] hard cap. The count floats with opportunity. Reading the count is part of reading the market.
• Drift alarm — a pass yielding 0 or >~50 survivors logs loudly: that’s the gates slipping, not the tape moving. Re-tune by config.
• Stale-board fallback — a legitimately-empty strict pass (e.g. before the day’s first run at 10:30 ET) falls back to the last good board rather than blanking the dashboard (is_stale_fallback).

Known bug being fixed: the universe API route currently hardcodes sp500_only=True, so the board shows 503 “S&P 500 members” instead of the configured ~903 S&P 900. The DB already holds all 903 rows; only the read path and its UI label are wrong. Fix resolves the tier from config.

Archetypes

Each S1 name is run through every registered archetype evaluator (app/screener/archetypes.py). Archetype IDs are the canonical descriptive strings (oversold_sympathy, positioning_extreme, …); the short codes A1–A6 / N1–N3 are display metadata only (ARCHETYPE_META).

A4 and A6 evaluators exist in code but are absent from _SYNC_ARCHETYPE_EVALUATORS, so they don’t fire yet — wiring them is part of the build-out. The N* short archetypes are designed (see their pages) and not yet implemented. Full per-pattern detail: /kb/archetype.

A7 is an async deep-path candidate-GENERATOR — deliberately absent from _SYNC_ARCHETYPE_EVALUATORS (it needs DB reads for its dealer-net-delta history

• a confirmed-earnings gate). It runs per-name over the S2 ∪ pinned set in the deep stage and feeds the screener_predictions paper-tracking loop ONLY — never a trade signal. Sparse-by-design ABSTAIN like the options legs; additionally abstains until each name accrues ≥40 days of its own dealer-net-delta history (forward accrual — quiet for weeks after launch by design). Stays uncalibrated (never alerts) until 30+ matured paper-tracked rows exist. See /kb/archetype/a7-earnings-dealer-unwind.

Options legs are sparse by design. The A3 (skew), A4 (IV-percentile), and A6 (implied-move) legs read the latest co_options_daily row, which is watchlist-only (populated by scaffold_company.py, never for the full universe). For the vast majority of S1 names these legs abstain — contribute nothing, no error. Only scaffolded / promoted names carry an options row.

Both feature paths must agree. Every new screener feature is wired into BOTH the universe sweep (build_features_for_universe) and the per-ticker deep path (score_a*_for_ticker → get_features_as_of). Drift between them shipped a 0-candidate bug once; tests/test_feature_path_parity.py enforces parity.

Promotion & demotion lifecycle

Candidate status machine LIVE

screener_candidates status values: new → persisting → decayed → exited (terminal) or promoted (terminal). Transitions are always UPDATEs, never DELETEs. Per-archetype relevance windows (trading days) are the SSOT in screener_config.json § lifecycle.relevance_windows. Access module: app/signals/screener_candidates.py.

Manual promotion to the watchlist LIVE

POST /api/v1/screener/promote transitions a candidacy to promoted, INSERT OR IGNOREs a minimal co_entity row so the name appears on /company/watchlist immediately, then launches scripts/scaffold_company.py as a detached subprocess (start_new_session=True) so the scaffold’s known teardown hang can’t wedge the API worker.

The two-axis lifecycle (target) BUILDING

This is where the vision’s two axes — automatic data tier × manual visibility — become concrete:

• Stateful S2 membership — a table tracking ticker, promoted_at, demoted_at (NULL = active), archetypes_at_promotion, last_qualified_date. A name is “in S2” until a pass stops surfacing it.
• Auto-demotion — a previous S2 member absent from today’s survivors is demoted: demoted_at set, deep + intraday refresh stop. A cheap daily price bar keeps running so the forward outcome stays measurable; nothing is deleted.
• Manual Promote / Demote — a “Promote to Watchlist” control (screener board row + company page) pins a name permanently; a “Demote” control at the bottom of the company page unpins it. Pinning is a visibility flag independent of the data tier — a pinned name keeps the full treatment regardless of S2 status.

Refresh cadences by tier

The 5-min watchlist quote refresh mirrors SPY’s hot_looper but runs as a separate looper (watchlist_hot_looper, target) — a watchlist-quote failure must never stall SPY’s macro signals. One brokerage batch quote covers the whole union (pinned ∪ S2-active), so the rate cost is one request per tick. A refreshed spot price additionally lets the gamma rail recompute its zone + distance-to-walls intraday from the stored walls — no options chain re-fetch.

Data tables & retention

Retention is sacred — every screener + company table is keep-forever by default. A DELETE against any of them without a RETENTION_*_DAYS env-var gate is a P0 revert (tests/test_retention_guard.py enforces). The append-only archetype-fire history across screener_candidates / screener_predictions is the substrate the predictive layer reads — see the vision page.

Core tables: screener_universe, screener_features_daily, screener_candidates, screener_predictions, plus the company-side co_entity, co_price_daily, co_options_daily, co_option_strikes_daily, gamma_profile_daily, co_estimates / co_estimates_periods, co_sentiment_daily / co_sentiment_snapshots, co_ownership, universe_price_daily. Full retention table + env-var overrides live in the repo’s CLAUDE.md.

Where pitfalls arise

• Gate drift → blank or flooded board. Mitigated by the drift alarm + stale fallback; fix by config tuning, never post-hoc truncation.
• Feature-path drift → a name scores differently by evaluation route. Pinned by the parity test; add every new feature to BOTH paths.
• Confusing visibility with data → demoting from the dropdown is not deleting data; auto-demotion is not removing history.
• Sparse options legs → most universe names abstain on A3/A4/A6; that’s correct, not a gap to “fix” by fetching options for all 903 (cost-prohibitive).

Related

• The Screener System (vision)
• Archetypes
• Screener API routes: /api/v1/screener/{candidates, universe, promote, calibration} — see /kb/api.
• The live board: /screener.