The collector
A oneshot unit fired hourly with jitter, running one collector pass out of the deployed checkout.
Scheduled runs are tagged with a run trigger so they are told apart from a hand-run in the fetch_run table.
What one pass does
Reads every enabled document, archives each under the sha256 of its bytes, appends the observations it parsed, runs change detection, records what it saw and did not see, evaluates the armed triggers, and closes the run with a summary.
Three phases, three transactions
Detection, discovery and triggers each run in their own transaction with their own log line. A phase that fails is rolled back and named on stderr, and the run still records everything it read.
That isolation matters most for the third one: a broken trigger must never cost the market record an hour of reads.
A read reports three things
| Outcome | Where it goes |
|---|---|
| Cells found | Observations |
| Registered skus not found | An absence count |
| Priced rows the registry never heard of | The candidate queue |
Three consecutive successful reads without a sku close its periods and mark it delisted. A read that carries it again opens a new period and makes it active.
No market event is written by any of that. A delisting is what the record saw. Calling it a deprecation is a claim, entered by hand.
The half-catalogue guard
A read where more than half of a source's registered skus are absent is recorded as a parse failure with the count, whatever the adapter returned, and touches no period.
Half a provider's catalogue vanishing in one hour is a page that changed shape. Delisting a whole board on that guess is the expensive mistake, and the guard exists because the cheap mistake, one hour of missing reads, is recoverable.
One failing source does not fail the run
Each source runs in its own transaction. A failure is recorded with its error, every other source still lands, and the run exits zero. Only a run that could not start at all exits non-zero.
A source whose adapter the deployed revision does not carry is recorded as a parse failure naming that reason, so a configuration error is named rather than appearing as a source that quietly disappeared.
