Source
One fetchable document, and the adapter that knows how to read it.
Sources are registry rows in witness.source.
Fields
| Field | Meaning |
|---|---|
id | Stable source id |
providerId | Who publishes it |
url | What the collector fetches |
publicUrl | The page a citation should point a human at |
adapter | The parser that reads this document |
enabled | Whether the collector reads it |
cadenceMinutes | How often, default 60 |
robotsStatus | What robots.txt said |
notes | Free text |
Why url and publicUrl are two columns
What a machine should read is not always what a person should read. Anthropic's pricing parses cleanly from the documentation's Markdown mirror; Google's needs a locale-pinned URL to answer consistently.
Citing those would send a reader to a mirror or to a URL carrying a locale parameter they did not choose. So the tape cites publicUrl and the collector fetches url, and a citation never points at scaffolding.
Provider domains are the inclusion rule
witness.provider carries a domains array, and it does real work. A curated market event is refused unless its source URL sits on one of the provider's own registrable domains, subdomains included.
That turns "is this the provider's own announcement?" from a judgement call made per entry into a constraint the database enforces. A press summary, a news article or a well-informed blog post cannot become a curated event, however accurate it is.
Health
curl https://vitna.ai/api/v1/sources/healthPublic, deliberately. A source that has stopped answering is part of the record: a tape that silently stopped updating because a page moved looks exactly like a market that stopped moving.
The endpoint flags a source stale past its cadence and failing at three or more consecutive failures. It answers 503 when the record cannot be read at all, rather than reporting green from a committed file, which would be the one lie this system exists to prevent.
Fetch outcomes
Every read writes one witness.fetch row with a final status, inserted once after both the fetch and the parse have been decided, so nothing about a fetch is ever rewritten:
| Status | Meaning |
|---|---|
ok | Fetched and parsed. Has a contentSha256. |
fetch_failed | Could not be retrieved. Has an error. |
parse_failed | Retrieved, but the adapter could not read it. Has an error. |
robots_blocked | robots.txt disallowed the read. |
Two database constraints hold the shape: an ok row must have evidence, and a non-ok row must have a reason.
