Get started

Rollout and decision watchdog

Source

Decision-oriented observability over live workloads and active lane rules: whether current prices, compliance terms, three-provenance capabilities, and runtime telemetry still support the standing decision, and where each shadow or route rule stands on the canary ladder.

The two questions it answers

Ordinary runtime metrics report request counts and latency percentiles without knowing what ruling authorized the lane or what assumptions that ruling relied upon. The watchdog (platform/engine/watchdog.mjs) joins the routing record back to the decision ledger to answer two concrete operational questions:

1. Does reality still support the standing decision?

  • Price assumption drift: compares current Witness or entered unit cost against the unit cost pinned in standingDecision.inputs (pricePeriodIds and enteredPriceIds), flagging moves above maxPriceDriftPct (default +10%).
  • Compliance and capability eligibility: re-evaluates the serving SKU against witness.provider_term (requiresZeroRetention, requiresNoTraining, requiresRegions, allowsProviders) and three-provenance capability bounds (requiresCapabilities, requiresVerifiedCapabilities, minContextTokens, minThroughputToksSec, maxTtftMs).
  • Quality and critical failures: checks newest aperture.evidence measurements against qualityFloor and edgeCaseVeto.
  • Operational reliability and latency: checks fallback chain reliance (fallbackSharePct <= maxFallbackSharePct, default 10%), lane unserved attempt rate (unservedRatePct <= maxUnservedRatePct, default 5%), and observed latency against constraints.latencyP90Ms.

2. Where does each lane rule stand on the canary ladder?

text
watching  ->  shadow  ->  5% canary  ->  25%  ->  50%  ->  accepted incumbent

For each active rule in Rules within a lane, the watchdog inspects its authority (underDecisionId and promotedDecisionId), candidate eligibility, evaluation evidence, and runtime telemetry (shadowRule: "rule:<n>" would-have matches and serve: "rule:<n>" routed canary attempts):

Rule verdictModeWhen emitted
hold_shadowshadowAccumulating shadow matches (< minShadowMatches), awaiting evaluation evidence, or awaiting a post-shadow decision (standing.decidedAt < rule.addedAt)
promote_readyshadowShadow volume, eligibility, quality floor, post-shadow standing decision, and closed readingsWindow + sourceRef citation are all present
retire_shadowshadowTarget SKU fails workload eligibility, falls below qualityFloor, or hits a vetoed critical failure
healthy_canaryrouteRouted canary is serving within error, latency, compliance, and quality bounds while accumulating minCanaryUnits
expand_or_adopt_readyrouteRouted canary cleared minCanaryUnits, qualityFloor, and economic threshold; ready for the next nested FNV-1a share rung (5 -> 25 -> 50) or full lane adoption
rollback_recommendedrouteRouted canary breached maxUnservedRatePct, latencyP90Ms, qualityFloor, edgeCaseVeto, or compliance/capability eligibility
stale_authorityeitherRule's authorizing decision was superseded by a newer accepted decision on the lane

Epistemic boundary and zero silent mutation

Two boundaries hold across the CLI, the org endpoint, and the stateless endpoint:

  • Availability is never mistaken for answer quality. Runtime unserved counts (failed + refused + timed_out) measure transport and provider availability (unservedRatePct). Answer quality (successRatePct) enters exclusively from recorded aperture.evidence evaluations.
  • Zero silent mutation. Neither `GET /api/v1/org/watchdog`, `POST /api/v1/watchdog`, nor `aperture:watchdog` modifies aperture.lane_rule or aperture.decision. Instead, every actionable verdict emits the exact copy-ready operator command (aperture:decide, aperture:rule:promote, aperture:rule:add, or aperture:rule:remove) so every routing change resolves to an explicit, dated ruling.