Multi-class unit profiles and compound pipelines
Evaluate end-to-end AI workflows that fan out across multiple component classes, attribute runtime x-vitna-run trace volume back to each stage, and separate decision-ready migrations from exploratory class-peer levers.
Multi-class unit profiles across all 10 Witness classes
Production AI workloads consume more than language model tokens. UNIT_TERMS in platform/engine/decide.mjs maps every one of the 33 registered Witness price dimensions across all 10 component classes into unitProfile keys:
| Class | Profile keys | Witness dimensions |
|---|---|---|
model | inputTokensPerUnit, cachedInputTokensPerUnit, cacheWrite5mTokensPerUnit, cacheWrite1hTokensPerUnit, outputTokensPerUnit, batchInputTokensPerUnit, batchOutputTokensPerUnit, priorityInputTokensPerUnit, priorityOutputTokensPerUnit, flexInputTokensPerUnit, flexOutputTokensPerUnit, outputImagesPerUnit, batchOutputImagesPerUnit | input_per_mtok, cached_input_per_mtok, cache_write_5m_per_mtok, cache_write_1h_per_mtok, output_per_mtok, batch_input_per_mtok, batch_output_per_mtok, priority_input_per_mtok, priority_output_per_mtok, flex_input_per_mtok, flex_output_per_mtok, output_per_image, batch_output_per_image |
compute | gpuHoursPerUnit | gpu_hour_on_demand |
retrieval | embedTokensPerUnit, rerankTokensPerUnit | embed_per_mtok, rerank_per_mtok |
speech | audioMinutesBatchPerUnit, audioMinutesStreamingPerUnit, ttsCharsPerUnit | audio_minute_batch, audio_minute_streaming, tts_per_mchar |
tooling | toolCallsPerUnit, pageFetchesPerUnit, containerHoursPerUnit, sessionHoursPerUnit, storageGbDaysPerUnit | tool_call_per_1k, page_fetch_per_1k, container_hour, session_hour, storage_per_gb_day |
extraction | ocrPagesPerUnit, layoutPagesPerUnit | page_ocr, page_layout |
translation | translateInputCharsPerUnit, translateOutputCharsPerUnit | translate_input_per_mchar, translate_output_per_mchar |
image | imagesPerUnit, batchImagesPerUnit, imageMegapixelsPerUnit | image_per_image, batch_image_per_image, image_per_megapixel |
video | videoSecondsPerUnit | video_per_second |
music | musicTracksPerUnit | music_per_track |
Because register.mjs, reconcile.mjs, stackfile.mjs, decide(), and evaluatePipeline() all read from that single UNIT_TERMS table, non-model workloads participate in the exact same decision briefs, invoice reconciliations, counterfactuals, and signed policy workflows as language models.
Compound pipeline Bill of Materials
Evaluating stages one lane at a time hides two structural properties of real agentic and multimodal systems:
- Stage fan-out dominates unit price: a web-search stage called 8 times and a layout-extraction stage parsing 15 pages per report frequently cost more per end-user run than the final synthesis model called once.
- Sequential reliability compounds: across critical-path stages with recorded
successRatePctevaluations, compound pass rate is the product of stage pass rates, raisingcostPerSuccessfulRunUsdabove raw listcostPerRunUsd.
evaluatePipeline (platform/engine/pipeline.mjs) computes:
- Per-stage
unitCostUsd,invocationsPerRun,runCostUsd,monthlyCostUsd,costSharePct, compliance/capability eligibility, quality, and p90 latency. - Spend rollups
breakdown.byClassandbreakdown.byProvider, plus explicitbreakdown.provenancesplitting Witness archive observations (witnessedRunCostUsd) from customer-entered private infrastructure rates (enteredRunCostUsd). - Pipeline bottlenecks (
dominantCostStage,dominantCostClass,dominantLatencyStage, andweakestQualityStage).
Runtime trace attribution (x-vitna-run)
When callers pass x-vitna-run (and optional x-vitna-agent) headers through vitna-runtime, every JSONL ledger line carries runId and agentId. summarizeLedgerForPipeline groups ledger rows by runId to measure empirical upstream invocationsPerRun per lane, local response cache hits (cacheHits), clean vs fallback-degraded vs failed runs (runCompletionRatePct), per-run call distributions, and per-lane p50/p90/p95 latencies.
When paired with evaluatePipeline, empirical runtime_ledger fan-out replaces static stage guesses and reports invocationDriftPct against declared fan-out.
Verified switches vs exploratory peer levers
evaluatePipeline keeps two strictly separated optimization buckets:
- `decisionReadySwitches`: stages where a candidate clears compliance terms, three-provenance capabilities,
qualityFloor,edgeCaseVeto, andeconomicThresholdPctinsidedecide(), returningoutcome: "switch". OnlydecisionReadySwitchesloweroptimization.optimizedCostPerRunUsd. - `exploratoryTests`: eligible candidates (including active Witness class peers discovered when
discoverPeers: true) that cost less at list price while still awaiting workloadsuccessRatePctevaluations or critical-failure clearance. Exploratory levers carry zero quality claim and emit exactaperture:candidate:addandaperture:evidence:addcommands.
Evaluate compound pipelines from the CLI with `aperture:pipeline`, for an authenticated organisation via `GET /api/v1/org/pipeline`, or statelesly via `POST /api/v1/pipeline` and MCP vitna_analyze_pipeline.
