Get started

CI evaluation evidence

Source

Record evaluation measurements from CI pipelines and read back historical measurements for your organisation's workloads, with built-in normalization for Promptfoo, Braintrust and UK AISI Inspect AI outputs.

text
GET /api/v1/org/evidence
POST /api/v1/org/evidence
text
Authorization: Bearer vitna_push_...
bash
curl -H "authorization: Bearer vitna_push_..." \
  "https://vitna.ai/api/v1/org/evidence?workload=deep-research&kind=eval"

curl -X POST https://vitna.ai/api/v1/org/evidence \
  -H "authorization: Bearer vitna_push_..." \
  -H "content-type: application/json" \
  -d '{
    "adapter": "promptfoo",
    "workload": "deep-research",
    "sku": "openai/gpt-5.6-luna",
    "label": "pr-412-gpt-5.6-luna",
    "sourceRef": "https://github.com/convexity/app/actions/runs/91823",
    "report": {
      "timestamp": "2026-09-14T18:22:00Z",
      "results": {
        "stats": { "successes": 193, "failures": 7, "errors": 0 }
      }
    }
  }'

Adapters and append-only rules

Reads (GET) accept either a platform or push key and filter by workload, sku, kind (eval, production or market) and limit (1 to 200, default 50). Writes (POST) require a platform key (vitna_key_...).

Supported adapter values:

  • vitna (default): native JSON carrying workload, sku (or skuLabel), label, measuredAt (YYYY-MM-DD), n, sourceRef and metrics (successRatePct, latencyP90Ms, edgeCaseFailures).
  • promptfoo: normalizes npx promptfoo eval -o output.json rows or stats into n, successRatePct, sorted p90 latency (latencyP90Ms) and failed test case identifiers (edgeCaseFailures).
  • braintrust: normalizes Braintrust Eval() experiment summaries or row dumps (scoreKey selects a named scorer when an experiment tracks several).
  • inspect: normalizes UK AISI inspect_ai JSON logs (inspect eval --log-format json), extracting sample count, accuracy, p90 working time in milliseconds and failed sample IDs.

Evidence is append-only and unique on (workload_id, label). Re-submitting an existing label answers 409 duplicate_evidence rather than overwriting the measurement a prior decision cited.

Response shape

json
{
  "orgId": "convexity",
  "supportedAdapters": ["vitna", "promptfoo", "braintrust", "inspect"],
  "measurements": [
    {
      "id": 42,
      "workloadId": "convexity/deep-research",
      "skuId": "openai/gpt-5.6-luna",
      "isPrivateSku": false,
      "skuLabel": null,
      "label": "pr-412-gpt-5.6-luna",
      "kind": "eval",
      "measuredAt": "2026-09-14",
      "n": 200,
      "metrics": {
        "successRatePct": 96.5,
        "latencyP90Ms": 1420
      },
      "sourceRef": "https://github.com/convexity/app/actions/runs/91823",
      "notes": null,
      "recordedAt": "2026-09-14T18:22:10.000Z"
    }
  ],
  "meta": {
    "generatedAt": "2026-09-14T18:22:10.000Z",
    "count": 1
  }
}