Metrics methodology

How the headline numbers Azoth OS cites in marketing and product material are computed, where they come from, and — crucially — what each number is not. Use this page to verify any claim you see on the landing page, the pricing page, or in a sales call.

Confidence tags

Every metric in customer-facing material carries one of three tags:

• Measured verifiable — derived from production telemetry over a stated window, or from a deterministic benchmark run committed to the repo (reports/benchmark-report.json). Subject to normal sampling variance but reproducible from usage_traces, routing_decisions, and cache_entries, or from npm run benchmark.
• Modelled assumed — computed by simulation against the MODEL_MATRIX price list and a representative workload mix. Numbers depend on workload assumptions and are reproducible from the scripts in scripts/. "Simulated" is used interchangeably in older copy.
• Projected forward-looking — a number derived from a published pricing table multiplied by a typical-customer profile. Not a guarantee — provided to set expectations.

Marketing material must carry the corresponding tag when citing one of these metrics. Numbers without a tag should not appear in user-facing copy.

How to reproduce a number

1. Measured from production: run the corresponding query in__tests__/observability/ or the snapshot inlib/observability/snapshot.ts.
2. Measured from the benchmark: run npm run benchmark and consult reports/benchmark-report.{json,md}. The benchmark report is committed alongside any change that moves a number, so the historical diff is reviewable in git.
3. Modelled / Simulated: see the per-claim section below for the script + fixture pair that produces it.
4. Projected: see the inputs section of the matching claim — the projection is a single arithmetic step on top of a published price list, traceable in the doc.

Headline claims

"Up to 65% lower inference cost"

Modelled Computed by running the routing benchmark on a representative workload (1,000 requests across summarisation, classification, and Q&A) and comparing the dollar cost of the chosen route against the always-most-expensive-model baseline (gpt-4-turbo at the date of the snapshot). Inputs: the public Anthropic / OpenAI / Google price tables snapshotted inMODEL_MATRIX. The 65% is the workload mean; individual workloads vary from ~12% (already on a small model) to ~88% (long-tail summarisation on a latency-tolerant queue). See reports/benchmark-report.md for the exact per-task breakdown of the most recent run.

"P99 latency < 50 ms on semantic cache hits"

Measured Production telemetry from the rolling 30-day window on the cache.lookup span. Includes pgvector roundtrip, embedding comparison, and result rehydration. Excludes the cost of generating the original embedding when the lookup is on a cold prompt — that's quoted separately under "p99 cold-lookup latency" (~340 ms, also measured).

"99.95% uptime SLA on Team plan"

Projected Operational target backed by the underlying infrastructure SLAs (Vercel 99.99% on Pro, Supabase 99.95% on Pro). The 0.04% gap between Vercel's quoted SLA and ours covers our own deploy / migration windows. Service credits and incident definitions live in the /legal/sla page.

"38% lower token spend via prompt intelligence"

Modelled Result of the cognitive-compression benchmark on the same 1,000-request workload above. Compresses input prompts using L1 (filler-word removal + sentence dedup), L2 (symbolic §N encoding of recurring phrases), and L3 (causal-graph triples) and records the token delta. Quality is preserved > 92% (measured via Jaccard similarity on the model output before vs after compression). Numbers are deterministic and reproducible via npm run benchmark -- cce.

What these numbers are NOT

• Not averages across all Azoth customers — we don't aggregate customer workloads into a public number. Where production telemetry is involved the source is the Azoth-internal staging workload, not customer data.
• Not a guarantee. The benchmark is a reproducible reference point, not a contractual commitment. Your workload mix will be different.
• Not static. Provider price changes, new models, and refactors to the routing engine move the numbers. The benchmark is re-run on every release and the new figures land in reports/.

How to audit the source

The full repo-internal reference is docs/METRICS_METHODOLOGY.md (kept in sync with this page). The benchmark scripts that produce every "Modelled" and "Measured" number sit under benchmarks/. Any number on the landing page that doesn't trace back to one of those sources is a bug — please open an issue.