Start free →

Enrichments

Metrics methodology

How the figures Azoth OS cites in marketing and product material are computed, where they come from, and — above all — what each one is not. Use this page to check any figure you read on the site or hear in a sales call.

Confidence tags

Measured
Derived from production telemetry over a stated window, or from a deterministic benchmark committed to the repository (reports/benchmark-report.json, npm run benchmark).
Modelled
Computed by simulation against a price list and a representative workload; it depends on those assumptions. Older copy may say “simulated”.
Projected
A forward-looking figure: a published price list applied to a typical-customer profile. Not a guarantee.

A figure in customer-facing material must carry its tag. A figure without a tag should not appear.

The figures, and where each comes from

Cost per call · Measured, per call
From the tokens the provider counted and the price in force in the model catalog; returned in the X-Azoth-Cost-Usd header and sealed with the decision. Your own figures are in your dashboard.
Energy per call · Modelled, per call
An estimate under the GreenOps methodology, returned in the X-Azoth-Energy-Wh header. Absolute grams are advisory; the avoided fraction is the reportable figure.
Reused answers · Measured on the benchmark harness
The cache hit rate of npm run benchmark on its offline scenarios (reports/benchmark-report.md). Your own rate depends on your workload.
Latency Azoth adds · Not published
No figure yet: a p50 and p95 measurement is on the roadmap.
Availability · On the status page
The observed state is on the status page. No uptime figure is published here; where a contract sets an SLA, it states its own.

The quality the router weighs · Measured, per workspace

  • The quality term of routing is the model’s share of the workspace’s own reference cases that passed, over the last 90 days, once at least 10 cases were judged.
  • Otherwise it is 0.5, neutral: what the workspace has not measured weighs the same for every model, and the choice falls to latency and cost.
  • A quality floor eliminates a model only when its measured pass rate is below it, never an unmeasured one. The cost term reads the model catalog’s prices.
  • Each routing decision says which it was: “N% of M reference cases passed”, or “nothing measured yet”. It is a statement about this workspace’s cases, on its own criteria, not about a model in general.

What these figures are not

  • Not a promise about your bill: say “up to” or “median”, and cite the tag.
  • Not an average over Azoth’s customers: no customer workload is aggregated into a public figure.
  • Not fixed: provider prices, new models and changes to the engines move them, and the benchmark runs again with each release.
  • Not an uptime promise: the observed state is on the status page.

The source

The repository reference is docs/METRICS_METHODOLOGY.md, kept in step with this page. A figure on the site that does not trace back to it is a bug — please tell us.