Platform

Semantic Cache

Semantically equivalent queries — even with different phrasing — return cached responses in milliseconds. No model call, no cost, no latency. Average elimination rate: 30%.

How it works

Incoming prompts are embedded and compared against a vector index of past queries. If cosine similarity exceeds your configured threshold, the cached response is returned immediately. The threshold balances recall (avoid stale answers) against elimination rate (save more tokens).

Usage

Cache is enabled by default. The response header X-Azoth-Cache tells you whether a call was served from cache.

// Disable cache for a specific call (e.g. real-time data):
const response = await openai.chat.completions.create({
  model: 'gpt-4o',
  messages,
  azoth_cache: false,
});

// Response header on a cache hit:
// X-Azoth-Cache: hit
// X-Azoth-Cache-Latency: 3ms

Cache key scope

By default, the cache is scoped to your workspace. You can narrow it to an agent or widen it to cross-workspace via the dashboard under Settings → Semantic Cache.