Gateway

Governance

Per-request controls run as pipeline stages before and after the engine call. All limits are enforced in-process by default; set storage.redis_url to share them across replicas (see Deployment).

Access keys

Each access_keys entry is one credential with its own governance:

access_keys:
  - ak: ak-demo-123
    product: demo            # product group (for product-level QPM)
    tenant: acme             # optional tenant membership
    qps: 100                 # per-key request rate
    daily_token_quota: 1000000
    tokens_per_minute: 600   # optional TPM window
    expires_at_epoch_secs: 1767225600  # optional; expired keys 403
    banned: false            # optional; banned keys 403
    model_quotas: {gpt-4o: 200000}     # optional per-model daily caps

Tenants

A tenant groups keys under shared governance: one pooled QPS bucket for all its keys, a model entitlement allowlist (unlisted models 403 and disappear from GET /v1/models), per-model daily-token quota defaults (each key metered separately against the same value; per-key model_quotas override), and an optional fallback_model — an over-quota request degrades to it (a typed-family request such as embeddings, rerank or decisions only when the fallback’s protocol serves its surface). With no usable fallback — none configured, one whose protocol cannot serve the surface, a request that already names the fallback, or one that engages or replays reasoning and so stays on its model — the request is refused with 400 service_quota_exceeded_exception (a degraded response echoes the requested model name; the ledger records both requested and served; a model’s own fallback_models chain for upstream failures is a separate mechanism, below). The per-key daily cap stays the hard backstop, and unconfigured (key, model) pairs never touch a counter.

Model fallback

A model may name fallback_models, tried in order when the request fails upstream — a vendor 5xx, 429, or a 401/402/403 credential or billing refusal, a connection failure, or no healthy account left for the model (503) — after the account-level failover within the model is exhausted, and only while no byte has reached the client (a failure after a stream has begun stays a failure). Gateway-side denials (quotas, rate limits, entitlement, bad requests) and gateway-internal errors never fall back, and a request carrying signed thinking stays pinned to its model (a signature only replays against the model that produced it). Each hop re-runs the pipeline for the next model: entitlement, model QPM, quota reservation and account selection run per hop against that hop’s model (a failed hop’s QPM permit is spent; its reservations are refunded). Tenant/key QPS and product QPM spend at most one permit per external request: the first hop that reaches them takes it and later hops reuse it, and a request served from the cache never reaches them. The per-model daily counter follows the hop’s target, including when that target degrades to the tenant fallback; a hop over its quota with no usable tenant fallback ends the chain with that refusal. A fallback the caller’s tenant is not entitled to is skipped, the response echoes the requested name, the ledger records both requested and served (served_model), the decision trail carries fallback: <from> -> <to>: <why>, and gateway_model_fallbacks_total{from, to} counts the hops. A chain does not recurse: only the requested model’s list applies. The chain covers the REST surfaces and batch items; a realtime session never switches models. It is distinct from a tenant’s fallback_model above, which is a single swap the gateway makes on its own over-quota or moderation decision. A chain naming the model itself, an unknown model or a duplicate entry is rejected at load.

models:
  - {name: gpt-4o, protocol: openai-chat, fallback_models: [gpt-4o-mini, claude-haiku]}

Limits

Limit Scope Config
QPS per access key access_keys[].qps
QPS pooled per tenant tenants[].qps
Daily tokens per access key access_keys[].daily_token_quota (fleet/Redis: rolls at UTC midnight; single-node in-memory: a ~daily background reset)
Daily tokens per (key, model) tenants[].model_quotas default, access_keys[].model_quotas override (over it: the tenant fallback_model, else refused)
Daily tokens per end user tenants[].user_daily_token_quota (soft)
Daily cost pooled per tenant tenants[].daily_cost_quota_micros (micro-dollars of charged price; soft)
Daily cost per access key tenants[].key_daily_cost_quota_micros (soft)
Daily cost per end user tenants[].user_daily_cost_quota_micros (soft), per-user override via /admin/tenants/{tenant}/users/{user}/budget
Monthly cost pooled per tenant / per key / per end user tenants[].monthly_cost_quota_micros, key_monthly_cost_quota_micros, user_monthly_cost_quota_micros (UTC calendar month; soft; monthly_cost_rollover carries the unspent remainder); the per-user cap takes a per-user override
TPM per access key access_keys[].tokens_per_minute
QPM per model models[].qpm
QPM per product products[].qpm

Exceeding a QPS, QPM or TPM limit returns 429; the daily-token, per-model (with no usable fallback), cost-budget and per-user caps return 400 service_quota_exceeded_exception. QPS uses a smooth GCRA limiter in-process (in Redis: a fixed 1s window for qps ≥ 1, a 1/qps-second window below); the token/window counters are fixed windows. When Redis is configured and unreachable, limits fail open (requests pass) and a warning is logged — a persistent outage never silently wedges the gateway.

A batch item refused by one of these limits waits 1s, 2s and 4s between three retries before it fails.

Key QPS is checked before tenant QPS in the model pipeline, realtime turns, MCP requests and video polls. A request denied by tenant QPS has already spent a key-QPS permit; a request denied by key QPS spends no tenant permit.

abuse.tiers counts a key’s REST rejections for its own limits only: key QPS and key TPM. Tenant QPS, product QPM and model QPM are pooled; their rejections answer the same 429 and never count, so a sibling’s traffic cannot suspend a key. A key that stays inside its own limits is never counted — including one whose qps is set above its tenant’s, which can drain the pool without ever reaching a limit of its own: keep a key’s qps at or under the tenant’s. Batch items, realtime turns, MCP requests and video polls do not count.

Daily-token and TPM admission reserve then settle: on admission a cheap estimate (prompt heuristic + requested max_tokens) is reserved atomically, so concurrent in-flight requests count against the budget instead of all passing a stale check and jointly overshooting. Billing settles the reservation to actual usage; a failed request refunds it. The daily reserve settles on its admission day; the TPM reserve settles only into the minute window that admitted it — a request that outlives its window leaves the next window’s reservations alone. Charged price is the model’s list price, or a tenant’s model_prices override; when an account declares cost_*_price the ledger also records the vendor cost, so margin is queryable via /admin/usage. A protocol: decisions model without a tenant override charges the vendor’s reported usage.cost instead of the list price, and the cost budgets count that charge. Surfaces that meter no tokens bill per unit instead — TTS characters, transcription seconds, rerank search units, web searches (one unit per query), generated images, generated video seconds — at the model’s unit_price_micros (see Configuration); the unit count rides on the ledger row and the usage aggregates as billed_units. An async video bills when its poll first sees done: the submit row carries 0 units, the completion row (its request_id is the video id) the clip’s seconds at the price quoted at submit and, when the vendor reports one (xAI’s cost_in_usd_ticks), the exact vendor cost instead of the account’s unit price; that completion row also accrues the cost budgets of the submitting key — of the polling key when the submitter no longer exists (the tenant and user budgets are the same either way). Audio tokens and 1-hour cache writes take their own token_rate weights, a long_context tier re-prices calls past a prompt size, and batch_discount scales what /v1/batches items cost.

A streaming response that breaks after delivery has begun (client disconnect or upstream failure) is billed for what was delivered: the vendor’s usage frame never arrives, so the token count is estimated from the request and the delivered text. A disconnect before any bytes are sent bills nothing.

MCP tool access

/mcp/{server} proxies the Model Context Protocol servers declared under mcp_servers. A key reaches only the servers named in its mcp_servers entitlement, and mcp_tools narrows a server to an allowlist: tools/list is filtered to it before the client sees the catalog, and a tools/call for any other tool is answered with a JSON-RPC error inside the gateway. A server the key is not entitled to answers 404 like an unknown one. Every tool call and every denial lands in the security-event stream (surface = mcp, rule = mcp:<server>, action = call:<tool> or deny:<tool>, attributed to the key’s owner) before the call is forwarded, and gateway_mcp_requests_total{server, method, result} counts the traffic. An Mcp-Session-Id is bound to the key that first received it, so one customer cannot resume or end another’s session; the key’s QPS and the tenant’s pooled QPS apply, and GET listen streams count against max_live_streams_per_key (a tenant under security.moderate may not open one). Keys created through the admin API carry no MCP entitlement; the servers’ own credentials stay in the gateway’s environment (mcp_servers[].api_key_env), so an agent never holds them.

A server that takes OAuth 2.0 instead of a static bearer declares mcp_servers[].oauth: the gateway runs the client-credentials grant (or the refresh-token grant from a seeded refresh token, keeping a rotated one in memory) against token_url, caches the access token per server until 30 s before its expires_in (half its lifetime for a token shorter than a minute; a reply without expires_in counts as an hour), and on a 401 from the server fetches a fresh token and retries the call once. Tokens are dropped on config reload. The client secret and refresh-token seed are read from the environment at fetch time.

A tenant whose security.moderate is on has every served tools/call, resources/read and prompts/get result reviewed by the moderator behind moderation: — the same review the chat and realtime surfaces apply to inbound text, here over every string value under result, joined by newlines, except the base64 blob and data of an image or audio block or a blob resource, which pass untouched. The reply is buffered for the review, up to the server’s max_reply_bytes. A mask rewrites the text in place ([MASKED]); a denial, a degrade verdict, a mask that matches nothing, or a reply the gateway cannot parse replaces the whole reply with one JSON-RPC error -32001 carrying the reason, so nothing unreviewed leaves; a moderator failure follows moderation_fail_open. The GET listen stream carries server notifications and is not reviewed; for a reviewed tenant Last-Event-ID is not forwarded, so a result cannot replay through it. Both outcomes are mcp security events with rule = moderation and action = mask (hits = spans) or block. Tenants without moderate stream results as before.

mcp_servers:
  - {name: tools, endpoint: http://tools.internal:3001/mcp, api_key_env: TOOLS_TOKEN}
access_keys:
  - {ak: ak-agent, product: p, qps: 20, daily_token_quota: 1000000,
     mcp_servers: [tools], mcp_tools: {tools: [search, read_file]}}

Request cache

A model with cache_ttl_seconds set caches non-streaming responses for that TTL (bounded, moka-backed). A cache hit is free: it short-circuits account selection, the engine call, and billing/quota — a hit consumes no quota and writes no ledger record. Offline batch items bypass the cache entirely (read and write) so per-item billing stays accurate.

models:
  - name: cached-mini
    protocol: openai-chat
    cache_ttl_seconds: 60

Content safety

security.dlp_redact redacts emails and phone numbers from inbound content (chat messages, the Responses body, and the family typed params) and from the outbound message — it guards against accidental leaks and matches within each text field, so a value a client cuts across two content parts or spells out is not caught; security.blocklist rejects requests containing listed terms with a content_filter finish (not billed).

security:
  dlp_redact: true
  blocklist: ["badword"]

Outbound redaction needs the whole message (a masked span may straddle two SSE deltas), so whenever a tenant redacts output — dlp_redact or detect_secrets — a streaming response is buffered and the redacted text replayed rather than forwarded token-by-token — DLP trades incremental delivery for a guarantee that no unmasked text reaches the client. Billing is settled after that replay: a disconnect before the first redacted chunk refunds the reservation, while a partial replay bills only an estimate of the delivered text. Turn both dlp_redact and detect_secrets off to keep incremental delivery; note the embedded demo config ships with dlp_redact on.

security.moderate routes the inbound text through the external moderator named by the top-level moderation: section — today AWS Bedrock Guardrails (kind: bedrock_guardrail, ApplyGuardrail on the configured guardrail id and version, authenticated with a Bedrock API key). A BLOCKED assessment (denied topic, custom or managed word, content filter, blocked PII) denies the request with the policy names in the reason; ANONYMIZED PII entities mask their matches in place before the prompt leaves the gateway (Bedrock anonymizes only under source: OUTPUT; the default INPUT blocks); no intervention allows. A deny, a degrade and a non-empty mask are recorded as moderation security events. A moderator error follows the tenant’s moderation_fail_open posture (default: deny). The moderator is built at startup, so changing moderation: needs a restart.

Per-user attribution and billing

Every ledger row carries a user_id, request_id, and created_at_epoch_secs. The effective user is the key’s owner (one key = one user, the enterprise model) if set, else request metadata — the x-gw-user header, OpenAI’s user field, or Anthropic’s metadata.user_id. owner is authoritative; the metadata hint is only trusted for shared keys. This holds on every surface — REST, realtime (the x-gw-user hint is captured at WS connect), and batch (each item’s user, or the submitter’s x-gw-user, is persisted with the item so a fleet drainer still attributes and budgets it). GET /admin/usage/users?user=&since=&until= returns per-(user, model) cost over a billing period (add format=csv for export). TenantConf.user_daily_token_quota sets a soft per-user daily cap — enforced with a hard 400 service_quota_exceeded_exception once exceeded; “soft” means check-then-consume, so concurrent in-flight turns can overshoot it by one before the counter accrues. The per-(key, model) quota is soft the same way: it is a routing trigger (the fallback-model degrade), while the reserved per-key daily quota hard-caps spend. The daily cost budgets — daily_cost_quota_micros pooled over the tenant’s keys, key_daily_cost_quota_micros per key, user_daily_cost_quota_micros per end user — are soft caps of the same kind over the ledger’s charged cost_micros (list price, the tenant’s model_prices, or a decisions reply’s usage.cost, so unit-priced surfaces count too); they apply on every surface, realtime and batch included, and a response-cache hit, which bills nothing, counts nothing. The monthly counterparts — monthly_cost_quota_micros, key_monthly_cost_quota_micros, user_monthly_cost_quota_micros — meter the same charged cost over the UTC calendar month and can be set alongside the daily caps. With monthly_cost_rollover: true a month’s cap grows by the current cap minus the previous month’s recorded spend, at most one further cap — a month with no recorded spend (the first month, or a flushed Redis) carries a whole cap; the carry never compounds, and a cap changed between months applies to the carry retroactively. Monthly counters live outside the daily reset: in Redis under gw:counter:m:<yyyymm>:… with a 62-day TTL, in-process swept at the daily reset down to the current and previous month. Rollover costs one extra counter read per configured monthly scope at admission and one at settlement; without it the monthly check is the same single read as the daily one. Per-user overrides. The tenant’s user_daily_cost_quota_micros, user_monthly_cost_quota_micros and user_daily_token_quota are defaults for every user. PUT /admin/tenants/{tenant}/users/{user}/budget gives one user their own value for any of the three: a cap, "unlimited", or null to inherit (see API). The counter stays the user’s, summed over all of their keys, so the cap holds however many keys the user has. An override replaces only the per-user scope; the tenant pool and per-key caps still apply, and the tightest cap wins. A tenant with per-key $10/day, a $50/day pool and a user at $5/day stops that user at $5. "unlimited" lifts only the per-user cap: that user still stops at the per-key cap and draws on the pool, and their spend keeps counting, so a cap set later starts from what they already spent. A user cap applies even when the tenant sets no per-user default, but a per-user counter only runs while some cap or "unlimited" applies to that user: set a tenant default (overriding the exceptions) so a cap added mid-month sees the month’s spend. Overrides live in Postgres with storage.postgres_url (in-process, lost on restart, without it) and are cached per instance; a request with an attributed user costs one in-process lookup at admission, carried to settlement, and a request without one costs none. If that lookup fails (cache miss and Postgres unreachable), the request is refused with 503 rather than run uncapped: falling back to the tenant default would loosen a user capped below it. Per-user caps follow attribution, so a key without an owner lets the caller pick the user it is charged to; with require_key_owner: true a tenant refuses to create such keys, and a key without an owner that predates the flag is refused at authentication (403).

Keys without a tenant take a declared default tenant’s budgets. Reaching any budget raises a budget_exhausted alert on the webhook — subject tenant:<name>, key:<ak_id> or user:<tenant>/<id>, detail the window’s total against the cap — muted per alerts.dedup_seconds while it holds.

A background task folds completed ledger minutes into durable per-(minute, tenant, user, model) rollup buckets. Each pass recomputes from the rollup watermark or the trailing 20-minute window, whichever reaches further back — the first run rolls a pre-existing ledger whole, and a stalled task catches up on its own. Buckets only ever grow (a recompute over a partially pruned ledger keeps the more complete aggregate), and on Postgres a fleet elects one replica per pass via an advisory lock. The in-process store keeps buckets for 62 days and at most one million of them, dropping the oldest minutes first; Postgres keeps them until an operator prunes the table. Usage queries are served from those buckets plus the raw ledger tail, so per-user cost stays correct after storage.ledger_max_rows prunes old billing rows. since/until bounds are minute-aligned, so a repeated query returns the same result whether a minute is served from its bucket or still raw.

Enterprise content policy

security: is global by default; a tenant may override it whole with tenants[].security. Beyond the blocklist and DLP, a policy can carry blocklist_action (block denies, flag records, shadow trials a rule), named regex_rules (each with its own action), and detect_secrets (masks API keys / credentials). Every rule that fires is recorded — without the offending text — to the security-event stream, queryable at GET /admin/audit/events. Set moderate: true to route inbound text through an external moderator wired into the handler (moderation_fail_open picks the posture on a moderator error).

The same policy applies on every surface, realtime included: a /v1/realtime WebSocket runs the blocklist, regex rules, DLP redaction, and the external moderator on inbound frames, so it is not a bypass. Every hit is audited: inbound block/flag/moderation and DLP redactions are recorded per frame; outbound DLP redactions (which stream token-by-token) are summed across the turn and recorded as one event at the turn boundary — the redaction still applies to every frame, only its audit is aggregated (a store write per token would be too hot).

Signed Anthropic thinking blocks get the same coverage without breaking their signatures: inbound scans read every field of a signed block — opaque signature/data included, so no field is a smuggling channel — but rewrites skip them. A moderation mask that would land inside signed prose denies the request instead, and a response whose signed prose trips the blocklist or DLP is served with the protected blocks stripped rather than failed.

Audit trails

This site is open source. Improve this page.