Gateway

Configuration

One YAML file configures the gateway. Resolution order:

1. GW_CONFIG=<path> — explicit config file
2. otherwise the embedded default (the repo’s conf/gateway.yaml)
3. with storage.postgres_url set, the Postgres config store’s newest version replaces the file at boot and on every reload; the file (or the embedded default) only seeds an empty store

GW_HOST / GW_PORT override listen.host / listen.port at runtime (the container image sets GW_HOST=0.0.0.0). GW_CONTENT_KEY (64 hex chars = 32 bytes) is the deployment key that seals retained content at rest; without it, full retention refuses to store raw text and falls back to redacted.

Sections

listen

listen:
  host: 127.0.0.1
  port: 8080
  max_request_bytes: 33554432   # optional: largest request body buffered, read at startup (default 32 MiB)

storage — durable records and fleet backends

storage:
  sqlite_path: /var/lib/gw/store.db   # empty/absent = in-memory
  postgres_url: postgres://gw:secret@db.internal/gw   # fleet-shared backend
  redis_url: redis://cache.internal:6379              # shared counters + health
  shared_cache: false                 # also share the response cache in Redis (needs redis_url)
  ledger_max_rows: 100000             # prune oldest billing rows past the cap; 0 = unlimited
  postgres_max_connections: 10        # store and key-store connection-pool size; 0 = the built-in defaults (10 store, 5 key store)

The billing ledger, uploaded files, and batch jobs live here. In-memory by default (lost on restart; batch jobs are kept 30 days); a SQLite path makes them durable on one node. postgres_url turns Postgres into the fleet backend: the source of truth for config (versioned documents + a change feed every instance follows), the shared access-key table, the shared ledger/files/batches store, and a distributed batch queue (any instance claims and runs submitted batches). redis_url shares rate/quota/TPM counters, monthly cost counters, account-health cooldowns and the per-model availability counts behind /admin/models/status across instances; shared_cache: true additionally moves the request cache into Redis so a hit on one instance serves the fleet (off = each instance caches in-process, a miss just recomputes). ledger_max_rows is not a hard cap: pruning spares rows not yet folded into the usage rollup, so the table can briefly exceed the cap under rollup lag.

access_keys — client authentication and per-key governance

access_keys:
  - ak: ak-demo-123          # bearer / x-api-key value clients send, or its
                             # `sha256:` ak_id to keep the key out of the config
    product: demo            # product group (for product-level QPM)
    tenant: acme             # optional; absent = the unrestricted `default` tenant
    owner: alice             # optional; binds the key to one end user (authoritative
                             # for per-user attribution; a shared key omits it and
                             # falls back to the request's `x-gw-user` / `user`)
    qps: 100                 # per-key request rate
    daily_token_quota: 1000000
    mcp_servers: [tools]         # MCP servers this key may reach (default none)
    mcp_tools: {tools: [search]} # per-server tool allowlist (absent = every tool)
    tokens_per_minute: 600   # optional TPM window limit
    expires_at_epoch_secs: 1767225600  # optional expiry (403 after)
    banned: false            # optional; a banned key 403s but stays listed
    model_quotas:            # optional per-model daily caps (override tenant defaults)
      gpt-4o: 200000

tenants — pooled limits, entitlement, quota defaults

tenants:
  - name: acme
    qps: 50                  # pooled across ALL of acme's keys
    models: [gpt-4o, gpt-4o-mini]   # entitlement allowlist; absent = every model
    model_quotas:            # per-model daily-token defaults, applied per key
      gpt-4o: 100000
    fallback_model: gpt-4o-mini     # over-quota requests degrade here (a typed surface only to a model of its protocol); without a usable one they are refused
    admin_token_env: ACME_ADMIN_TOKEN   # optional tenant-scoped /admin token
    model_prices:            # optional per-model charged-price override for this tenant
      gpt-4o: {input_price_per_1k_micros: 5000, output_price_per_1k_micros: 20000}
      tts-1: {unit_price_micros: 20}
    user_daily_token_quota: 100000  # optional soft per-end-user daily cap
    daily_cost_quota_micros: 50000000      # optional: $50/day pooled across acme's keys
    key_daily_cost_quota_micros: 10000000  # optional: $10/day per key
    user_daily_cost_quota_micros: 2000000  # optional: $2/day per end user
    monthly_cost_quota_micros: 1000000000  # optional: $1000/calendar month (UTC) pooled
    key_monthly_cost_quota_micros: 200000000   # optional: $200/month per key
    user_monthly_cost_quota_micros: 40000000   # optional: $40/month per end user
    monthly_cost_rollover: true            # optional: last month's unspent budget carries over (at most one month's cap)
    require_key_owner: true                # optional: refuse keys without an owner, so per-user caps hold
    security:                # optional; overrides the global `security:` WHOLE for this tenant
      blocklist: ["forbidden"]
      blocklist_action: flag        # block | flag | shadow
      detect_secrets: true
      regex_rules:
        - {name: ssn, pattern: '\d{3}-\d{2}-\d{4}', action: block}
    retention:               # optional prompt/response retention; absent = retain nothing
      content: redacted      # none | redacted | full  (full needs GW_CONTENT_KEY)
      days: 30               # purge after N days; 0 = keep until manually purged

Keys without a tenant join the implicit default tenant (no pooled limits, entitled to every model), so a flat config keeps working unchanged. The model catalog (GET /v1/models) filters to the caller’s entitlement.

user_daily_token_quota, the *_cost_quota_micros budgets, security, and retention are enterprise controls detailed in Governance; security replaces the global policy outright when present (it is not merged field-by-field).

models — public model names and dispatch

models:
  - name: gpt-4o                     # name clients request
    protocol: openai-chat            # wire protocol (or set `provider:` instead; a video or search model must also set `provider:`)
    input_price_per_1k_micros: 2500  # billing rates (micros per 1k tokens)
    output_price_per_1k_micros: 10000
    unit_price_micros: 0             # per non-token unit: TTS character, transcription second, rerank search unit, web search query
    qpm: 60                          # optional model-level rate limit
    cache_ttl_seconds: 60            # optional request-level response cache
    token_rate:                      # optional per-component billing weights
      read_cache: 0.1                #   cache reads at 10% of the input price
      write_cache: 1.25              #   (prompt/completion/reasoning default 1.0)
      write_cache_1h: 2.0            #   1-hour cache writes (default = write_cache)
      audio_prompt: 16.67            #   audio input tokens (default = prompt weight)
      audio_completion: 8.33         #   audio output tokens (default = completion weight)
    long_context: {threshold_tokens: 200000, prompt_weight: 2.0, completion_weight: 1.5}  # optional tier past a prompt size
    batch_discount: 0.5              # optional: /v1/batches items at this fraction of the price; must be finite and in (0.0, 1.0]
    prompt_cache: true               # anthropic-messages only: prompt-cache breakpoints
    fallback_models: [gpt-4o-mini]   # optional: tried in order on an upstream 5xx / 429 / 401-403 refusal / connection failure / no healthy account (rejected at load when unknown, self or duplicate)
    variants:                        # optional weighted canary split, sticky per user
      - {model: gpt-4o, weight: 90}  #   self-reference keeps a share here
      - {model: gpt-4o-next, weight: 10}

token_rate weights scale cost and quota consumption per token component; the ledger’s prompt/completion columns stay vendor-reported, while total_tokens is the weighted platform total. Audio tokens (realtime, audio chat: the vendor’s audio_tokens details), 1-hour cache writes (Anthropic’s cache_creation.ephemeral_1h_input_tokens) are subsets of prompt/completion and cache-write with their own weights (audio tokens are attributed to the fresh, non-cached prompt side — the flat OpenAI usage shape does not say whether a cached token was audio); long_context re-scales both billable sides once the prompt crosses the threshold (Anthropic’s >200k tier); batch_discount multiplies the charged and vendor cost of items served through /v1/batches. unit_price_micros prices what the surfaces without token usage meter — a tts model’s input characters, an stt model’s audio seconds (the vendor’s usage.seconds / duration, else the uploaded WAV/MP3’s own play length; fractions rounded up), a rerank model’s search_units, an image model’s images, a video model’s generated seconds, a search model’s queries (one per call) — and adds to cost_micros next to any token cost; the count lands in the ledger’s billed_units and the /admin/usage aggregates. An account’s cost_unit_price_micros is the vendor side of the same unit, for margin. prompt_cache (anthropic-messages models) marks the system prompt and the latest user turn as Anthropic prompt-cache breakpoints, so each turn of a conversation re-reads its prefix at the cache-read rate; it is per model and off by default because a long one-shot prompt would pay the cache-write premium for nothing. A /v1/messages client that sends system as blocks with its own cache_control (including ttl: 1h) keeps them as sent; the gateway only marks the last system block when that block carries no breakpoint of its own. A /v1/chat/completions system message whose content parts carry cache_control keeps them on an Anthropic-wire model the same way. On an openai-chat model the client’s own breakpoints on system blocks, message blocks and tool_result ride through as OpenAI content parts, which OpenRouter honors for Anthropic models; a breakpoint on a tool definition does not (Gemini’s compatible endpoint rejects the field), and a later breakpoint covers the tools anyway. variants splits a public name across other declared same-protocol models (one level): entitlement and the per-(AK, model) daily counter judge the public name, billing prices the served variant, and the response echoes the requested name. Selection hashes the effective user, so a user sticks to one backend across the fleet; a realtime session picks its variant once at the handshake and pins it for the whole session.

providers — first-class provider presets

providers:
  - name: openai
    kind: openai              # openai | anthropic | gemini | deepseek | openrouter | moonshot | xai | siliconflow
    api_key_env: OPENAI_API_KEY
    # endpoint / timeout_seconds / connect_retries / retry_status / secret_key_env may be
    # set here too and are inherited by every account naming this provider
    # for whatever the account leaves unset
models:
  - name: gpt-4o
    provider: openai          # fills the protocol with the kind's default
                              # and pins the model to that provider's accounts

A provider entry expands into an upstream account with the kind’s preset base URL (overridable via endpoint:, e.g. for OpenAI-compatible vendors) and served wire types; an explicit account with the same name wins.

accounts — upstream credential slots

accounts:
  - name: openai-main        # unique across accounts; a duplicate is rejected at load
    provider: openai
    priority: 1                # lower = preferred
    tier: ptu                  # ptu (provisioned, preferred) | paygo (default)
    protocols: ["openai-chat", "embeddings"]
    endpoint: ""               # empty → mock transport, or the provider's endpoint when
                               # `provider` names a declared one; real base URL → real
                               # upstream; explicit "mock://…" → stays on the mock
    timeout_seconds: 60        # per-attempt timeout (default 60): bounds a non-streaming
                               # attempt whole, grown to the request's max_tokens at 128k
                               # tokens/hour (at most 10 min); a streaming one gets it on
                               # the headers and then per gap between chunks
    connect_retries: 1         # connect-phase retries; also bounds retry_status replays
    retry_status: []           # statuses this vendor issues BEFORE the model runs, so a
                               # replay cannot double-bill — e.g. [429, 502] for a relay
                               # whose 502 means its own upstream call failed. 4xx/5xx
                               # only; unset inherits the provider's list, [] never
                               # replays (the default). Replays honour Retry-After capped
                               # at 30s per attempt on top of each attempt's timeout, so
                               # worst case ≈ (retries+1)×timeout + retries×30s; statuses
                               # that outlive the budget still count against account
                               # health and can fail over to another account
    api_key_env: ""            # env var name holding the API key (never the key itself)
    secret_key_env: ""         # AWS only: env var of the secret key (api_key_env = access key id)
    cost_input_price_per_1k_micros: 100   # optional: what this vendor charges us (margin accounting)
    cost_unit_price_micros: 0             #   and per non-token unit (see the model's unit_price_micros)
    cost_output_price_per_1k_micros: 400

Secrets never live in config files: api_key_env names an environment variable that is read per request. The optional cost_*_price fields record what the vendor charges, so the ledger carries vendor_cost_micros alongside the charged cost_micros and margin is queryable per tenant/model via GET /admin/usage. A vendor that prices the call itself overrides those fields: OpenRouter’s usage.cost in USD and xAI’s usage.cost_in_usd_ticks (1e-10 USD a tick) are recorded as the exact vendor cost on every wire that carries them. On a protocol: decisions model the reported usage.cost is also the charged cost_micros, unless the tenant sets a model_prices override for the model.

security, stability, products

security:                      # global default; a tenant may override it whole
  dlp_redact: true             # redact emails/phone numbers, both directions
  detect_secrets: true         # also mask API keys / credentials in inbound text
  blocklist: ["badword"]       # reject/flag requests containing listed terms
  blocklist_action: block      # block (deny) | flag (record) | shadow (trial a rule)
  regex_rules:                 # named recognizers, each with its own action
    - {name: ssn, pattern: '\d{3}-\d{2}-\d{4}', action: block}
  moderate: false              # route inbound text through the wired external moderator
                               # (its verdicts: allow / mask spans / degrade to the
                               # tenant fallback model / deny)
  moderation_fail_open: false  # on a moderator error: admit (true) or deny (false)

moderation:                    # the external moderator behind security.moderate; omit = allow-all
  kind: bedrock_guardrail      # AWS Bedrock Guardrails ApplyGuardrail
  endpoint: https://bedrock-runtime.us-east-1.amazonaws.com
  api_key_env: AWS_BEARER_TOKEN_BEDROCK   # Bedrock API key, sent as a bearer token
  guardrail_id: your-guardrail-id
  guardrail_version: "1"       # default DRAFT
  source: INPUT                # INPUT (default) | OUTPUT — Bedrock anonymizes PII only under OUTPUT
  timeout_seconds: 10

stability:
  failure_threshold: 3         # consecutive failures (5xx, timeouts, a 401/402/403 refusal) before an account cools down
  cooldown_seconds: 30
  availability_window_minutes: 5   # /admin/models/status judgment window (max 60)
  unstable_error_rate: 0.1         # window error rate that reports `unstable`
  unavailable_error_rate: 0.5      # ... and `unavailable`
  availability_min_samples: 20     # fewer samples than this reports `no_data`
  latency_routing: false           # rank same-priority accounts by observed call latency instead of round-robin

products:
  - name: myproduct
    qpm: 120                   # product-level request rate

abuse:                         # automatic suspension; omit = off; counts a key's REST 429s for its own qps/tokens_per_minute, never a pooled limit or a batch item
  tiers:                       # highest tier at or under the day's reject count wins
    - {rejects: 20, suspend_hours: 2}
    - {rejects: 30, suspend_hours: 24}

mcp_servers:                   # Model Context Protocol servers proxied at /mcp/{name}
  - name: tools
    endpoint: http://tools.internal:3001/mcp   # the server's Streamable HTTP endpoint
    api_key_env: TOOLS_TOKEN     # optional bearer token for the server, from the env
    timeout_seconds: 60
    max_reply_bytes: 16777216    # largest reply buffered for allowlist filtering or result review; must be > 0
  - name: crm
    endpoint: https://mcp.crm.example/mcp
    oauth:                       # exclusive with api_key_env: the gateway fetches the bearer itself
      token_url: https://auth.crm.example/oauth/token
      client_id: gateway
      client_secret_env: CRM_CLIENT_SECRET   # empty = a public client
      grant: client_credentials  # client_credentials (default) | refresh_token
      refresh_token_env: CRM_REFRESH_TOKEN   # seed for grant: refresh_token; a rotated token replaces it in memory
      scope: mcp:tools           # optional

alerts:                        # outbound webhook; omit = off
  webhook_url_env: GW_ALERT_WEBHOOK   # env var naming the URL (secrets stay out of config)
  dedup_seconds: 300           # mute repeats of the same (kind, subject)
                               # kinds: abuse_suspend, account_cooldown, model_availability, budget_exhausted

Every rule that fires (block / flag / DLP / moderation) is recorded without the prompt text to the security-event stream (GET /admin/audit/events). The same policy runs on the realtime WebSocket, so it is not a bypass. moderate needs a moderator wired into the handler — the default one allows everything. See Governance.

admin — the runtime-admin gate

admin:
  token_env: GW_ADMIN_TOKEN    # env var holding the global admin bearer token;
                               # absent (and no tenant admin_token_env) = the whole
                               # /admin/* surface answers 404

The global token manages everything; a tenant’s admin_token_env token is scoped to that tenant (see API — Admin).

Top-level flags

trust_proxy_headers: false     # audit source IP: false = the real TCP peer (unforgeable);
                               # true = trust x-real-ip / rightmost x-forwarded-for hop
                               # (only behind a proxy that sets them)
max_live_streams_per_key: 64   # concurrent realtime sessions + MCP listen streams one key may hold; 0 = unlimited

Observability

GET /metrics serves the Prometheus registry: gateway_requests_total (route/status), gateway_request_duration_seconds, gateway_node_duration_seconds (pipeline stage), gateway_tokens_total, gateway_cache_hits_total, gateway_ledger_write_failures_total, and gateway_upstream_connect_retries_total (account), gateway_upstream_status_retries_total (account, status), gateway_thinking_signature_review_total (result) and gateway_thinking_signature_cache_events_total (event), gateway_mcp_requests_total (server, method, result) and gateway_model_fallbacks_total (from, to). One structured access log line per successfully served request goes to stdout. Setting OTEL_EXPORTER_OTLP_ENDPOINT (or OTEL_EXPORTER_OTLP_TRACES_ENDPOINT) exports one span per request over OTLP/HTTP, sampled per OTEL_TRACES_SAMPLER; see Observability.

Going live against real upstreams

export OPENAI_KEY=sk-...        # your key, in your environment
# account in YAML: endpoint: "https://api.openai.com", api_key_env: "OPENAI_KEY"
cargo run -p gw-server

Accounts with an endpoint egress to it; accounts without one are served by the in-process mock — unless the account names a declared provider, whose endpoint and credentials it then inherits. An explicit endpoint: "mock://…" opts a provider-bound account back onto the mock. GW_TRANSPORT overrides the routing: mock forces zero egress (nothing leaves the process), http disables the mock so misconfigured accounts fail loudly instead of returning fake data.

This site is open source. Improve this page.