Gateway

Architecture

Cargo workspace, 11 crates, strictly layered — lower layers never depend on higher ones:

server → {views, task} → handler → dag → {engines, state} → {models, protocol, config} → consts

Crate	Layer	Role
consts	L0	error codes, the Protocol enum
models	L1	request/response domain types, typed params, usage, cost
protocol	L1	OpenAI / Anthropic wire types + cross-protocol conversions
config	L1	YAML config loading (conf/gateway.yaml)
state	L2	auth, account pool, quotas, rate limits, ledger, batch/file stores (in-process defaults; Postgres/Redis fleet backends)
engines	L2	engine implementations behind the Transport seam, SSE + AWS EventStream decoding, usage extraction, SigV4
dag	L3	4-layer pipeline executor + nodes
handler	L4	online/offline orchestration, DLP/blocklist plugins
task	L5	background tasks (quota reset, content purge, usage rollup, availability flush/alerts, webhook dispatch)
views	L5	axum HTTP/WebSocket handlers, protocol conversion
server	L6	binary entrypoint: config + state + transport wiring

Request flow

client ──► views (auth, parse, protocol normalize)
       ──► handler (pre plugins: blocklist, moderation, then DLP redact)
       ──► dag: preprocess        resolve model, quota check, cache lookup
              account_select      priority / PTU-first / cooldown-aware selection, latency-ranked tiers when enabled
              model_access        rate limits, engine call, failover on an upstream fault
              post_process        usage → billing ledger, cache store
       ──► handler (post plugins) ──► views (JSON or SSE re-emit)

A client disconnect does not cancel the pipeline: every request runs on its own task (run_pipeline / spawn_stream_pipeline), so once admitted, quota and ledger accounting run to completion even if the caller goes away. Spawned background work (offline batches) likewise outlives the submitting request.

The DAG executes four fixed layers; nodes within a layer run sequentially in declaration order. Within model_access, call_engine reselects once on an upstream 5xx or a 401/402/403 credential or billing refusal: the failed account is excluded and another of the model’s accounts is tried; a PTU→paygo switch is recorded as ptu_spillover in the ledger. Around the DAG, the handler re-runs all four layers for the next entry of the model’s fallback_models chain when a run ends in an upstream fault before any byte was sent.

Seams (traits)

Every boundary to the outside world is a trait with a deterministic default, so the whole pipeline is testable offline:

• Transport — engines never own an HTTP client. The server default is DispatchTransport: accounts without an endpoint resolve to mock:// sentinel URLs served in-process by MockTransport (deterministic vendor-shaped replies); real URLs go over HttpTransport (reqwest + rustls). GW_TRANSPORT=mock forces zero egress, GW_TRANSPORT=http disables the mock. Tests inject MockTransport directly.
• TokenEncoder — prompt token estimation capability (tiktoken cl100k_base BPE default, zero-dependency heuristic fallback). cost_calc uses it to bill an aborted stream from the delivered text when the vendor’s usage frame never arrives; token-aware PTU sizing is not ported yet.
• Store — durable records (billing ledger, uploaded files, batch jobs). MemoryStore by default; SqliteStore when storage.sqlite_path is configured; PostgresStore when storage.postgres_url is — shared across a fleet.
• KeyStore — the live access-key table. AkAuth (in-process DashMap) by default; PostgresKeyStore with storage.postgres_url — fleet-shared behind an auth cache each write invalidates fleet-wide by NOTIFY, admin key CRUD survives restarts.
• UserBudgetStore — per-user budget overrides inside a tenant. In-process by default; PostgresUserBudgets with storage.postgres_url, cached and invalidated the same way as keys.
• HealthStore — account cooldown/recovery. In-process breaker by default; RedisHealth with storage.redis_url — a tripped account is skipped by every instance. Only upstream faults count (5xx, timeouts, broken streams, a 401/402/403 credential or billing refusal), never a client-caused or rate-limit error.
• Governance — rate/quota/TPM counters. In-process by default; RedisGovernance shares them (including pooled tenant QPS) fleet-wide.
• Moderator — the external content review behind security.moderate; allow-all by default, AWS Bedrock Guardrails when moderation: names one.
• config store — with storage.postgres_url, config is versioned documents in Postgres (PUT /admin/config publishes; a LISTEN/NOTIFY change feed reloads every instance).
• metrics facade — metrics crate macros throughout; the server installs a Prometheus recorder and serves /metrics.
• Planned: see the issue tracker.

Testing

• Unit tests per crate; engine golden tests assert exact request wire shapes (via a recording transport) and response/usage parsing against recorded vendor fixtures.
• crates/server/tests/e2e.rs boots the full router in-process and exercises every API surface offline.
• live_path.rs drives the real reqwest stack against a loopback vendor to prove the go-live path end to end.

This site is open source. Improve this page.