- Sticky load balancing — a load-balanced virtual model routes every request of a session to the target that served it first, which keeps provider prompt caches warm and model behavior consistent mid-conversation.
- Sticky API-key selection — when one provider has several keys, every request in a detected session uses the same key, preserving provider prompt-cache affinity.
- Threaded audit logs — the dashboard’s Audit Logs page groups a session’s requests into one thread: the latest request as the row, with an expander that unfolds the older requests beneath it.
- Per-session usage and cost — usage records carry the same scoped session id, so the dashboard and admin API can report the requests, tokens, and provider spend for one conversation or agent task.
How a session is identified
The first matching signal wins:-
Session headers — a built-in registry covers the headers known tools
send, plus gateway conventions:
-
Body fields — for clients that mark sessions in the request body:
Anthropic
metadata.user_id(Claude Code embeds its session UUID there, both current and legacy formats are parsed),session_id(OpenRouter convention),litellm_session_id,prompt_cache_key(Zed and other OpenAI clients that reuse a cache key per conversation), and/v1/responsesconversationreferences. -
Automatic detection — chat and responses requests with no explicit
signal are grouped by their conversation opening: the model, system
context, tools, and the messages through the first user turn. Follow-up
requests resend that prefix unchanged, so an agent that simply replays its
history (Aider, Cline, Continue, …) still gets a stable session id
(
auto-…) with no client changes.
Sticky load balancing
When a request carries a session and resolves through a virtual model with several targets, the first request picks a target via the redirect’s strategy (round_robin or cost)
and later requests stick to it. If the pinned target becomes unavailable or
saturated, the strategy picks again and the session re-pins — a session is
never glued to a dead target, and the all-saturated honest-429 behavior is
unchanged.
Under the adaptive strategy the pin is handed to the route selector, which
decides on every request whether the session keeps it — whenever more than
one target is viable. (With a single viable target there is nothing to
choose, so the selector is skipped and the redirect behaves like a plain
alias.) “Unavailable” then
also covers a target the selector has backed off — one serving 429s or
failing repeatedly — which stays in the target list and so looks fine to the
checks above. The session moves to a healthy target and re-pins there. A
target that is merely slow still keeps its session: moving it would throw
away the provider prompt cache the pin exists to protect. See
Intelligent Routing.
Affinity is on by default and can be disabled per redirect:
session_affinity on
PUT /admin/virtual-models).
Pins are in-memory per instance (like rate-limit counters): after a restart or
on another replica, the next request of a session simply re-pins. Idle
sessions expire after 6 hours.
API-key affinity is deterministic rather than stored, so the same ordered key
set selects the same key across replicas and restarts. Disable it per provider
with session_sticky_keys: false,
<PROVIDER>[_SUFFIX]_SESSION_STICKY_KEYS=false, or the dashboard’s
Session-sticky API keys
checkbox. Requests with no detected session remain round robin.
Threaded audit logs
Audit entries record the session id (session_id), and the Audit Logs page
groups them by default (“Group by session” toggle). Each thread shows its
latest request with a count badge; the expander on the left unfolds the older
requests. The badge counts the complete session. Filters select which sessions
appear and which matching request represents each thread.
GET /admin/audit/sessions returns each thread’s request_count and latest
matching audit-log entry. GET /admin/audit/log?session_id=… returns all
requests in an expanded session.
Each grouped thread also has a usage-chart action. It opens Usage Analytics
with that session selected; expanding an individual request exposes the same
action on its session badge.
Per-session usage and cost
Usage Analytics includes a full-width Usage by Recent Session table. Each row represents one detected, user-path-scoped session and shows request count, input/output/total tokens, and input/output/total cost. Session chips in that table and the request log filter the whole usage page. The admin API exposes the same bounded report:limit defaults to 50 and is
capped at 200. The normal usage filters (session_id, user_path, model,
provider, label, and date range) apply.
Configuration
Everything is on by default.config.yaml:
Authorization, API-key headers, …) are rejected
as session sources, since the session id is persisted on audit and usage
records.
When to change the defaults: set SESSION_AUTO_DETECT=false if you only
want explicitly-tagged sessions grouped; set session_affinity: false on a
redirect when you prefer strict load spreading over cache affinity (for
example pure round-robin capacity balancing across identical deployments).