Budgets & rate limits

Budgets cap spend in tokens and/or money over a reset period; rate limits cap requests and tokens per minute. Both attach to any scope and are enforced across the whole scope chain at once — the tightest control wins.

Budgets

Dashboard: Operate ▸ Budgets. API: /admin/budgets (see Admin API). VIEW_BUDGETS to view; MANAGE_BUDGETS (admin/owner, or a team lead for a project in their team) to set.

Budgets are enforced pre-call on accumulated spend and accrued post-call from the event stream — so streaming works normally, and the one request that crosses the line still completes. An over-budget request is rejected with 429 before any vendor is called.

A budget is identified by four parts, so a single scope can hold several at once:

Each budget sets a hard tokenLimit and/or costLimit (denies when exceeded) plus an optional soft cost limit that warns but proceeds. So "monthly $5k hard cap on everything, plus a daily $200 soft warning, plus an hourly burst cap on the expensive model" can all live on one scope.

# Monthly $50 hard cost cap on an org (all models), warning at $40
curl -s -u admin:PASSWORD -X POST https://aam.example.com/admin/budgets \
  -H "Content-Type: application/json" \
  -d '{
    "scopeLevel": "ORG",
    "scopeId": "ORG-UUID",
    "costLimit": 50,
    "softCostLimit": 40,
    "currency": "USD",
    "period": "monthly"
  }'

Aggregate vs per-deployment budgets

This distinction is what makes cost-aware routing work:

• A provider-less budget (all deployments of an alias, or all models) is an aggregate cap, checked at the pre-routing gate. Exhausted means the whole alias is denied.
• A budget restricted to one provider is a per-deployment cap, checked per candidate during routing. Exhausted means the gateway fails over to a sibling deployment instead of denying.

Cap the premium backend per-deployment, leave aggregate headroom, and traffic spills to the cheap backend. Worked example: Cost-aware failover.

The two tabs

• Manage (per scope): an at-a-glance summary (budgets, near-limit, over, next reset) beside budget cards with live spend meters for cost and tokens against soft and hard thresholds.
• Explore (cross-scope FinOps): a filterable table over every budget in the org with burn rate, projected end-of-period, and time-to-exhaust, a spend trend and 7-day forecast chart, and model leaderboards (most expensive, most efficient by actual cost per Mtok, cheapest by catalog rate).

Rate limits

Dashboard: Operate ▸ Rate limits. API: /admin/rate-limits. MANAGE_GATEWAY (operator and up) to set or remove.

Rate limits cap requests per minute (RPM) and tokens per minute (TPM) per scope via a per-minute sliding window. Exceeding a limit returns 429 with a Retry-After header.

# 600 requests/min and 100k tokens/min on one key
curl -s -u admin:PASSWORD -X POST https://aam.example.com/admin/rate-limits \
  -H "Content-Type: application/json" \
  -d '{"scopeLevel": "KEY", "scopeId": "KEY-UUID", "rpmLimit": 600, "tpmLimit": 100000}'

Implementation notes that matter operationally:

• RPM admits-and-counts on the hot path; TPM is a pre-call read plus a post-call token accrual.
• Counters live behind a pluggable backend: in-JVM on a single instance (the default), or Valkey on the Distributed tier — the only backend that scales an org-wide counter across replicas.
• An unconfigured scope touches no counter store at all, so rate limiting adds nothing to a call until you set a limit.

The Limits tab shows live RPM/TPM meters against the effective cap — which is the tighter of your configured limit and any active SOAR throttle (shown as an amber banner with a "Stop throttle" action). The Activity tab shows the scope's request density as hour-of-day, day-of-week, and calendar heatmaps.

Related pages

• Enforce spend controls — a team cap plus a project throttle, verified.
• SIEM, UEBA & SOAR — automated throttling as a containment action.