OpenAI API gateway: manage SDK access across production apps

An OpenAI prototype often begins with one API key and one Python file. Six months later, the same key may be running in a background worker, a customer-facing feature, a data pipeline, and someone's test environment.

The SDK is not the problem. Shared authority is.

When several applications call OpenAI with the same provider credential, revoking one workload becomes disruptive, usage attribution becomes guesswork, and a retry bug in a small service can consume budget intended for the rest of the organisation. Rewriting every application around a new client library would add work without fixing the operating model.

A self-hosted AI gateway offers a narrower change. Applications keep using the OpenAI SDK, but connect with workload-specific virtual keys through one governed endpoint.

The use case: an invoice extraction worker

Consider a Python worker that reads uploaded invoices and returns structured fields for a finance review queue. It uses the official OpenAI SDK and Chat Completions API. The worker is useful, but it handles sensitive business text and its traffic is bursty near month end.

A production setup needs more than a valid API call:

• the worker should not receive the reusable OpenAI provider key;
• finance traffic needs its own rate and budget limits;
• a leaked worker credential should be revocable without interrupting other applications;
• configured policies should inspect the text handled through the gateway;
• operators need to identify the key, model, provider, usage, and outcome for each governed call.

Agent Access Manager can provide those controls for requests that pass through it. Direct access to the provider must also be restricted if the gateway is intended to be the enforcement boundary.

What changes in the request path

The application sends the same OpenAI-shaped request to a different base URL.

1. The OpenAI SDK calls Agent Access Manager at /v1/chat/completions with the worker's virtual key.
2. The gateway verifies that key and resolves its organisation, team, and project context.
3. Applicable rate limits, budgets, and request guardrails run before the provider call.
4. The requested model alias resolves to one or more configured deployments. The routing layer orders candidates and can try a fallback after a failed attempt.
5. The selected provider receives the request with its configured provider credential.
6. Usage and outcome data are published to the gateway's audit and billing pipeline. Configured response guardrails can inspect the completion before it reaches the worker.

The OpenAI provider credential remains in the gateway's encrypted catalogue and is not returned to the application.

Point the OpenAI Python SDK at the gateway

The official Python library supports a configurable base_url. Agent Access Manager's OpenAI-compatible endpoint includes the /v1 prefix, so the client configuration looks like this:

AAM_BASE_URL=https://gateway.example.com/v1
AAM_VIRTUAL_KEY=sk-replace-with-issued-virtual-key
AAM_MODEL=invoice-extractor

import json
import os
from openai import OpenAI

client = OpenAI(
    base_url=os.environ["AAM_BASE_URL"],
    api_key=os.environ["AAM_VIRTUAL_KEY"],
)

completion = client.chat.completions.create(
    model=os.environ["AAM_MODEL"],
    messages=[
        {
            "role": "system",
            "content": (
                "Extract the invoice number, supplier, currency, total, "
                "and due date. Return a JSON object and do not infer "
                "a value that is not present."
            ),
        },
        {"role": "user", "content": invoice_text},
    ],
    response_format={"type": "json_object"},
)

invoice = json.loads(completion.choices[0].message.content)

The worker still uses OpenAI and client.chat.completions.create. The base URL, credential, and model name now refer to the gateway. invoice-extractor is a catalogue alias rather than a provider model ID.

The OpenAI Python library documents both base_url and the OPENAI_BASE_URL environment variable in its official repository. The Agent Access Manager application guide covers the product-side connection.

Keep OpenAI-specific behaviour on the native path

For an OpenAI-protocol deployment, Agent Access Manager forwards the OpenAI chat shape without cross-provider translation. The implementation preserves additional request fields, message fields, and response fields through extensible JSON maps. That is why a parameter such as response_format can reach a compatible OpenAI upstream even though the gateway does not need to make a policy decision from that field.

Cross-provider routing has a different contract. If the same OpenAI-shaped request is routed to an Anthropic or Gemini deployment, the gateway translates the fields it understands. Provider-specific or future OpenAI fields may not have an equivalent and can be dropped or mapped on a best-effort basis.

Use an OpenAI-protocol deployment when the application depends on exact OpenAI behaviour. If you configure heterogeneous fallbacks, test the precise combination of structured output, tools, images, streaming, and error handling used by the application.

Replace shared provider keys with workload keys

Issue one virtual key for the invoice worker instead of copying a provider key used by other services. The key can be associated with the appropriate project or team, given an expiry, suspended during investigation, or revoked when the workload is retired.

Agent Access Manager generates high-entropy virtual keys and stores only their SHA-256 hashes. The secret is shown when issued; later views use non-secret metadata and a hint. OpenAI provider credentials are managed separately in the catalogue and encrypted at rest.

This separation changes incident response. Compromising the invoice worker's key no longer requires rotating the OpenAI credential used by every other governed application. It does not eliminate the need to protect the virtual key in a secret manager.

See virtual keys for issuance, scope, suspension, and revocation behaviour.

Set limits where the cost originates

Month-end invoice traffic may be legitimate. An infinite retry loop is not. Apply limits to the worker or finance project rather than relying only on an organisation-wide provider ceiling.

The current product supports:

• requests-per-minute controls;
• tokens-per-minute controls;
• token or configured-currency budgets;
• key, project, team, and organisation scope chains;
• hourly, daily, weekly, monthly, or non-resetting budget periods.

Rate-limit checks happen before a request. Usage-based spend accrues from provider-reported token data after the call. A request that crosses a hard budget can therefore complete before the new accumulated total blocks the next request. Configured currency accuracy also depends on the deployment pricing entered by the operator and the usage fields returned by the provider.

The budgets and rate limits guide documents the supported scope and enforcement model.

Screen invoice text without pretending policy is perfect

Invoices can contain bank details, tax identifiers, contact information, and text copied from email threads. Request guardrails can flag, redact, or block configured findings before the upstream call. On the Chat Completions path, response guardrails can also inspect non-streaming and streaming model output.

Start with synthetic samples that reflect the document formats the worker actually receives. Monitor findings before enabling broad blocking rules. A detector that treats every long number as a secret will stop valid invoices and teach the finance team to distrust the control.

Guardrails reduce exposure for traffic processed by configured policies. They do not prove that a document is safe, classify every jurisdiction's personal data correctly, or replace application validation. Read guardrails for the supported detector and policy actions.

Build an audit record that answers operational questions

For each governed call, the platform can record the virtual-key identity, requested alias, selected provider and model, status, token usage, timing, and configured cost. Guardrail findings are recorded as separate governance events.

That supports questions such as:

• Did the invoice worker cause the usage spike?
• Which deployment served its requests?
• Was the worker rate-limited or blocked by policy?
• Did failures concentrate on one provider deployment?
• How much configured spend was attributed to the finance scope?

The record is only as complete as the data the request path and provider expose. New provider usage fields can require product support before they are included, and cost reporting depends on configured pricing. Retention, export, and access policy remain deployment-owner decisions.

The audit, usage, and reports guide describes the current read model.

Production-readiness checklist

The result is a smaller blast radius

The application code changes in three places: base URL, credential, and model alias. The larger change is operational. Each workload gets separate authority and limits, while provider credentials, routing, policy, and activity records stay in an administered layer.

That does not make every OpenAI feature portable across providers, guarantee cost savings, or prevent direct-provider bypass by itself. It gives the organisation a concrete enforcement point for the OpenAI SDK traffic routed through it.