AethosHub
PricingRequest Demo

DocumentationAgent Access Manager

Ollama, vLLM & LiteLLM

Ollama, vLLM & LiteLLM

Anything that exposes an OpenAI-compatible endpoint registers as a provider with protocol: openai — which covers Ollama, vLLM, Groq, LiteLLM, and most self-hosted inference servers. Combined with the gateway's own on-prem deployment, this keeps inference and governance entirely inside your infrastructure.

Ollama

Register the provider (Ollama ignores the API key, but the field is required — use a placeholder):

Command
curl -s -u admin:PASSWORD -X POST https://aam.example.com/admin/providers \
  -H "Content-Type: application/json" \
  -d '{"name": "ollama", "baseUrl": "http://ollama:11434/v1", "apiKey": "unused", "protocol": "openai"}'

curl -s -u admin:PASSWORD -X POST https://aam.example.com/admin/deployments \
  -H "Content-Type: application/json" \
  -d '{"alias": "local-llama", "providerName": "ollama", "upstreamModel": "gemma3:4b"}'

Leave pricing empty — local inference is free, and usage still meters in tokens. Applications now call local-llama through the same endpoint, keys, guardrails, and audit as any cloud model.

vLLM

vLLM serves an OpenAI-compatible API natively. Register it the same way, with the model name you launched vLLM with as the upstreamModel. If your build rejects a particular OpenAI parameter, list it in the deployment's dropParams so calls degrade gracefully instead of failing.

The hybrid pattern: cloud + local under one alias

Register a cloud deployment and a local one under the same alias, price the cloud one, and put a per-deployment budget on it. The gateway load-balances, fails over on errors, and spills to the free local backend when the cloud budget is exhausted — see Cost-aware failover.

LiteLLM as a breadth extender

The gateway deliberately serves the endpoints real tools use (what is and isn't served). If you need LiteLLM's wider provider or endpoint catalogue, run a LiteLLM instance and register it as a provider — its OpenAI-compatible endpoint plus its key:

Command
curl -s -u admin:PASSWORD -X POST https://aam.example.com/admin/providers \
  -H "Content-Type: application/json" \
  -d '{"name": "litellm", "baseUrl": "http://litellm:4000/v1", "apiKey": "sk-litellm-master", "protocol": "openai"}'

Your virtual keys, budgets, guardrails, audit, and SIEM sit in front; LiteLLM's breadth sits behind. The gateway interoperates with it as an upstream rather than depending on it.