LiteLLM
A self-hosted LLM gateway and Python SDK that normalizes many hosted and local model providers behind OpenAI-compatible interfaces.
By BerriAI · reviewed 2026-10-21
What LiteLLM does
LiteLLM provides a gateway for routing, virtual keys, budgets, retries, fallbacks, logging, and provider normalization across hosted APIs and local inference servers. It is an application infrastructure layer rather than a model runtime: Ollama, vLLM, SGLang, cloud APIs, or other providers still perform inference.
Pricing model
Free open-source gateway with paid enterprise and managed options
Position in the stack
Where LiteLLM fits
LiteLLM is the gateway between applications and model providers. It normalizes APIs and centralizes routing, credentials, budgets, fallbacks, and observability; inference remains in connected providers.
Typical deployment flow
A practical sequence for evaluating LiteLLM before making it part of a permanent stack.
- STEP 1
Inventory providers
List endpoints, model aliases, credentials, costs, limits, and data policies.
- STEP 2
Deploy the gateway
Start the proxy with protected storage, keys, routing rules, and observability.
- STEP 3
Migrate and test clients
Verify streaming, tools, errors, fallbacks, and accounting through the gateway.
Best for
- One API across several model providers
- Centralized keys, budgets, routing, and fallbacks
- Separating applications from provider-specific APIs
Not the right layer for
- Running model weights without a separate inference backend
- A complete chat or RAG application
Capabilities
Capabilities refer to the tool's application layer. Hardware and model support can still depend on a connected inference engine.
Deployment and compatibility
- Deployment
- Self-hostedManaged cloudCluster
- Platforms
- Linux · macOS · Windows · Docker · Kubernetes
- Hardware backends
- Uses connected model provider
- Install methods
- Python package · Docker · Helm · Managed service
Models and integrations
- Model formats
- Uses connected local or hosted model provider
- Common integrations
- OpenAI-compatible clientsOllamavLLMSGLangLangfusePrometheusCloud model APIs
Strengths
- Broad provider normalization
- Centralized routing and operational controls
- Works in front of local and hosted inference
Limitations
- Adds another service to the request path
- Provider differences can still leak through a normalized API
Planning checklist
Before you choose LiteLLM
Answer these questions with the exact models, hardware, users, and data you expect to operate.
- Which provider behavior must be preserved?
- How will budgets map to users?
- What happens when a provider fails?
- Does the extra hop meet latency goals?
LiteLLM FAQ
Does LiteLLM run models?
No. It routes requests to local or hosted inference providers.
Why use it with vLLM?
It adds routing, keys, budgets, aliases, fallbacks, and logging in front of model servers.
Does one API make providers identical?
No. Context, tools, streaming, errors, behavior, and pricing still differ.
Official sources
Use these links to confirm current compatibility and installation requirements.
Related tools
A GPU cluster manager and model-serving control plane for heterogeneous infrastructure.
A model-serving platform for deploying language, embedding, reranking, image, and audio models.
A Kubernetes-native inference platform for standardizing scalable predictive and generative model services.
Execution evidence
Known working recipes using LiteLLM
Recipes connect hardware, a model artifact, tools, settings, verification, and a reportable result.
No verified recipe is linked to this record yet.
Compatibility estimates remain available in the planner. A recipe appears here only after its exact stack and verification protocol are documented.