Managed vLLM hosting
Pilae installs, patches, backs up and monitors vLLM for you, on your own servers or in any of 10+ Pilae Cloud regions, six of them in Switzerland and the EU.
On Business, from CHF 1,590EUR 1,690USD 1,990 a month, because vLLM needs dedicated hardware.
- AI and LLM
- Open source
- Replaces OpenAI API
- Runs on
- Your own hardware, or a Pilae Cloud region
- Replaces
- OpenAI API, Azure OpenAI
- Licence
- Apache-2.0
vLLM among the AI and LLM apps
Models and the interfaces your team talks to, inside your perimeter. Prompts and documents stay on your side of the tunnel.
vLLM system requirements
Before anything is deployed, this is what has to exist. We size it with you in the first session, and we say so when your own hardware is already enough.
All 5 requirements, and why each is there
- GPUNVIDIA, recent compute capabilityWith memory for the weights plus the KV cache at the context length and concurrency you need. AMD and Intel GPUs are supported through their own builds.
- Driver and runtimeNVIDIA driver + Container ToolkitThe official image carries its own CUDA libraries, and the host driver has to be new enough for them. On datacentre GPUs the compatibility libraries in the image can bridge an older driver; otherwise, moving to a newer image can mean upgrading the driver first.
- Model storageLocal NVMe, weights mirroredWeights are read from disk at every start, and upstream documents the default loader as most efficient on local storage. They are copied once from Hugging Face with the revision pinned.
- Shared memoryipc: hostvLLM uses PyTorch, which passes data between processes through shared memory, above all for tensor parallelism. Upstream says to give the container the host IPC namespace or a --shm-size, and we use the host namespace.
- AccessAPI key + /v1 allowlistThe API key covers /v1 and a few other prefixes. Routes such as /invocations, /tokenize and /pause answer without it, so the gate passes only the routes clients use.
Running vLLM in production: what it takes
Size, then pin and deploy
A vLLM image we have run, the model from the local mirror, and the tensor parallel size, context length and memory share in one config file in your repository. Usage statistics reporting to the project is switched off.
Put the API behind the gate
vLLM answers only on the private network, with a key per application. The gate in front passes the /v1 routes your clients need and nothing else.
Watch the queue and the KV cache
Probes check /health every 60 seconds. The metrics endpoint reports KV cache usage, waiting requests, time to first token and preemptions, and an engineer is alerted when the queue grows faster than it drains.
Back up the config and the weights
vLLM keeps no database. The config, compose files, mirrored weights and any fine-tuned adapters go offsite daily, encrypted, and the weights only change when the model does. On Enterprise, once a month we rebuild the server from them in a scratch environment and send it a test prompt.
Upgrade against your config
vLLM ships a minor release about every two weeks, and a deprecated flag can be gone two minor releases later. The Pilae Agent tests each upgrade against your config and model on a copy, waits for your approval, applies it in your window and keeps the previous image ready.
From our runbook
A vLLM server, configured from one file in your repository
# /etc/vllm/chat-model.yaml -- acme-inference-01, zur1, 2 x 80 GB GPUs # Started as: vllm serve --config /etc/vllm/chat-model.yaml # VLLM_API_KEY comes from the secret store; VLLM_NO_USAGE_STATS=1 is set. model: /models/chat-model # local mirror, never pulled at start served-model-name: chat-model # the name clients send as "model" tensor-parallel-size: 2 # one model split across both GPUs gpu-memory-utilization: 0.90 # vLLM's share; what weights leave is KV cache max-model-len: 32768 # longest prompt plus answer accepted max-num-seqs: 64 # sequences scheduled together enable-auto-tool-choice: true tool-call-parser: hermes host: 10.20.0.14 # private network address only port: 8000 uvicorn-log-level: warning
Migrating from OpenAI API to vLLM
Code written against the OpenAI API usually moves by changing three settings: the base URL, the key and the model name. vLLM serves the same Chat Completions, Completions, Embeddings and Responses endpoints, and tool calls work once the right parser is set for the model. What does not come across is the model. OpenAI's hosted models cannot be downloaded, so the work is choosing an open-weight model and testing it on your own prompts, which were tuned for another model and will need changes. Stored files, vector stores, fine-tuned models and Azure's content filters stay behind. Embeddings come from a separate embedding model, and vectors from two models cannot be mixed, so every vector index is rebuilt. Changing the code is the short part; that evaluation is the long one.
- AlternativesA ChatGPT Enterprise alternative you ownOne of 5 apps for teams leaving ChatGPT Enterprise.
- ServiceSaaS migrationA fixed-scope migration off Google Workspace, Microsoft 365, Zapier, Airtable and ChatGPT to open-source apps, with a parallel run and a cutover plan.
Moving from OpenAI API to vLLM, step by step
- Read the usage before choosing a model: Which endpoints, models and features your applications call, with token volumes and per-minute request peaks from the OpenAI or Azure usage records. Concurrency and context length decide the GPUs, not the number of users.
- Shortlist and test open-weight models: Two or three candidates whose licences fit, run against a set of your real prompts in a sandbox and scored beside the answers you get today. Prompts are adjusted here, not after the switch.
- Mirror the weights and size the server: The chosen weights are downloaded once, with the revision and checksums recorded. Gated models need access granted on Hugging Face to one of your users first, and some authors approve requests by hand. Tensor parallel size, context length and memory share are set from the measured load.
- Change the base URL, keep a fallback: Applications switch by configuration, one at a time, with the old key valid for one cycle. Anything that stores embeddings is re-indexed with the new embedding model before it switches.
What vLLM is, and who runs it
An OpenAI-compatible inference server for open-weight models, run on GPUs in Switzerland, the EU or your own datacentre, so prompts and answers never reach a model vendor.
Self-hosted vLLM for serving open-weight models
vLLM is an inference and serving engine for large language models. It started at UC Berkeley’s Sky Computing Lab and is now a hosted project of the PyTorch Foundation. PagedAttention keeps the KV cache in small blocks instead of one reservation per request, and continuous batching adds requests to the running batch as others finish. Tensor parallelism splits one model across the GPUs in a machine. Applications reach it through an OpenAI-compatible API.
It is built for many people or applications calling the same model at once. For one GPU machine serving a team, or several smaller models on modest hardware, Ollama is simpler to run. In a private AI deployment, vLLM usually sits behind a chat front end such as Open WebUI or LibreChat. Your own applications can also call it directly in place of the OpenAI API, and Langfuse beside it records each prompt and answer, so a change of model is judged on your own traffic.
Read the full briefFold the brief
vLLM in production: model weights, API access and GPU memory
vLLM runs on GPU machines dedicated to it, yours or ours in a
Pilae Cloud region: Zurich, Frankfurt, Falkenstein, Gravelines, Amsterdam or
Helsinki inside Switzerland and the EU, or one of six further afield. The weights are mirrored
locally with the revision recorded, so a restart never waits on a download. The server answers
only on your private network. Its API key covers /v1 and a few other
prefixes but leaves many routes open, including /invocations, which runs inference, so the gate
in front passes only the routes your clients call.
The failures that matter are about memory. vLLM refuses to start when the context length asked for does not fit in the KV cache, so the context length is set from the sizing before the first start. Under a burst it queues or preempts requests instead of failing them, so an overload shows up as slow answers rather than errors. Probes check the server every 60 seconds, and a queue that grows faster than it drains alerts an engineer. The config, compose files, weights and adapters are backed up daily, encrypted, to an offsite location in the country your contract names, and on Enterprise a scratch server is rebuilt from them every month, with the test recorded in the console. With a release every couple of weeks, the Pilae Agent runs each one against your config and model on a copy before you approve it for your window, and records the change in the console.
vLLM licence and model licences
vLLM is Apache-2.0. There is no paid edition and no enterprise directory, so nothing is kept back from the open-source code. Two other licences sit next to it. The model weights carry their own: some are Apache-2.0 or MIT, while others, such as Meta’s Llama models, come with their own licence and acceptable use policy. We check the one you choose before it is deployed. The CUDA libraries in the image come from NVIDIA’s own base image and are under NVIDIA’s licence terms. Our operation follows the published plans. Talk to us about the model you want to serve.
What Pilae is responsible for
The same commitments on every app we run, vLLM included, and what stays yours. How the platform does it.
Ours
- A pinned version: A version we have run, not whatever latest resolves to that day.
- A runbook: What it depends on, how it fails, what to do about it. In your repository.
- Daily backups: Encrypted and offsite, and on Enterprise restored every month. A backup nobody has restored is a file.
- A patch window: Security updates in a window you agreed, with a rollback ready.
- Someone watching: Every endpoint probed on the minute. An alert reaches a person, not a dashboard nobody opens.
Yours
- The credentials: You hold them. Ours are separate, named, logged and revocable with one command.
- The approval: We ask before anything changes outside an agreed window.
- The exit: The machine, the data, the compose files and the runbook are already yours. Nothing stops when our access does.
Who runs it
Engineers who answer for it
Our engineers review every plan the Pilae Agent prepares and answer for the result.
Your data stays where the contract says
On your own servers or in Pilae Cloud. On Business and Enterprise you choose the region, and your contract names each location.
A Swiss company, a Swiss contract
Pilae is a Swiss company, and your contract with it is governed by Swiss law.
vLLM needs dedicated hardware, so it runs on Business: CHF 1,590EUR 1,690USD 1,990 a month, one of up to three apps, on your own servers or in Pilae Cloud.
Compare the plansvLLM: common questions
Is vLLM open source?
Where do prompts and answers go?
Can our applications keep using the OpenAI SDK?
Should we run vLLM or Ollama?
What hardware does vLLM need?
Ask the engineers who would run your vLLM
Thirty minutes on the deployment you already have, or the one you are about to start.
- Go deeperPrivate AI in Switzerland: a private ChatGPT on your own infrastructureA private ChatGPT for your teams in Switzerland: Open WebUI, LibreChat or Dify, open-weight models in your perimeter, answers grounded in your own documents.
- CompareA Microsoft Copilot alternative you ownOne of 3 apps for teams leaving Microsoft 365 Copilot.
- CheckData residency in Switzerland and the EU for your open-source appsYour data stays in the country your contract names, Switzerland, the EU or your premises, with your contract naming each location.
Plan your vLLM deployment
Costs and savings
Private AI cost calculator: hardware, API and operations
Estimate private AI costs from actual usage, cancelled seats, model hosting, GPU utilization, retained tools and the work needed to operate it.
Open calculatorPrivate AI quantization: cost and quality trade-offs
Evaluate model quantization with memory, task quality, latency, compatibility and cost per accepted answer rather than assuming smaller is cheaper.
Read guide + download worksheetReduce private AI cost through GPU utilization
Evaluate private AI GPU utilization with accepted-task throughput, latency, request bursts, idle capacity and a reproducible cost worksheet.
Read guide + download worksheet
