Private AI deployment options: on premises, dedicated cloud or API
Compare on-premises inference, dedicated cloud and external APIs through data flows, operating responsibilities and practical private AI acceptance checks.
Last updated
In short
- Place embeddings, logs and backups as deliberately as model execution.
- Choose capacity using representative workloads and a measured acceptance test.
- External API use requires an explicit data-transfer decision.
On this page (5)
Architectures · Part of Private AI
Start with the data path
A private AI deployment is a decision about who can process information, where that processing happens and who can restore the service. An internal chat address does not settle those questions. An application can run in your building while sending prompts, document snippets or embeddings to an external provider.
This is a reference decision framework, not a benchmark or a statement that a particular configuration has been deployed. Use it to define a pilot for private AI, then record the actual application versions, model revisions, infrastructure and support responsibilities in the pilot agreement.
Draw one request as six separate steps: employee authentication, document retrieval, embedding or reranking, model inference, answer delivery and operational logging. Add ingestion, backups and administrator access alongside the request path. Open WebUI’s data-handling explanation makes the relevant distinction: connecting an external model provider sends the necessary prompts and responses to that provider. The interface’s location alone cannot determine the data boundary.
Compare the three deployment patterns
| Decision | On premises | Dedicated cloud | External model API |
|---|---|---|---|
| Inference location | Customer-controlled facility | Contracted compute location | Provider’s service boundary |
| Capacity responsibility | Hardware procurement, power and spare capacity | Reserved resources and quota planning | Provider limits and changing service availability |
| Network dependency | Can be designed for local operation | Connectivity to the hosting environment | Connectivity to the API provider |
| Operational work | Hardware plus application operations | Platform plus application operations | Integration, access controls and vendor oversight |
| Data-placement evidence | Asset inventory and network observations | Contracts, region settings and dependency inventory | Current terms, processing locations and retention settings |
| Main sizing risk | Buying capacity that does not fit the workload | Paying for idle accelerators or insufficient reservations | Usage growth, rate limits and model availability |
There is no universal winner. A factory with unreliable connectivity might prioritize local inference. A team with variable demand might prefer a dedicated environment that can be resized. An external API may be appropriate for approved public material while restricted documents remain on a separate path. A mixed design should make that routing visible and enforceable; it should not silently send a failed internal request to an outside model.
Make the architecture concrete
Consider a synthetic pilot with two document classes: a published product manual and an unpublished maintenance incident report. Define which models may receive each class before connecting the corpus. The routing rule belongs in server-side policy, not in instructions asking employees to avoid mistakes.
| Component | Required design decision | Evidence to collect |
|---|---|---|
| Identity gateway | Allowed users and administrator roles | Successful and denied authentication records |
| Retrieval service | Corpus permissions and deletion propagation | Document IDs returned to each test account |
| Embedding service | Local or external execution | Destination host and model revision |
| Inference gateway | Allowed models, quotas and timeouts | Sanitized request trace and failure outcome |
| Logs | Whether prompts or documents are recorded | Sample log review and retention configuration |
| Backup storage | Approved location and encryption-key custody | Restore record and access inventory |
Place the model server on a private segment. A bearer token is one layer, not a complete perimeter: the vLLM security documentation explicitly distinguishes protected API paths from other endpoints. Expose only the application-facing routes needed by the pilot, restrict management interfaces and treat model-loading capabilities as privileged operations.
Pin versions and licences before sizing
Record the inference runtime version, accelerator driver, model repository revision, quantization format, embedding model and application release. Replacing only the model can change memory use, answer quality and acceptable-use obligations. A permissive runtime licence does not grant rights to every model it can load.
Open WebUI has its own licensing and branding conditions. Review those conditions for the intended distribution and branding; do not describe every component in this stack as interchangeable open-source software. Commercial support, offline activation and production support for accelerator drivers are separate procurement questions.
Use representative tasks for sizing: short questions, long documents, concurrent requests and interrupted sessions. Record the number of simultaneous requests, input and output lengths, response latency and error rate. These are measurements the pilot must produce, not performance figures this reference design promises.
Define acceptance and recovery
The pilot should pass four different tests. First, an unauthorized account cannot access the interface or its direct inference endpoint. Second, the restricted document never appears in an external request trace. Third, a saturated model returns an understandable timeout without losing accepted work or exposing another user’s context. Fourth, a backup restores configuration and required data into an isolated environment.
Retain the previous model revision and configuration when upgrading. Keep a small evaluation set with expected source documents and unacceptable disclosures. If a new model fails that set, route new sessions to the prior revision and preserve diagnostic evidence without publishing sensitive prompts. Database migrations may prevent simply downgrading an application container; rehearse recovery from the matching database backup.
A useful decision record ends with named owners for capacity, identity, patching, model approval and recovery. Bring that record, the two document classes and the expected workload to the deployment discussion. Pilae pricing can inform service planning once those boundaries are known; it cannot substitute for defining them.
Sources and further reading
Use the linked documentation to confirm the version, edition and contract that apply to your project.
