Private AI: what stays inside your perimeter and what does not

“Private AI” is becoming a marketing phrase, and like most marketing phrases it is starting to mean less. Some offers called private AI send every prompt to a public model provider and store the chat history locally. Others run the model on your hardware and still phone home for web search, telemetry or embeddings.

Neither is dishonest in itself. But if your reason for private AI is that certain data must not leave your perimeter, you need to know exactly which flows cross it. This post draws that map for a typical private assistant, and says plainly what we do and do not keep inside.

What we mean by perimeter

Your perimeter is the set of machines and networks you control, or that an operator controls on your behalf under a contract that names where they are. For a Pilae deployment that means your own servers on premises, dedicated machines in Pilae Cloud in Switzerland or the EU, or both.

Inside the perimeter, apps answer only on a private network. The one way in from outside is a single hardened gate on port 443. That matters for the rest of this post: “inside” means reachable only on that private mesh, not merely hosted in a particular country.

What stays inside

In a private assistant built on Open WebUI, LibreChat or Dify with open-weight models, the sensitive parts stay on your machines.

Prompts and answers. Every question a user types and every answer the model gives is processed by a model running on machines dedicated to you and stored in a database on your machines.

Uploaded files and document collections. Documents indexed for retrieval are parsed, split and stored inside the perimeter. So are the vector embeddings computed from them, as long as the embedding model runs locally too. That last condition is worth checking in any setup, including ours.

The model itself. Open-weight models such as Llama, Mistral or Qwen run on your GPU servers or on dedicated GPUs in Pilae Cloud. Inference happens there. No prompt is sent to a model provider to get an answer.

Conversation history and user accounts. Chat logs, shared assistants, user preferences and access rights live in the app’s own database. Users sign in with your directory, through Keycloak or Entra ID, so identities stay under your control.

Logs. Application logs, including the ones that might contain fragments of prompts, stay on the machines that produced them. The console reads them there when an engineer needs them and keeps no copy.

For most organisations, this list covers the data that made them consider private AI in the first place: contracts, code, customer emails, medical or financial records, internal strategy.

Typed into a consumer chatbot instead, the same documents fall under that service’s terms for the user’s plan. When we checked on 7 November 2026, OpenAI’s help centre said of ChatGPT Free, Plus and Pro that “we may use your content to train our models” unless the user opts out (after a thumbs-up or thumbs-down, “the entire conversation” may still be used for training), and that by default it does not use inputs or outputs from ChatGPT Business, Enterprise, Edu or its API. Google’s Gemini Apps privacy hub said that with Keep Activity on, Google uses activity to improve its services “including training generative AI models”, stated no different rule for paid plans (work and school accounts may have different terms), and keeps chats reviewed by humans for up to three years.

What crosses the boundary, and why

Now the part most vendors skip. Some traffic does cross the perimeter in a normal private AI deployment. None of it has to carry your prompts or documents, but you should know what it is.

Model downloads. Weights come from somewhere, usually a public model hub. This is inbound traffic, and it can reveal which model you use. For air-gapped sites we bring the weights in as a planned, approved change, so the machines never reach the hub at all.

Software updates. Container images and security fixes are pulled from upstream registries. Again inbound, and again possible to route through a mirror for sites without outbound access.

Operational metrics. Our monitoring runs 60-second probes and collects machine metrics such as CPU, memory, disk and whether each service answers. Those metrics reach our control plane so that alerts reach Pilae engineers around the clock. They describe the health of the machine, not the content of a conversation.

Backups. Every app is backed up daily and stored offsite, which by definition means the data leaves the machine. It does not leave in a readable form: each backup is encrypted on your machine before it is sent, and stored in the country you choose. See backups for the details.

Network coordination. The private mesh has a coordination service, covered by our data processing agreement, that exchanges keys and addresses so your machines can find each other. Traffic between machines travels in encrypted tunnels, and the coordination service cannot read it.

Operator access. Pilae engineers reach your machines over the private network to apply approved changes and run restores. That is access, not data export, but it is a flow you should see in writing. Every change the Pilae Agent applies is planned, approved and recorded.

External models, if you choose them. Some organisations want a mix: local models for sensitive work, and a frontier model through an external API for general tasks. The same interface can route selected users or assistants to an external provider. When you enable that, those prompts do leave, to the provider you choose. In our setup this is off by default. The provider is disclosed in your data processing agreement and used only if you enable it.

Tools that reach the internet. Web search, URL fetching and some plugins send queries outside by design. A query typed into a web-search tool goes to that search engine. We leave these off unless you ask for them, and when you do, we decide with you which groups can use them.

App telemetry. Some open-source apps include optional usage telemetry or update checks. We disable these where the app allows it.

Questions to ask any private AI offer

Whether or not you work with us, these questions separate a private deployment from a local front end on a public service.

1. Where does inference run? On which machines, in which country, under whose control?
2. Where are embeddings computed? A local chat model with a remote embedding API sends every document you index to that API.
3. What is enabled by default? Web search, external models, telemetry, update checks. Ask for the list and for each item’s default.
4. What leaves in backups, and is it encrypted before it leaves? Where is the key?
5. Who has administrative access, and which law do they answer to? See CLOUD Act and your data location for how we answer this: models and data stay on your premises or in the region you choose.
6. Can it run with no outbound connection at all? If the answer is no, find out which dependency stops it.
7. Which sub-processors are involved? Ask for the list and the location of each.

A good operator answers all seven in writing, before you sign.

How Pilae fits

We think private AI should mean one thing: the content of your work stays on machines you control, and every flow that crosses the boundary is known, listed and switched on by you rather than by default. That is how we build private AI, on your premises or on dedicated GPUs in Pilae Cloud.

• The whole assistant inside. The chat interface, the model server, the embedding model and the retrieval index all run inside your perimeter, with open-weight models chosen per use case (private AI).
• Off until you switch it on. External models and internet tools stay off until you decide otherwise, and when you enable them, you decide which groups can use them.
• Flows that leave on purpose. Machine metrics reach the console so engineers can answer alerts (monitoring), and backups leave encrypted on your machine before they are sent. Both are written into your contract.
• Data location named in the contract. Your premises or Zurich by default, or another Pilae Cloud region you choose (data residency), with sign-in through your own directory.
• Updates you approve. The Pilae Agent plans each model and app update, waits for your approval and applies it in your window, with a backup first.

If you want to see what the map looks like for your organisation, talk to an engineer. We go through your use cases, your documents and your constraints, and write down which flows exist before anything is installed.