Private AI pilot acceptance scorecard
Evaluate private AI with an editable task scorecard covering answer quality, access boundaries, latency, cost and operating acceptance.
Last updated
In short
- Choose representative tasks and blocking failures before selecting the model.
- Score access isolation separately from answer quality.
- Measure cost per accepted task including review, retries and operation.
On this page (6)
Procurement and evaluation · Part of Private AI
Proposed tasks, allowed sources, quality and privacy gates, latency and human-review measurements.
Download the private-AI scorecardDecide what the pilot must prove
A private AI pilot should end with an explicit decision: approve a defined workload, extend the evaluation to resolve a named gap, or stop. A pleasant demonstration is not enough. The organization needs to know which tasks the system performs acceptably, what information it can reach and how much human work remains.
The downloadable scorecard contains proposed tasks and empty result fields. It is an evaluation design, not a benchmark report. No success rate, latency or cost in this guide represents a measured Pilae deployment.
Use the NIST AI Risk Management Framework as a reference for organizing risk considerations. The practical task selection and thresholds still belong to the organization using the system. A pilot score does not confer regulatory approval or make every future use acceptable.
Select tasks from actual work
Interview the people expected to use the assistant. Capture the input, acceptable output, consequence of a wrong answer and current review effort. Prefer a bounded task such as drafting an answer from an approved handbook over the vague objective “help employees work faster.”
| Task class | Useful sample | Acceptance evidence | Blocking issue |
|---|---|---|---|
| Retrieval | Answer a policy question with a source | Reviewer can follow citation to applicable text | Invented policy or inaccessible source |
| Summarization | Summarize a permitted contract | Material obligations retained; uncertainty visible | Omitted exception changes the meaning |
| Extraction | Populate a small record from a document | Each required field reconciles to source | Silent fabrication of absent values |
| Access control | Ask about another department’s file | Request denied across chat, search and previews | Title, snippet or content leaks |
| Failure handling | Model or document store unavailable | Clear failure and traceable support record | Unapproved external fallback |
Include normal cases, ambiguous cases and cases where the correct answer is “the available evidence does not establish this.” Record which examples were used to tune prompts and hold back others for the final comparison.
Freeze the configuration before comparing results
Record interface version, model identifier and licence, model-server version, quantization, context limit, embedding model, retrieval settings and identity rules. Keep the prompt template and source-document versions with the run record. A comparison is difficult to interpret if the model or document collection changes between attempts.
Open WebUI supports local and external model connections. Hosting the interface yourself therefore does not prove inference remains local. Inventory every endpoint used for chat, embeddings, reranking, transcription, search, tools and telemetry. For a local-only requirement, acceptance includes confirming that disallowed destinations are unavailable and that the workflow still behaves as specified.
Use gates and scores for different purposes
Quality can be scored with an agreed rubric: unacceptable, substantial repair, minor repair or accepted as delivered. Privacy and authorization should generally be gates. A high average quality score cannot compensate for a single unauthorized document disclosure.
Have reviewers assess outputs against expected facts, not writing style alone. Where practical, hide the candidate system’s identity during assessment. Record disagreements and adjudicate them against the source material. Do not compress every criterion into one number that conceals an important failure.
As an illustrative calculation, if 40 proposed tasks yield 30 accepted outputs, the acceptance rate would be 75%. Those are invented arithmetic inputs, not a result. The record must distinguish task attempts, accepted tasks, retries and excluded cases; otherwise repeated attempts can make a weak workflow appear successful.
Measure the whole cost of an accepted outcome
Record billable tokens or GPU operating hours, application infrastructure, licences, human review and repair time. Separate cash invoices from valued staff capacity. Compare against the existing process and include any SaaS seats that remain necessary.
Latency should describe the user experience: time to first useful response, completion time and the tail under realistic concurrency. One fast request on an idle system does not establish capacity for an organization. Use a staged load within approved limits and stop if it affects production services.
Approve an operating scope, not a general capability
Before rollout, assign ownership for model updates, document ingestion, access changes, incident handling and restoring the knowledge store. Pilae’s responsibility model provides a starting point for allocating those duties in a managed deployment.
The final decision should name approved users, tasks, data classes, model configuration, remaining limitations and a review date. Keep a way to disable the assistant or revert its configuration without losing source documents. Re-run relevant cases after a model, prompt, connector or permission change.
A useful next step is to fill the scorecard with ten real tasks and their business owners. That creates a specific brief for an AI pilot, with evidence requirements that a supplier or internal team can actually satisfy.
Sources and further reading
Use the linked documentation to confirm the version, edition and contract that apply to your project.
