A private AI workflow for contract review
Design a private AI contract-review pilot with synthetic clauses, source citations, access boundaries, human review and explicit acceptance tests.
Last updated
In short
- Use AI for bounded extraction and comparison while a qualified person owns legal judgment.
- Require a source passage and a visible “not found” outcome for each requested fact.
- Private hosting does not remove extraction errors, access risks or the need for evaluation.
On this page (6)
Business workflows · Part of Legal and fiduciary
Editable CSV with a worked example and fields for your own assessment. Example rows are illustrative, not measured results.
Download the synthetic contract-review exerciseA useful first contract workflow asks an assistant to locate and organize evidence: notice periods, renewal terms, defined parties or a clause that needs human review. It does not ask the model to decide whether a contract is safe to sign. The pilot should make errors visible and preserve the qualified reviewer’s decision.
The downloadable exercise uses invented clauses. It contains no client document and no measured model result. It is a starting fixture for an organization’s own evaluation, not legal advice or a certification of any AI system.
Define a narrow output
Ask for a structured result with the field requested, extracted value, exact source location and an uncertainty note. A missing clause should produce “not found in the supplied material,” not a plausible answer from general knowledge.
| Requested field | Useful output | Unacceptable shortcut |
|---|---|---|
| Renewal | Term and referenced clause | Guessing a standard renewal period |
| Notice | Required period and delivery conditions | Omitting an exception |
| Parties | Names as defined in the document | Merging similar entities |
| Liability wording | Located clause for reviewer assessment | Declaring legal adequacy |
| Missing material | Explicit list of referenced but absent schedules | Filling the gap from memory |
For a synthetic example: “The agreement renews for one year unless written notice is received at least sixty days before expiry.” The extraction task is to identify the one-year renewal and sixty-day notice condition with its source. It is not to infer whether that clause is enforceable or advisable.
Map the processing boundary
Open WebUI retrieval documentation describes how document material can be brought into a conversation. Its knowledge-base documentation also explains that processing location depends on extraction, embedding and model configuration. Verify every stage, including OCR, logs, temporary files and any external fallback.
A locally hosted interface can still call an external provider. If client material must stay within a particular environment, disable or constrain routes that would violate that requirement and verify the actual configuration. Do not assume “private AI” in a product name establishes the data flow.
Limit access to the matter team. Separate client collections and test with users who should not see the material. A document’s contents must be treated as untrusted input; embedded instructions should not grant access or authorize external actions.
Prepare the document and evaluation set
Start with rights-cleared or synthetic documents covering clean text, scans, tables, cross-references and amendments. Include a clause that was superseded and a schedule that is deliberately missing. Record which version is authoritative.
Have a qualified reviewer create the expected extraction and locate the supporting passages. Keep that answer key separate from prompts used during evaluation. Agree what counts as a major error: wrong party, missed exception, unsupported statement or access leakage should not disappear inside a broad average score.
NIST’s AI Risk Management Framework provides a general structure for identifying and managing AI risks. For this workflow, translate that into named owners, a documented task boundary and observed acceptance results rather than a vague assurance that a model is safe.
Run a controlled review
- Upload the approved sample under the intended access role.
- Request the bounded extraction with source locations.
- Check each answer against the original document, not just the generated summary.
- Record missing, incorrect and unsupported outputs separately.
- Repeat after changing document format, model or retrieval settings.
- Test a user without access and a document containing misleading instructions.
Do not report a success percentage until the exercise has actually been run. For a small sample, retain the individual cases so reviewers can see what the number represents.
Keep decisions and drafts separate
Generated wording should remain a draft with a clear reviewer. Do not automatically send a redline, approve a clause or update a matter record based only on model output. If the workflow later connects to another system, require an explicit authorization step and record the action separately from the assistant’s suggestion.
Version the prompt, model and retrieval settings used for each review. Preserve source-document identity and the human decision under the organization’s retention policy. Avoid logging sensitive text unnecessarily, but keep enough controlled evidence to investigate a disputed output.
Accept or stop the pilot
A pilot is useful only if it saves net reviewer effort while meeting the accuracy and access requirements. Count correction time, false confidence and failed extraction. A smaller model or cheaper infrastructure is not an improvement if it makes review slower.
Before broader use, test deletion, permission revocation and recovery of the required configuration. Retain a manual process so work can continue when the model or retrieval service is unavailable. Bring the task definition and exercise results to the private AI pilot review and the legal-industry briefing. The decision should follow evidence about this workflow, with professional judgment remaining with the reviewer.
Sources and further reading
Use the linked documentation to confirm the version, edition and contract that apply to your project.
