Reduce private AI cost through GPU utilization
Evaluate private AI GPU utilization with accepted-task throughput, latency, request bursts, idle capacity and a reproducible cost worksheet.
Last updated
In short
- GPU utilization is a diagnostic, not the business output you are buying.
- Measure accepted tasks under a latency target before comparing cost.
- Scheduling saves billed compute only when the provider and service requirements allow it.
On this page (6)
Costs and savings ยท Part of Private AI
Editable CSV with a worked example and fields for your own assessment. Example rows are illustrative, not measured results.
Download the GPU workload and cost worksheetA GPU costs money when it is reserved, including quiet periods. Higher utilization can improve the economics of a suitable workload, but a busy device can also be serving requests too slowly or producing answers that need rewriting. The useful target is more accepted work within the required response time.
This guide provides a reference experiment and an editable worksheet. It does not report a Pilae benchmark or recommend a GPU for a fixed number of employees.
Define the unit of useful work
Choose a task the service is meant to complete: classify a document, answer a sourced policy question or draft a summary for review. Define acceptance before measuring throughput. Count rejected answers, retries and human corrections rather than hiding them inside a raw token total.
| Measurement | What it tells you | What it does not prove |
|---|---|---|
| GPU utilization | How busy a device appears | Whether users receive useful answers |
| Time to first token | Initial responsiveness | How long the complete answer takes |
| End-to-end latency | Time from request to completion | Factual quality |
| Throughput | Work completed over time | Acceptable peak-period experience |
| Accepted-task rate | Output that meets the rubric | The cost of review unless recorded |
| Queue age | Whether demand outruns service | That adding a GPU is the only fix |
The vLLM metrics documentation describes runtime signals. Interpret them alongside the user-level measurements, not as a replacement for them.
Capture the demand shape
Record arrival times, input length, expected output length and concurrency. Separate interactive chat from batch extraction. Keep a few long-context cases: a short-prompt benchmark can miss the requests that dominate memory or latency.
Include the quiet period as well as the peak. A system that is busy for two hours a day and idle for the rest has different economics from a continuous workload. Sharing a model service across suitable teams may improve use, but only if access, data placement and service-level boundaries remain acceptable.
Run one controlled change at a time
The vLLM serving benchmark exposes workload and concurrency controls. Freeze the model revision, runtime, hardware, precision, context limits and task dataset before comparing configurations. Record the baseline, then vary one setting or arrival pattern.
A sensible sequence is to test the normal arrival profile, a burst, a sustained peak and a recovery period. Check whether the queue drains after demand falls. Stop increasing concurrency when latency or acceptance fails the agreed threshold; maximum throughput is not necessarily the right operating point.
This is a procedure to run on an isolated test environment with approved data. It is not evidence that any particular settings have already passed.
Calculate the complete unit cost
For a hypothetical monthly service costing 3,000 currency units, 30,000 accepted tasks imply 0.10 per task. At 10,000 accepted tasks, the same fixed bill implies 0.30. If quality drops and only half the outputs are accepted, the denominator changes accordingly.
Include application infrastructure, GPUs, storage, operations, licences and the review/rework cost you intend to compare. Show setup amortization separately from first-year cash. Compare an API alternative using equivalent tasks and its actual billable categories. Cost-optimization guidance also describes batching and lower-cost processing for work that can wait; those options are not equivalent to immediate interactive service.
Scheduling has conditions
Stopping noncritical compute outside working hours can reduce a bill only if billing stops for that resource. Persistent storage, reserved commitments and other components may continue. Include boot/model-loading time and the effect on early or late users. A weekday schedule is not a round-the-clock service.
For owned hardware, switching off changes power use but not the purchase already made. Distinguish incremental cash from depreciation or replacement economics. Do not turn a reduction in active hours into the same percentage reduction in total cost.
Keep a fallback and a change record
Before altering production concurrency or scheduling, preserve the working configuration and define a return condition. Test overload behavior, health checks and the path for urgent work. If fallback uses an external model API, confirm that the data policy allows the request to leave the local environment and include its cost.
Record latency percentiles, accepted-task counts, failures and the billed resource period in the worksheet. The private AI calculator can then use grounded inputs. If the workload stays small or irregular, a metered API may be cheaper; a local service may still be selected for placement or control. Keep that decision explicit.
Sources and further reading
Use the linked documentation to confirm the version, edition and contract that apply to your project.
