NVIDIA's Certified Systems Configuration Guide, updated August 6, 2026, separates inference and deep-learning training systems and says the workload, dataset, model and use case affect hardware selection. Those variables need to be measured before a team compares graphics processing units (GPUs), application-specific integrated circuits (ASICs) or accelerator services offered by cloud providers.

The selection starts with six questions: how often the model changes, whether the job is training or inference, the acceptable latency, peak throughput, memory demand and software portability. Chip specifications can narrow a shortlist, but they do not show how preprocessing, storage, networking and model loading will affect the complete service. A controlled trial with the buyer's own workload provides the evidence for a purchase or cloud commitment.

Measure each workload before comparing hardware

Create one record for every service rather than averaging several jobs into a single infrastructure profile. Record the model and version, parameter count, numerical precision, input and output sizes, batch size, concurrency, daily request volume and peak requests per second. Add the required model-quality threshold, availability target, data sensitivity and expected growth over the next three years.

Training, online inference and offline batch processing need separate records. Training runs may be constrained by accelerator memory, interconnect bandwidth and scaling efficiency, while an online service can fail its objective because of tail latency even when average throughput looks strong. An offline job can often trade completion time against price by running when capacity is cheaper or otherwise idle.

If measured traffic and latency data do not exist, the first procurement step is a small trial rather than a full rack purchase. The trial should capture normal and peak demand over a representative period. It should also separate warm-up time from steady-state performance so a fast sustained run does not conceal slow startup or model loading.

Workload stability determines the value of specialization

GPUs usually keep more options open when model architectures, frameworks or numerical formats are still changing. Their programmable hardware and established software tools can also make it easier to move capacity among unrelated jobs. Cloud rental can extend that flexibility when future demand is uncertain, although service availability, data-transfer charges and regional prices still need review.

A specialized accelerator becomes a candidate when the workload is stable, runs at enough volume to keep the hardware occupied and faces a clear limit on latency, energy or unit cost. imec IC-Link identifies throughput, latency and energy efficiency as three comparison metrics and says an ASIC reaches its strongest results only when the workload matches its design and data keeps the hardware utilized. The source is an ASIC design service, so its guidance is useful for defining questions but does not replace an independent run.

Peak operations per second should not decide the purchase. The host processor, accelerator memory, network path, storage system, compiler and runtime can each become the bottleneck. NVIDIA's guide likewise treats processor capacity, system memory, PCI Express topology, networking and storage as parts of an inference or training configuration rather than accessories to the accelerator.

Build a three-year cost range

Total cost of ownership (TCO) starts with purchased hardware or cloud usage, reserved capacity and the cost of meeting traffic above the reservation. For owned systems, the model also needs electricity, cooling, rack space, networking, storage and any facility work needed to support the equipment. For cloud services, it needs idle commitments, temporary peak capacity, data transfer and price differences among the regions the service can lawfully use.

Software and labor belong in the same calculation. Include drivers, compilers, monitoring, platform licenses, model conversion, performance tuning and on-call operations. A lower compute price can be offset by engineering work if unsupported operators, unfamiliar tools or a proprietary runtime slow deployment.

Reliability and exit costs complete the range. Buyers need to price spare capacity, replacement time, supply lead times, backups and recovery tests, then estimate the work required to move models and data to another platform. The result should show at least a base case and low- and high-utilization cases because a single utilization assumption can reverse the comparison between owned and rented capacity.

Run the same acceptance test on every candidate

Each candidate should run a normal-load case, a peak case and a failure case. Hold the model revision, dataset, numerical precision, software versions and quality threshold constant. Record any vendor-specific optimization because a comparison is not reproducible if only one system receives an undisclosed change to the model or runtime.

MLCommons says MLPerf Inference uses Offline, Server and Interactive scenarios for datacenter systems and SingleStream, MultiStream and Offline scenarios for edge systems; its load generator tracks latency and validates accuracy. The guide also describes its Closed Division as an apples-to-apples comparison using the same model and reference setup. A private procurement test need not reproduce MLPerf, but it can follow the same discipline of fixed inputs, declared scenarios and a quality check alongside speed.

For online inference, record 50th-, 95th- and 99th-percentile latency (P50, P95 and P99), throughput, error rate, output quality, power and cost per successful request. Generative-model tests may also need time to first token and the rate at which later tokens arrive. Training tests should record the time and cost required to reach a fixed quality target rather than comparing only the speed of individual steps.

The failure run should reflect the proposed deployment. It can remove a device or node, interrupt a dependency, restrict network capacity or force the service to recover from a saved checkpoint. Record dropped work, recovery time and whether the remaining capacity still meets the service-level objective (SLO).

Turn the measurements into a decision record

Set pass-or-fail gates first. A candidate that misses the quality threshold, peak-load SLO, data-control requirements or facility limits should not win through a weighted average. These conditions need written thresholds before results arrive.

Score the candidates that pass. Compare three-year cost, deployment time, software fit, operational skills, supply risk and portability. Weight each factor according to the service being purchased, then retain the raw measurements and assumptions with the score.

Record why the result could change. The decision should name the utilization, electricity price, demand forecast, software version and expected model life used in the calculation. It should also set a review trigger, such as a major model revision, a sustained traffic change or a new contract price.

Frequently asked questions

Is inference always better on an ASIC?
No. Stable, heavily used inference is a candidate for specialized hardware, but low or variable demand can leave that hardware idle. Frequent model changes can also make GPU flexibility worth more than a lower theoretical unit cost.

Does a higher FLOPS figure guarantee a faster service?
No. Floating-point operations per second (FLOPS) describes compute capacity under defined conditions, not end-to-end application time. Memory transfers, preprocessing, networking, software optimization and batching can determine the result.

Are owned accelerators always cheaper than cloud capacity?
No general answer applies. The comparison changes with utilization, electricity, cooling, staffing, financing, redundancy, cloud discounts and exit costs. A three-year range using the same demand forecast is more informative than comparing a server price with an hourly cloud rate.

Which metric should control the acceptance test?
The controlling metric is the service requirement that must not be missed, such as tail latency, completion time, output quality or cost per successful request. Throughput remains relevant, but it cannot compensate for a failed quality or latency threshold.