Hosting for AI Applications: Sizing CPU, GPU, and Storage

Narrow down the AI workload first

Hardware selection starts with the task, not with a GPU. When it comes to Inference An existing model processes inputs and generates responses, for example. Fine-tuning adapts a pre-trained model to additional data or a specific task; full training, on the other hand, builds the model weights from scratch during a separate training run.

These goals result in different load profiles: A chat service must respond to requests, while a training job processes data over an extended period and saves interim results. For AI Hosting You should therefore determine whether response time, the number of requests processed, training duration, or perhaps the cost of a specific task is the primary focus.

Document the model version and runtime, as well as the number of parameters, architecture, precision, and any planned quantization. Also note the expected input and output lengths, concurrent requests or batch sizes, and peak load. This allows you to compare quotes and subsequent tests based on the same assumptions, rather than simply comparing nominal hardware specifications.

With an internal chatbot, a single short query can run without any problems, while multiple long conversations can overload memory. The KV cache maintains additional states during text generation; its requirements depend, among other things, on the model and the processing involved. Therefore, plan not only for the average dialogue, but also for long contexts and concurrent use.

Also document what data is being processed, where it may be stored, and what level of availability and recovery time is required. These operational goals influence the choice between a single server and a more complex architecture. The assumptions serve as a starting point: Only a test with a representative load will show whether they hold up.

CPU and RAM in the pipeline

Even when a model is running on a GPU, the CPU still plays a role along the data path. It can handle requests, tokenize text, read files, and prepare input for model processing. If this preparation is too slow, the GPU is left waiting for work; a more powerful accelerator won’t necessarily resolve this bottleneck.

The Working memory Depending on the application, it incorporates model components, prepared data, and file caches. In addition, it includes memory for processes that load and transform training data. Therefore, plan not only for the model itself, but also for the path from the dataset to the finished batch, and allow for sufficient reserves to handle processing peaks.

PyTorch DataLoaders can use multiple workers for parallel data processing. However, additional workers also consume memory; even the shared memory space can become limited. Therefore, increase their number gradually and monitor data latency and memory usage, rather than adopting a fixed value as a universal recommendation.

More CPU cores are particularly helpful when the tasks at hand can be executed in parallel. However, if storage accesses, a single processing step, or the GPU itself limit throughput, additional cores will make little difference. Clock speed behavior can also be relevant for short queries; for more background on this, see the article on CPU Governors on Hosting Servers.

For small models or infrequent queries, CPU inference is a viable alternative to a permanently deployed GPU system. Compare both options using the same model, appropriate precision, and the expected load profile. The key factors are whether the CPU can achieve the required response time even during peak loads and what the actual costs will be.

Understanding GPUs and VRAM Properly

For fully GPU-resident inference, the model weights must fit within the available GPU memory. The VRAM Requirements But it doesn't end there: The runtime requires additional areas for intermediate results and temporary buffers. Therefore, a map that just happens to fit the weights mathematically does not provide a reliable foundation for production operation.

In autoregressive text generation, the KV cache is also used. It stores previously computed key-value pairs so they can be reused in subsequent generation steps. Long contexts and concurrently processed requests can increase this additional memory requirement; the extent of this increase depends on the model and the processing.

The documentation for Transformers 4.48.2 describes, among other things, cache quantization and paging as ways to conserve VRAM. The benefits depend on the implementation and the workload; additional processing or data transfers can reduce performance. Therefore, evaluate memory consumption and response time together, rather than automatically treating memory savings as a performance gain.

During training, inference alone is certainly not enough. In addition to the weights, storage is required for gradients, optimizer states, and stored activations. The size of these components depends, among other things, on the optimizer, the precision, and the training method; longer sequences or larger batches can increase activation memory in particular.

For fully GPU-resident execution, the available VRAM must be sufficient to hold the required weights and runtime data. Hybrid methods may store parts of the processing or model weights outside the GPU; their suitability and the cost of data transfers depend on runtime. This should be distinguished from the KV cache offloading described in Transformers 4.48.2.

Consider the model, runtime, and planned parallelism together, and monitor peak memory usage under representative load. Sufficient memory for the chosen implementation does not necessarily guarantee the desired response time or utilization.

Calculate an approximate model memory size

An initial memory estimate is based on the number of model parameters and the format in which their weights are actually stored. It provides a lower bound for this memory usage, but does not yet provide a complete sizing of the GPU server. In particular, it does not indicate how much memory a specific request will require at runtime.

Illustrative calculation example; not a measurement: Seven billion parameters multiplied by two bytes equals 14 billion bytes for the weights. Divided by 1,073,741,824 bytes per GiB, that comes to approximately 13.0 GiB. This assumes that all weights in question are indeed uniformly represented as two bytes per parameter; other formats or additional representations would alter the calculation.

Not included are the KV Cache for text generation, temporary buffers, and runtime overhead. The cache can grow significantly, especially with long contexts; the number of requests being processed simultaneously also affects the required reserve. The cache documentation for Transformers 4.48.2 also describes methods that can save memory but, depending on the method, may come at the cost of speed.

The weight calculation is certainly not sufficient for training. The documentation for Transformers 4.35.0 estimates 18 bytes per parameter, plus activations, for an example mixed-precision configuration using AdamW. This figure also accounts for other components such as gradients and optimizer states; the activations depend, among other things, on sequence length and batch size.

The 18 bytes are not a universal multiplier for every training run: optimizers, precision, and training methods can all affect the requirements. Therefore, neither this number nor the 13.0 GiB can be used to derive a recommendation for a specific VRAM size. The specific runtime and the expected peak memory usage under representative load are decisive factors in making this selection.

Compare Workloads and Bottlenecks

The same model weights can result in very different requirements depending on the task. The table shows possible bottlenecks; it does not represent a ranking, nor does it guarantee that these are actual bottlenecks. Which resource is actually the limiting factor depends, among other things, on the model, data pipeline, parallelization, and workload.

When developing an internal chat bot, the model and generation are only part of the planning process. Define a maximum input length, an expected output length, and the number of concurrent users. You can then verify whether the response time remains acceptable during peak loads and whether the weights, cache, and buffer all fit into the available memory at the same time.

A document search with generative responses involves a longer data path: documents are imported and processed, embeddings are generated, an index is searched, and relevant content is passed to the language model. Import and search may have different CPU, RAM, and storage requirements than the generation process. Therefore, it’s worth tracking the times and memory requirements for these steps separately.

When fine-tuning, you can Gradient Accumulation allow for a larger effective batch size, even though only smaller individual batches are processed at a time. The gradients from multiple steps are accumulated before the parameters are updated. This can be helpful when memory is limited, but it does not guarantee an increase in throughput; additional processing steps can take time.

The data feed also deserves its own consideration: If preprocessing or loading lags behind model processing, a more powerful GPU won’t necessarily help. Conversely, a fast data pipeline cannot compensate for insufficient VRAM. Therefore, compare alternatives using the same inputs and a transparent load profile.

Planning Storage and GPU Connections

When planning storage, it is helpful to distinguish between data types and access patterns: Training datasets are read repeatedly, and model weights are loaded before use. Both require a reliable source from which they can be made available again after a host change. Capacity alone is not enough; load time and access patterns also affect operations.

Temporary work data can be stored on a local NVMe drive if a fast workspace is needed. Check with your provider to see if this data will survive reboots and host changes. Checkpoints and backups, on the other hand, require a carefully planned retention strategy and a verified recovery path; a local scratch directory is not automatically an independent backup.

Shared or object-based storage can facilitate the distribution and storage of data sets, weights, and checkpoints. The key factors are whether the application supports the chosen access method and whether data can be loaded and restored within the specified time frame. An additional local workspace can be useful, but it does not replace permanent storage.

GPUDirect Storage Under the right hardware, software, and file system conditions, this enables a more direct data path between storage and GPU memory. This is particularly useful when input and output are on the critical path and the application uses the supported path. If the CPU must first decode or preprocess the data, this can limit the benefits.

When using multiple GPUs, a distinction must be made between replication and distribution. Replication provides copies of the model to distribute requests; it does not automatically combine separate VRAM regions into a shared memory. The documentation for vLLM 0.20.1 describes this type of data-parallel deployment; this does not imply anything about any other runtime version.

Tensor and pipeline parallelism, on the other hand, distribute parts of the processing and create communication requirements. To address this, check the connections between GPUs and, if multiple servers are involved, the network topology and the support provided by the model and runtime. The documentation for vLLM 0.8.5 covers these strategies for distributed serving; its specific instructions are version-specific.

Detecting Bottlenecks Under Load

A meaningful comparison begins with a documented load profile. Record the model version, precision, specific hardware variant, and runtime version. Include typical and maximum context lengths, output lengths, and the number of concurrent requests. This allows you to test future changes to the infrastructure under the same conditions.

When performing inference, measure the time to the first token and the generation rate. Also track errors and response times under normal load and during expected peaks. A high average throughput is not sufficient if individual requests take too long or fail due to long contexts or high parallelism.

For training and fine-tuning, steps or processed examples per unit of time are a good starting point. Also, observe how long the data pipeline makes you wait for the next batch. PyTorch DataLoaders can be prepared using multiple workers; their memory requirements and the load on shared memory should also be factored into the evaluation.

Track peaks in GPU memory and RAM usage, as well as storage utilization and wait times. On an existing NVIDIA system, the read command shows nvidia-smi Depending on the device and driver, this includes, among other things, GPU status and memory usage. It does not alter any production configuration, but it does not replace measurements taken by the application.

Low GPU utilization, on its own, does not indicate a CPU or storage problem. Compare it over time with data latency, request history, and storage usage, and, if possible, change only one relevant variable per comparison. Also document the limitations of the test, such as when rare load spikes or particularly long inputs are still missing.

Operating AI Infrastructure Reliably

Operational planning requires a verifiable combination of GPU variant, driver, framework, and model runtime. Before installing updates, determine which versions are supported together and how to revert to the previous combination if problems arise. For specialized data paths such as GPUDirect Storage, the hardware, software, and file system must also be compatible with the intended use.

Determine who is authorized to read or modify models, training data, logs, and administrative interfaces. For sensitive data, the storage location, data transfer, and retention period must also comply with data protection requirements. Verify these points for all involved services, not just the server on which the model runs.

A Checkpoint Saves a training state for later resumption. Plan which states your training software needs for this purpose, how often they should be saved, and how long they should be retained. Model weights for deployment and checkpoints for resumption serve different purposes and should be labeled accordingly.

Local working storage is not automatically permanent and is no substitute for an independent backup. Ask your provider what happens to local data in the event of a reboot, host replacement, or contract expiration. Store necessary data separately from your workspace and test the restoration process in practice, including the time it takes to become operational again.

For distributed jobs, monitoring covers not only the individual GPUs but also nodes, network connections, and the progress of the entire application. Plan from which saved state a job will restart after a failure and which resources must be available for this. If model parts are distributed across multiple servers, their connections also affect operation; the notes in the vLLM 0.8.5 documentation are version-specific.

Select a Hosting Model and GPU Server

For irregular jobs, a temporary employee can be provided GPU Server This makes sense: You pay for the specific provision you need based on the terms of the offer, rather than maintaining capacity on an ongoing basis. Be sure to compare start times, availability at the desired time, and costs for stored data and data transmission.

With a consistent load, it’s worth considering a dedicated infrastructure. It can offer more control over runtime and data paths, but requires separate planning for updates, monitoring, and recovery. A managed inference service handles parts of this operation; in return, check which models, settings, and data paths are actually available there.

To select the machine learning infrastructure, please provide details regarding specific GPU model Consider: VRAM capacity, memory bandwidth, supported computing formats, and form factor. Do not apply the maximum values of a product family to every variant without verification. Compare performance specifications only with their test conditions and the intended model, rather than basing your decision solely on a theoretical peak value.

The scaling method is also part of the offering: Replicated instances can distribute requests, while model partitioning requires support from the runtime and connection topology. The vLLM documentation covers data parallelism in version 0.20.1 and distributed serving in version 0.8.5; these statements should not be applied to other versions without verification.

First, define the model and load profile; then estimate storage and data path requirements and request appropriate quotes. Next, test response time, errors, and resource requirements using representative queries or training data. You can find additional information to help you evaluate your options at GPU Hosting for Web Applications. Without this workload comparison, it is not possible to make a general GPU recommendation.