{"id":22128,"date":"2026-11-12T15:00:00","date_gmt":"2026-11-12T14:00:00","guid":{"rendered":"https:\/\/webhosting.de\/?p=22128"},"modified":"2026-11-05T08:25:52","modified_gmt":"2026-11-05T07:25:52","slug":"hosting-ki-applications-cpu-gpu-storage","status":"publish","type":"post","link":"https:\/\/webhosting.de\/en\/hosting-ki-anwendungen-cpu-gpu-storage\/","title":{"rendered":"Hosting for AI Applications: Sizing CPU, GPU, and Storage"},"content":{"rendered":"<div class=\"wh-article\" data-wh-layout=\"2.1.2\" style=\"width:100%;max-width:820px;margin:0 auto;color:#263b4b;font-family:inherit;font-size:18px;line-height:1.8;text-align:start;overflow-wrap:break-word;box-sizing:border-box\"><p class=\"wh-lead\" style=\"font-size:20px;line-height:1.7;color:#233746;margin:0 0 1.1em\">For AI hosting, the specific workload determines how much CPU power, GPU memory, and storage you need. <strong style=\"font-weight:700;color:inherit\">Model, Precision, and Parallelism<\/strong> are just as important as response time and data access. In text models, cache and runtime buffers\u2014in addition to weights\u2014also consume memory; long contexts and concurrent requests can increase memory requirements.  A preliminary calculation can help with the initial selection. You can then verify whether the infrastructure is actually suitable by testing it with representative load and your application\u2019s requirements.<\/p>\n<nav class=\"wh-toc\" aria-label=\"Contents of this article\" style=\"display:block;margin:28px 0 38px;padding:22px;border:1px solid #d9e4e8;border-radius:14px;background:#f4f8f8\"><p class=\"wh-toc-title\" style=\"font-size:13px;font-weight:700;letter-spacing:.08em;color:#49656b;margin:0 0 14px\">Go directly to the section<\/p><div class=\"wh-toc-grid\" style=\"display:grid;grid-template-columns:repeat(auto-fit,minmax(min(100%,280px),1fr));gap:10px\"><div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#workload-anforderungen\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Narrow down the AI workload first<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#cpu-ram-datenzufuhr\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">CPU and RAM in the pipeline<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#gpu-vram-grundlagen\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Understanding GPUs and VRAM Properly<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#speicher-rechenbeispiel\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Calculate an approximate model memory size<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#workloads-vergleichen\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Compare Workloads and Bottlenecks<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#storage-netzwerk-planen\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Planning Storage and GPU Connections<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#messung-fehlersuche\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Detecting Bottlenecks Under Load<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#betrieb-wiederherstellung\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Operating AI Infrastructure Reliably<\/a><\/div>\n<div class=\"wh-toc-item\" style=\"display:block;min-width:0;margin:0;padding:0\"><a class=\"wh-toc-link\" href=\"#hosting-entscheidung\" style=\"display:block;height:100%;padding:12px 15px;border:1px solid #dce6e8;border-radius:9px;background:#fff;color:#18575b;font-size:14px;line-height:1.5;font-weight:600;text-decoration:none;box-sizing:border-box\">Select a Hosting Model and GPU Server<\/a><\/div>\n<\/div><\/nav><section class=\"wh-section\" aria-labelledby=\"workload-anforderungen\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"workload-anforderungen\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Narrow down the AI workload first<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Hardware selection starts with the task, not with a GPU. When it comes to <strong style=\"font-weight:700;color:inherit\">Inference<\/strong> An existing model processes inputs and generates responses, for example. Fine-tuning adapts a pre-trained model to additional data or a specific task; full training, on the other hand, builds the model weights from scratch during a separate training run.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">These goals result in different load profiles: A chat service must respond to requests, while a training job processes data over an extended period and saves interim results. For <strong style=\"font-weight:700;color:inherit\">AI Hosting<\/strong> You should therefore determine whether response time, the number of requests processed, training duration, or perhaps the cost of a specific task is the primary focus.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Document the model version and runtime, as well as the number of parameters, architecture, precision, and any planned quantization. Also note the expected input and output lengths, concurrent requests or batch sizes, and peak load. This allows you to compare quotes and subsequent tests based on the same assumptions, rather than simply comparing nominal hardware specifications.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">With an internal chatbot, a single short query can run without any problems, while multiple long conversations can overload memory. The KV cache maintains additional states during text generation; its requirements depend, among other things, on the model and the processing involved. Therefore, plan not only for the average dialogue, but also for long contexts and concurrent use. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Also document what data is being processed, where it may be stored, and what level of availability and recovery time is required. These operational goals influence the choice between a single server and a more complex architecture. The assumptions serve as a starting point: Only a test with a representative load will show whether they hold up.<\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"cpu-ram-datenzufuhr\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"cpu-ram-datenzufuhr\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">CPU and RAM in the pipeline<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Even when a model is running on a GPU, the CPU still plays a role along the data path. It can handle requests, tokenize text, read files, and prepare input for model processing. If this preparation is too slow, the GPU is left waiting for work; a more powerful accelerator won\u2019t necessarily resolve this bottleneck.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The <strong style=\"font-weight:700;color:inherit\">Working memory<\/strong> Depending on the application, it incorporates model components, prepared data, and file caches. In addition, it includes memory for processes that load and transform training data. Therefore, plan not only for the model itself, but also for the path from the dataset to the finished batch, and allow for sufficient reserves to handle processing peaks.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">PyTorch DataLoaders can use multiple workers for parallel data processing. However, additional workers also consume memory; even the shared memory space can become limited. Therefore, increase their number gradually and monitor data latency and memory usage, rather than adopting a fixed value as a universal recommendation. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">More CPU cores are particularly helpful when the tasks at hand can be executed in parallel. However, if storage accesses, a single processing step, or the GPU itself limit throughput, additional cores will make little difference. Clock speed behavior can also be relevant for short queries; for more background on this, see the article on <a href=\"https:\/\/webhosting.de\/en\/cpu-governor-hosting-server-performance\/\">CPU Governors on Hosting Servers<\/a>.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">For small models or infrequent queries, CPU inference is a viable alternative to a permanently deployed GPU system. Compare both options using the same model, appropriate precision, and the expected load profile. The key factors are whether the CPU can achieve the required response time even during peak loads and what the actual costs will be.<\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"gpu-vram-grundlagen\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"gpu-vram-grundlagen\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Understanding GPUs and VRAM Properly<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">For fully GPU-resident inference, the model weights must fit within the available GPU memory. The <strong style=\"font-weight:700;color:inherit\">VRAM Requirements<\/strong> But it doesn't end there: The runtime requires additional areas for intermediate results and temporary buffers. Therefore, a map that just happens to fit the weights mathematically does not provide a reliable foundation for production operation.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">In autoregressive text generation, the KV cache is also used. It stores previously computed key-value pairs so they can be reused in subsequent generation steps. Long contexts and concurrently processed requests can increase this additional memory requirement; the extent of this increase depends on the model and the processing. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The documentation for Transformers 4.48.2 describes, among other things, cache quantization and paging as ways to conserve VRAM. The benefits depend on the implementation and the workload; additional processing or data transfers can reduce performance. Therefore, evaluate memory consumption and response time together, rather than automatically treating memory savings as a performance gain. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">During training, inference alone is certainly not enough. In addition to the weights, storage is required for gradients, optimizer states, and stored activations. The size of these components depends, among other things, on the optimizer, the precision, and the training method; longer sequences or larger batches can increase activation memory in particular. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">For fully GPU-resident execution, the available VRAM must be sufficient to hold the required weights and runtime data. Hybrid methods may store parts of the processing or model weights outside the GPU; their suitability and the cost of data transfers depend on runtime. This should be distinguished from the KV cache offloading described in Transformers 4.48.2. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Consider the model, runtime, and planned parallelism together, and monitor peak memory usage under representative load. Sufficient memory for the chosen implementation does not necessarily guarantee the desired response time or utilization.<\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"speicher-rechenbeispiel\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"speicher-rechenbeispiel\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Calculate an approximate model memory size<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">An initial memory estimate is based on the number of model parameters and the format in which their weights are actually stored. It provides a lower bound for this memory usage, but does not yet provide a complete sizing of the GPU server. In particular, it does not indicate how much memory a specific request will require at runtime.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><strong style=\"font-weight:700;color:inherit\">Illustrative calculation example; not a measurement:<\/strong> Seven billion parameters multiplied by two bytes equals 14 billion bytes for the weights. Divided by 1,073,741,824 bytes per GiB, that comes to approximately 13.0 GiB. This assumes that all weights in question are indeed uniformly represented as two bytes per parameter; other formats or additional representations would alter the calculation.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Not included are the <strong style=\"font-weight:700;color:inherit\">KV Cache<\/strong> for text generation, temporary buffers, and runtime overhead. The cache can grow significantly, especially with long contexts; the number of requests being processed simultaneously also affects the required reserve. The cache documentation for Transformers 4.48.2 also describes methods that can save memory but, depending on the method, may come at the cost of speed. <\/p>\n<figure class=\"wp-block-image size-large wh-figure\" aria-describedby=\"wh-caption-detail1\" style=\"display:block;float:none;clear:both;width:100%;max-width:760px;margin:34px auto 40px;border:1px solid #dae5e9;border-radius:14px;overflow:hidden;background:#f5f8fa;box-shadow:0 10px 28px -17px rgba(24,47,60,.28);box-sizing:border-box\"><div class=\"wh-figure-media\" style=\"display:block;background:#eef3f5;line-height:0\"><img loading=\"lazy\" width=\"800\" height=\"534\" src=\"https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-modellspeicher-gewichte-cache-b2bdcd19-detail1-c712ad8679-1024x683.webp\" class=\"wp-image-22138\" alt=\"AI-generated illustration of a container with separate blocks for model weights, cache, and runtime buffer.\" decoding=\"async\" srcset=\"https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-modellspeicher-gewichte-cache-b2bdcd19-detail1-c712ad8679-1024x683.webp 1024w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-modellspeicher-gewichte-cache-b2bdcd19-detail1-c712ad8679-300x200.webp 300w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-modellspeicher-gewichte-cache-b2bdcd19-detail1-c712ad8679-768x512.webp 768w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-modellspeicher-gewichte-cache-b2bdcd19-detail1-c712ad8679-18x12.webp 18w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-modellspeicher-gewichte-cache-b2bdcd19-detail1-c712ad8679.webp 1536w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" ><\/div><figcaption class=\"wh-caption\" id=\"wh-caption-detail1\" style=\"display:block;margin:0;padding:12px 18px 15px;border-top:1px solid #dce5e9;color:#506575;background:#f5f8fa;font-size:14px;line-height:1.55;text-align:start\">AI-generated illustration: In addition to the weight calculation, further memory components are involved in inference.<\/figcaption><\/figure><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The weight calculation is certainly not sufficient for training. The documentation for Transformers 4.35.0 estimates 18 bytes per parameter, plus activations, for an example mixed-precision configuration using AdamW. This figure also accounts for other components such as gradients and optimizer states; the activations depend, among other things, on sequence length and batch size. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The 18 bytes are not a universal multiplier for every training run: optimizers, precision, and training methods can all affect the requirements. Therefore, neither this number nor the 13.0 GiB can be used to derive a recommendation for a specific VRAM size. The specific runtime and the expected peak memory usage under representative load are decisive factors in making this selection.<\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"workloads-vergleichen\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"workloads-vergleichen\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Compare Workloads and Bottlenecks<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The same model weights can result in very different requirements depending on the task. The table shows possible bottlenecks; it does not represent a ranking, nor does it guarantee that these are actual bottlenecks. Which resource is actually the limiting factor depends, among other things, on the model, data pipeline, parallelization, and workload.<\/p>\n<div class=\"wh-table-scroll\" tabindex=\"0\" role=\"region\" aria-label=\"Workloads: Memory Components and Practical Test Points\" style=\"width:100%;max-width:100%;overflow-x:auto;margin:28px 0;border:1px solid #dce5e9;border-radius:14px;background:#fff;box-shadow:0 10px 28px -16px rgba(24,47,60,.32);box-sizing:border-box\"><table style=\"width:100%;min-width:580px;border-collapse:separate;border-spacing:0;border:0;margin:0;font-size:15px;line-height:1.6;background:#fff\"><caption style=\"padding:20px;text-align:start;font-size:17px;line-height:1.5;font-weight:700;color:#173246;background:#fff\">Workloads: Memory Components and Practical Test Points<\/caption><thead><tr><th scope=\"col\" style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#19394d;color:#fff\">Workload<\/th><th scope=\"col\" style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#19394d;color:#fff\">Potential Bottlenecks<\/th><th scope=\"col\" style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#19394d;color:#fff\">Additional Storage<\/th><th scope=\"col\" style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#19394d;color:#fff\">Key Figure and Alternative<\/th><\/tr><\/thead><tbody><tr><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">CPU Inference<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">CPU Processing Time and Data Input<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">RAM for runtime and input<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">Check response time; compare GPU inference<\/td><\/tr><tr><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">GPU Inference<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">VRAM and Data Feed<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">Cache and Temporary Buffers<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">Check latency and memory usage; compare CPU inference<\/td><\/tr><tr><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">Fine-Tuning<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">GPU Memory and Data Processing<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">Gradients, Optimizer States, and Activations<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#fff;color:#294252\">Check steps per time; consider gradient accumulation<\/td><\/tr><tr><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">Distributed Serving<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">Communication between GPUs or servers<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">Cache and Runtime Buffer<\/td><td style=\"padding:15px 18px;border:0;border-bottom:1px solid #e2e9ed;text-align:start;vertical-align:top;white-space:normal;font-size:15px;line-height:1.6;background:#f4f7f9;color:#294252\">Check response time and throughput; if there is sufficient memory, compare a GPU<\/td><\/tr><\/tbody><\/table><\/div><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">When developing an internal chat bot, the model and generation are only part of the planning process. Define a maximum input length, an expected output length, and the number of concurrent users. You can then verify whether the response time remains acceptable during peak loads and whether the weights, cache, and buffer all fit into the available memory at the same time.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">A document search with generative responses involves a longer data path: documents are imported and processed, embeddings are generated, an index is searched, and relevant content is passed to the language model. Import and search may have different CPU, RAM, and storage requirements than the generation process. Therefore, it\u2019s worth tracking the times and memory requirements for these steps separately.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">When fine-tuning, you can <strong style=\"font-weight:700;color:inherit\">Gradient Accumulation<\/strong> allow for a larger effective batch size, even though only smaller individual batches are processed at a time. The gradients from multiple steps are accumulated before the parameters are updated. This can be helpful when memory is limited, but it does not guarantee an increase in throughput; additional processing steps can take time. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The data feed also deserves its own consideration: If preprocessing or loading lags behind model processing, a more powerful GPU won\u2019t necessarily help. Conversely, a fast data pipeline cannot compensate for insufficient VRAM. Therefore, compare alternatives using the same inputs and a transparent load profile. <\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"storage-netzwerk-planen\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"storage-netzwerk-planen\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Planning Storage and GPU Connections<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">When planning storage, it is helpful to distinguish between data types and access patterns: Training datasets are read repeatedly, and model weights are loaded before use. Both require a reliable source from which they can be made available again after a host change. Capacity alone is not enough; load time and access patterns also affect operations.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Temporary work data can be stored on a local NVMe drive if a fast workspace is needed. Check with your provider to see if this data will survive reboots and host changes. Checkpoints and backups, on the other hand, require a carefully planned retention strategy and a verified recovery path; a local scratch directory is not automatically an independent backup.<\/p>\n<figure class=\"wp-block-image size-large wh-figure\" aria-describedby=\"wh-caption-detail2\" style=\"display:block;float:none;clear:both;width:100%;max-width:760px;margin:34px auto 40px;border:1px solid #dae5e9;border-radius:14px;overflow:hidden;background:#f5f8fa;box-shadow:0 10px 28px -17px rgba(24,47,60,.28);box-sizing:border-box\"><div class=\"wh-figure-media\" style=\"display:block;background:#eef3f5;line-height:0\"><img loading=\"lazy\" width=\"800\" height=\"534\" src=\"https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-storage-arbeitsbereich-ablage-b2bdcd19-detail2-a6a81ed938-1024x683.webp\" class=\"wp-image-22140\" alt=\"An AI-generated stock image of an NVMe drive and an external storage drive on a desk mat.\" decoding=\"async\" srcset=\"https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-storage-arbeitsbereich-ablage-b2bdcd19-detail2-a6a81ed938-1024x683.webp 1024w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-storage-arbeitsbereich-ablage-b2bdcd19-detail2-a6a81ed938-300x200.webp 300w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-storage-arbeitsbereich-ablage-b2bdcd19-detail2-a6a81ed938-768x512.webp 768w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-storage-arbeitsbereich-ablage-b2bdcd19-detail2-a6a81ed938-18x12.webp 18w, https:\/\/webhosting.de\/wp-content\/uploads\/2026\/11\/ki-storage-arbeitsbereich-ablage-b2bdcd19-detail2-a6a81ed938.webp 1536w\" sizes=\"auto, (max-width: 800px) 100vw, 800px\" ><\/div><figcaption class=\"wh-caption\" id=\"wh-caption-detail2\" style=\"display:block;margin:0;padding:12px 18px 15px;border-top:1px solid #dce5e9;color:#506575;background:#f5f8fa;font-size:14px;line-height:1.55;text-align:start\">AI-generated illustrative image: Local workspaces and independent data storage serve different purposes.<\/figcaption><\/figure><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Shared or object-based storage can facilitate the distribution and storage of data sets, weights, and checkpoints. The key factors are whether the application supports the chosen access method and whether data can be loaded and restored within the specified time frame. An additional local workspace can be useful, but it does not replace permanent storage.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><strong style=\"font-weight:700;color:inherit\">GPUDirect Storage<\/strong> Under the right hardware, software, and file system conditions, this enables a more direct data path between storage and GPU memory. This is particularly useful when input and output are on the critical path and the application uses the supported path. If the CPU must first decode or preprocess the data, this can limit the benefits. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">When using multiple GPUs, a distinction must be made between replication and distribution. Replication provides copies of the model to distribute requests; it does not automatically combine separate VRAM regions into a shared memory. The documentation for vLLM 0.20.1 describes this type of data-parallel deployment; this does not imply anything about any other runtime version. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Tensor and pipeline parallelism, on the other hand, distribute parts of the processing and create communication requirements. To address this, check the connections between GPUs and, if multiple servers are involved, the network topology and the support provided by the model and runtime. The documentation for vLLM 0.8.5 covers these strategies for distributed serving; its specific instructions are version-specific. <\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"messung-fehlersuche\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"messung-fehlersuche\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Detecting Bottlenecks Under Load<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">A meaningful comparison begins with a documented <strong style=\"font-weight:700;color:inherit\">load profile<\/strong>. Record the model version, precision, specific hardware variant, and runtime version. Include typical and maximum context lengths, output lengths, and the number of concurrent requests. This allows you to test future changes to the infrastructure under the same conditions.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">When performing inference, measure the time to the first token and the generation rate. Also track errors and response times under normal load and during expected peaks. A high average throughput is not sufficient if individual requests take too long or fail due to long contexts or high parallelism.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">For training and fine-tuning, steps or processed examples per unit of time are a good starting point. Also, observe how long the data pipeline makes you wait for the next batch. PyTorch DataLoaders can be prepared using multiple workers; their memory requirements and the load on shared memory should also be factored into the evaluation. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Track peaks in GPU memory and RAM usage, as well as storage utilization and wait times. On an existing NVIDIA system, the read command shows <code data-no-translation style=\"font:600 .87em\/1.5 ui-monospace,monospace;color:#235063;background:#edf3f5;border-radius:4px;padding:.1em .3em\">nvidia-smi<\/code> Depending on the device and driver, this includes, among other things, GPU status and memory usage. It does not alter any production configuration, but it does not replace measurements taken by the application. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Low GPU utilization, on its own, does not indicate a CPU or storage problem. Compare it over time with data latency, request history, and storage usage, and, if possible, change only one relevant variable per comparison. Also document the limitations of the test, such as when rare load spikes or particularly long inputs are still missing.<\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"betrieb-wiederherstellung\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"betrieb-wiederherstellung\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Operating AI Infrastructure Reliably<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Operational planning requires a verifiable combination of GPU variant, driver, framework, and model runtime. Before installing updates, determine which versions are supported together and how to revert to the previous combination if problems arise. For specialized data paths such as GPUDirect Storage, the hardware, software, and file system must also be compatible with the intended use. <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Determine who is authorized to read or modify models, training data, logs, and administrative interfaces. For sensitive data, the storage location, data transfer, and retention period must also comply with data protection requirements. Verify these points for all involved services, not just the server on which the model runs.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">A <strong style=\"font-weight:700;color:inherit\">Checkpoint<\/strong> Saves a training state for later resumption. Plan which states your training software needs for this purpose, how often they should be saved, and how long they should be retained. Model weights for deployment and checkpoints for resumption serve different purposes and should be labeled accordingly.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Local working storage is not automatically permanent and is no substitute for an independent backup. Ask your provider what happens to local data in the event of a reboot, host replacement, or contract expiration. Store necessary data separately from your workspace and test the restoration process in practice, including the time it takes to become operational again.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">For distributed jobs, monitoring covers not only the individual GPUs but also nodes, network connections, and the progress of the entire application. Plan from which saved state a job will restart after a failure and which resources must be available for this. If model parts are distributed across multiple servers, their connections also affect operation; the notes in the vLLM 0.8.5 documentation are version-specific. <\/p>\n<\/section><section class=\"wh-section\" aria-labelledby=\"hosting-entscheidung\" style=\"display:block;margin:0 0 36px;min-width:0\"><h2 id=\"hosting-entscheidung\" style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Select a Hosting Model and GPU Server<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">For irregular jobs, a temporary employee can be provided <strong style=\"font-weight:700;color:inherit\">GPU Server<\/strong> This makes sense: You pay for the specific provision you need based on the terms of the offer, rather than maintaining capacity on an ongoing basis. Be sure to compare start times, availability at the desired time, and costs for stored data and data transmission.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">With a consistent load, it\u2019s worth considering a dedicated infrastructure. It can offer more control over runtime and data paths, but requires separate planning for updates, monitoring, and recovery. A managed inference service handles parts of this operation; in return, check which models, settings, and data paths are actually available there.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">To select the machine learning infrastructure, please provide details regarding <strong style=\"font-weight:700;color:inherit\">specific GPU model<\/strong> Consider: VRAM capacity, memory bandwidth, supported computing formats, and form factor. Do not apply the maximum values of a product family to every variant without verification. Compare performance specifications only with their test conditions and the intended model, rather than basing your decision solely on a theoretical peak value.<\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">The scaling method is also part of the offering: Replicated instances can distribute requests, while model partitioning requires support from the runtime and connection topology. The vLLM documentation covers data parallelism in version 0.20.1 and distributed serving in version 0.8.5; these statements should not be applied to other versions without verification.  <\/p>\n<p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">First, define the model and load profile; then estimate storage and data path requirements and request appropriate quotes. Next, test response time, errors, and resource requirements using representative queries or training data. You can find additional information to help you evaluate your options at <a href=\"https:\/\/webhosting.de\/en\/gpu-hosting-machine-learning-performance-infrastructure\/\">GPU Hosting for Web Applications<\/a>. Without this workload comparison, it is not possible to make a general GPU recommendation.<\/p>\n<\/section><section class=\"wh-sources\" style=\"margin:40px 0 0;padding:24px 0 0;border-top:1px solid #dce5e9;color:#596b7b;font-size:14px;line-height:1.65\"><h2 style=\"margin:38px 0 18px;color:#172f41;font-size:clamp(24px,1.25rem + 1vw,30px);font-weight:700;line-height:1.3;scroll-margin-top:115px\">Sources and Current State of Knowledge<\/h2><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Status of the research: <time datetime=\"2026-11-04\">2026-11-04<\/time><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\">Research date: November 4, 2026. Version-specific statements refer to the documentation versions mentioned in the article. No chart: Comparable performance data is not available.<\/p><div class=\"wh-source-urls\"><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/huggingface.co\/docs\/transformers\/v4.48.2\/kv_cache<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/docs.pytorch.org\/tutorials\/intermediate\/intermediate_data_loading_tutorial.html<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/huggingface.co\/docs\/transformers\/v4.35.0\/en\/model_memory_anatomy<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/docs.vllm.ai\/en\/v0.8.5\/serving\/distributed_serving.html<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/huggingface.co\/docs\/transformers\/grad_accumulation<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/docs.nvidia.com\/gpudirect-storage\/overview-guide\/<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/docs.vllm.ai\/en\/v0.20.1\/serving\/data_parallel_deployment\/<\/span><\/p><p style=\"margin:0 0 1.1em;font-size:inherit;line-height:inherit\"><span class=\"wh-source-url\" data-no-translation dir=\"ltr\" style=\"overflow-wrap:anywhere;font:12px\/1.6 ui-monospace,monospace;direction:ltr\">https:\/\/docs.nvidia.com\/deploy\/nvidia-smi\/index.html<\/span><\/p><\/div><\/section><\/div>","protected":false},"excerpt":{"rendered":"<p>What infrastructure does an AI application need? The model, workload, and data path are key factors. This article explains how to allocate CPU, GPU memory, storage, and network resources for inference and training.<\/p>","protected":false},"author":1,"featured_media":22137,"comment_status":"closed","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[922],"tags":[],"class_list":["post-22128","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-technologie"],"acf":[],"_links":{"self":[{"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/posts\/22128","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/comments?post=22128"}],"version-history":[{"count":3,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/posts\/22128\/revisions"}],"predecessor-version":[{"id":22141,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/posts\/22128\/revisions\/22141"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/media\/22137"}],"wp:attachment":[{"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/media?parent=22128"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/categories?post=22128"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/webhosting.de\/en\/wp-json\/wp\/v2\/tags?post=22128"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}