Choosing the Right Hardware for Deployment on WriftAI

Choosing hardware for model inference is not as simple as selecting the GPU with the most compute or the largest amount of VRAM. A more powerful accelerator can produce better benchmark numbers, but that does not necessarily mean it is the right choice for a production workload.

In practice, hardware requirements vary significantly across models. Lightweight models may not benefit enough from additional compute and memory to justify the higher resource cost, while larger generative models may require substantially more compute and memory.

This makes hardware selection less about choosing the most powerful GPU and more about matching the model's memory footprint, execution pattern, workload size, and latency requirements to the available resources. To explore the available hardware configurations discussed in this post, visit the pricing page.

Memory Footprint

Whether a model fits comfortably within the available memory depends on more than its weights. GPU workloads also require memory for activations, intermediate tensors, and, for LLMs, KV cache. Context length, batch size, and concurrency can further increase memory requirements.

However, having enough VRAM to load a model does not mean that a higher-end configuration is necessary. If a model fits comfortably on a smaller resource and meets the required performance under load, allocating a GPU with significantly more memory may provide little additional value. Higher-memory configurations become relevant when the workload requires more memory headroom, compute, or throughput.

For example, lightweight models such as sentence-transformers/all-miniLM-L6-V2 may fit comfortably on cpu-4x, while larger LLMs and generative workloads like ibm-granite/granite-4.2-30b or microsoft/phi-4-mini-instruct may require an option with a GPU depending on their runtime requirements and workload characteristics.

Model Architecture and Workload

Parameter count alone does not determine how a model uses hardware. The execution pattern and expected workload can have a significant impact on compute requirements, memory usage, and the amount of parallel work available to the GPU. Hardware selection should therefore consider not only the number of parameters, but also input size, generation length, inference steps, memory requirements, parallelism, data-transfer overhead, and end-to-end latency. These factors should also be weighed against cost, since higher-priced hardware is only beneficial when its additional resources translate into meaningful performance gains.

Single-Pass Encoder Models

An encoder model such as Facebook/RoBERTa-large processes the input in a single forward pass, without autoregressive generation or a growing KV cache. Since inputs beyond 512 tokens are truncated, the model was stress-tested at this max token limit to represent its highest input workload. At this limit, the model used approximately 1 GiB of VRAM. The workload was tested across cpu-4x, l4-mini, and t4.

While t4 provided the best cold-start latency, its additional compute and memory capacity did not provide a meaningful advantage during warm inference. l4-mini delivered the best warm performance while providing a significant improvement over cpu-4x and sufficient capacity for the workload, making it the more appropriate hardware configuration on WriftAI than the higher-capacity t4.

For such lightweight models, the actual computation is too small to keep the GPU busy. Moving inputs from CPU to GPU, launching GPU kernels, and synchronizing results can introduce overhead that reduces the benefit of GPU acceleration. As a result, even when GPU execution makes the model's computation significantly faster, the overall inference latency may not decrease by the same amount.

Figure 1 - Visualization showing how adding a GPU may not significantly improve end-to-end inference performance when model computation is not the primary bottleneck.

Generative Models

Generative architectures have a very different execution pattern. Diffusion models such as stability-ai/stable-diffusion-3.5-large, repeatedly run their denoising network across multiple inference steps, while autoregressive LLMs like ibm-granite/granite-4.2-30b, repeatedly execute the model as tokens are generated and maintain KV-cache state. These workloads involve considerably more computation and can place greater demands on GPU memory and compute resources.

For example, black-forest-labs/FLUX.2-klein-9b is a 9B-parameter rectified-flow transformer for text-to-image generation, while tongyi-mai/Z-Image-Turbo is a 6B-parameter image-generation model using an accelerated 8-step generation process. Higher memory bandwidth can benefit these workloads by speeding up repeated data access, while sufficient free GPU memory provides headroom for the model and runtime allocations. Higher-capacity configurations such as l40s or h100 can therefore be more relevant when their resources align with the workload.

Encoder-Decoder Models

Architecture can also determine which GPU characteristics are most beneficial. For example, falconsai/text-summarization uses an encoder-decoder architecture. While the encoder processes the input in a forward pass, the decoder generates the output autoregressively, repeatedly processing tokens. This execution pattern can benefit from higher memory bandwidth, particularly during decoding, because model weights and intermediate data need to be accessed repeatedly. As a result, a model may fit and run on a lower-capacity GPU such as l4-mini but perform better on a t4 when the latter provides more suitable memory-bandwidth characteristics for the workload.

CPU Sufficiency for Lightweight Inference

GPU acceleration can significantly improve inference performance, but it is not always required. For lightweight models, CPU inference may already provide sufficiently low latency, making GPU allocation unnecessary when the application's performance requirements are already met.

meta-llama/prompt-guard-86m and sentence-transformers/all-miniLM-L6-V2 were evaluated on both CPU and GPU. The results below show their cold and warm inference latency under the respective test configurations.

CPU Based Benchmarks

CPU testing was performed using a cpu-4x instance.

GPU Based Benchmarks

GPU testing was performed using a t4 instance

Although GPU inference provides better performance for both models, meta-llama/prompt-guard-86m and sentence-transformers/all-miniLM-L6-V2 are lightweight models that already achieve sufficiently low inference latency on CPU. This allows the required performance to be achieved without allocating GPU resources, while also reducing infrastructure and operational costs.

Using CPU-only dependencies also reduces the deployment size by approximately 3×, resulting in faster downloads and shorter cold starts. This becomes particularly useful for workloads where models are not usually kept warm, as lightweight models can also benefit from smaller cold-start times.

Hardware Selection on WriftAI

The hardware selection process can be approached as a sequence of checks. Start by checking whether the model fits within the available memory. If it does, evaluate whether the hardware is suitable for the model's architecture and execution pattern. Next, assess the expected workload and GPU utilization, then benchmark the configuration to determine whether it meets the required latency and performance.

Figure 2 - Hardware selection workflow based on memory, model architecture, workload, and performance requirements.

Conclusion

Choosing the right hardware comes down to finding the smallest resource that can comfortably run the model while meeting the workload's performance requirements. Memory determines whether the model can run, while its architecture, workload, and GPU utilization determine how effectively the available hardware can be used. Benchmarking these factors helps identify the configuration that provides the required performance without unnecessary resource and operational costs.