AI Server Build: How to Size One for the Models You Actually Run

An AI server build is settled by one calculation, not a parts list. Take your model’s parameter count, multiply by the bytes each parameter uses, and you have the floor for GPU memory. NVIDIA’s NIM memory troubleshooting guide writes it as total_parameters × bytes_per_parameter ÷ tensor_parallel_size. A 70B model at BF16 needs roughly 140 GB for weights alone. At FP8, 70 GB.

That number decides your GPU, your power supply, and whether one machine is enough. Everything else is plumbing around it.

Start with the memory math

Bytes per parameter, per NVIDIA’s documentation: BF16 and FP16 use 2, FP8 uses 1, and INT4 or NVFP4 use 0.5.

The same guide works through concrete cases. Llama 3.1 8B at BF16 comes to 16 GB, which fits a 24 GB card with space left for cache. Llama 3.3 70B at BF16 split across four GPUs lands at 35 GB each. Drop that 70B to FP8 and it fits on two GPUs, still 35 GB each.

Weights are the floor, though — never the total. The KV cache, activations, and CUDA graphs all want memory too. vLLM budgets a fraction of the card with --gpu-memory-utilization, which defaults to 0.9, and the KV cache then expands greedily into whatever is left over. On a small card that leftover slice is thin: 10% of a 24 GB card is 2.4 GB, while graph capture alone can ask for 1–5 GB.

So size for the largest model you will genuinely serve, at the precision you will genuinely run, and leave real headroom.

Capacity and bandwidth are two separate purchases

This is where spec sheets mislead people. NVIDIA’s DGX Spark hardware overview lists 128 GB of unified LPDDR5x at 273 GB/s, and NVIDIA says a single unit handles models up to 200 billion parameters. A GeForce RTX 5090 has 32 GB of GDDR7 at 1,792 GB/s.

One holds a model four times larger. The other moves memory roughly six and a half times faster.

Token generation is bound by memory bandwidth, because every token requires reading through the model’s weights. Capacity buys you the ability to load a model at all. Bandwidth buys you the speed at which it answers. A machine that comfortably holds a 120B model can still feel slow, and that is not a defect — it is the bandwidth showing.

Pick your side deliberately. Very large models on a budget favour unified memory. Fast responses from a 14B–32B model favour a discrete card with fast VRAM.

Choosing the GPU

A 24 GB card remains the sensible entry point for an AI server build. It handles 8B–14B models at BF16, or roughly 32B once quantized to 4-bit, and used RTX 3090s are still the value pick across current build guides.

The RTX 5090 raises that to 32 GB, at 575 W through a single 16-pin connector. Check your case before ordering: NVIDIA’s user guide specifies room for a 304 mm card in the Founders Edition.

At the top of the single-card range, the RTX PRO 6000 Blackwell carries 96 GB of GDDR7 at 600 W. That capacity is enough to hold a 70B model at FP8 with room for context on one card, which sidesteps multi-GPU complexity entirely. NVIDIA also sells a Max-Q variant that trades clocks for lower power — worth considering when airflow is limited.

When a second GPU helps, and when it doesn’t

Two cards reliably buy capacity. Whether they buy speed depends on your software path.

The llama.cpp multi-GPU documentation defaults to layer split — pipeline parallelism, where each GPU holds a contiguous slice of layers and the KV cache for a layer lives on the GPU that owns it. The docs note this mode “can tolerate slow interconnect speeds between GPUs.” The newer tensor mode splits both weights and cache across cards to cut latency, but it “does multiple cross-GPU reductions per layer” and is “much more bottlenecked by the GPU interconnect speed.” Their own summary is the clearest statement of the tradeoff: pipeline-parallel maximizes batch throughput, tensor-parallel minimizes latency.

Tensor mode is still marked experimental, and its requirements are strict. Flash attention must be on. The KV cache must stay unquantized. Several architecture families are unsupported, including MoE and hybrid models such as DeepSeek2 and Grok, plus Mamba-style models, which fall back to layer split.

vLLM gives similar advice from the serving side. Its parallelism and scaling guide recommends setting tensor_parallel_size to the number of GPUs in a single box, but where cards lack NVLink it prefers pipeline parallelism for “higher throughput and lower communication overhead.” Uneven splits also favour pipeline, since it supports them.

The practical consequence for your build: PCIe lanes on the platform matter more than CPU core count. Community multi-GPU setup notes go further and recommend seating both cards on CPU lanes rather than chipset lanes.

Read More → PC for Deep Learning: 2026 Build Guide That Saves Money

The part most build guides skip

Concurrency multiplies memory, and the defaults are conservative. The Ollama FAQ sets OLLAMA_NUM_PARALLEL to 1 and the context window to 4096 tokens, and memory scales with both together — “a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation.” OLLAMA_MAX_LOADED_MODELS defaults to three per GPU, and OLLAMA_MAX_QUEUE holds 512 requests before returning a 503.

A server for five people is therefore not the same machine as a server for one, even running the identical model. The weights don’t change; the cache does.

Two levers help before you buy more hardware. KV cache quantization through OLLAMA_KV_CACHE_TYPE cuts usage to about half at q8_0 and a quarter at q4_0, with flash attention enabled. And vLLM prints both GPU KV cache size and Maximum concurrency at startup, calculated as if every request used the full context length — read that line before deciding you need a second card.

One quick diagnostic: ollama ps reports whether a model sits at 100% GPU or is split with the CPU. A split means you are paying system-RAM latency on every token.

A server on your network needs a door

Ollama binds to 127.0.0.1:11434 by default. Changing OLLAMA_HOST to 0.0.0.0 opens it to the network, and Ollama ships with no authentication of its own. Put a reverse proxy, firewall rule, or tunnel in front of it before anything else touches that port.

vLLM is blunter about distributed setups. Its documentation warns that traffic between nodes is unencrypted and that the data exchange format could allow arbitrary code execution if an attacker reaches the network, so VLLM_HOST_IP should be bound to a private segment.

This is the real line between an AI workstation and an AI server. The moment the box listens on a LAN, it is an unauthenticated inference endpoint until you place something in front of it.

Power, heat, and the always-on bill

575 W for an RTX 5090, 600 W for an RTX PRO 6000, each on one 16-pin connector. Two of those plus a CPU is a different electrical conversation than a gaming PC, and power-supply headroom matters more in a machine that never sleeps.

Even small AI systems are power-sensitive. NVIDIA states the DGX Spark requires its bundled 240 W adapter and warns that a weaker one risks “reduced system performance, failure to boot, or unexpected shutdowns.”

Idle draw is the line item people forget. A server that answers prompts for twenty minutes a day still pays for the platform the other twenty-three hours. Electricity rates vary far too much for a universal figure, so measure at the wall and apply your own rate.

Cooling deserves the same thought. Open-air cards recirculate hot air inside the case, and stacked cards in a closed chassis throttle. Blower-style or passively cooled server variants exist precisely for front-to-back airflow.

CPU, RAM, and storage: enough, not maximal

The CPU does not need to be exotic. Lane count and responsiveness matter more than raw cores, and multi-GPU is what pushes you toward platforms with more CPU lanes.

For system RAM, matching or exceeding your total VRAM is a sane floor, since weights are staged through memory on the way to the GPU. Ollama’s documentation notes that a model fitting entirely on one GPU “typically provides the best performance” by reducing PCI bus traffic, and otherwise spreads it across available GPUs.

Storage should be NVMe. Individual model files run to tens of gigabytes, and a working library grows faster than anyone expects.

Read More → The Local AI PC Build Guide Most Sites Get Wrong: VRAM Isn’t Everything

Sizing it, end to end

Run your own numbers with the formula above:

• 8B at BF16 — 16 GB of weights. A 24 GB card, with usable context room.
• 32B at 4-bit — about 16 GB of weights. Also a 24 GB card, but context and concurrency consume the remainder quickly.
• 70B at FP8 — 70 GB. One 96 GB card, or two cards at 35 GB each with tensor parallelism.
• 70B at BF16 — 140 GB. Four GPUs, or unified memory with the bandwidth tradeoff accepted.

Then add your target context multiplied by expected parallel requests. That total, not the weight figure, is what you actually buy for.

Mistakes worth avoiding

Buying capacity when you needed bandwidth. Sizing for weights and forgetting the KV cache. Assuming two GPUs halve response time when the default split mode is built for throughput. Leaving the context at 4096 and blaming the model for short memory. Exposing port 11434 to the network and calling it done.

Get the memory math right first. The rest of the build follows from it.

Read More → VRAM vs RAM for AI: What Local LLMs Need