Running High-Parameter Models on Consumer Hardware

Aug 7

Running High-Parameter Models on Consumer Hardware

VRAM, Quantization, and Optimization

1. The Local Inference Bottleneck

The shift toward local AI isn't just a hobbyist trend anymore. Privacy, latency, zero recurring API costs, and workflows that don't answer to someone else's content policy are pulling a growing number of engineers away from hosted inference and onto their own hardware. But the moment you try to load a serious model — 30B parameters and up — you hit a wall, and it's rarely the wall you expected.

Most people brace for a compute problem. They picture their GPU's CUDA cores maxed out, fans screaming, FLOPs the bottleneck. In practice, that's not where you get stopped. The wall is memory: does the model even fit in VRAM in the first place. A 4090 has enormous compute headroom relative to what local inference actually demands. What it doesn't have is enough memory to hold a 70B parameter model at full precision, and no amount of compute fixes that.

This is the core thesis worth internalizing before you spend money on hardware: you don't need an enterprise cluster to run 30B+ or 70B-parameter models. You need a clear understanding of precision, layer offloading, and quantization formats. A single 24GB consumer card, used correctly, can run models that on paper “require” 4x that much memory. The rest of this post is the math and the mechanics behind how.

2. Demystifying VRAM Mathematics: How Much Memory Do You Actually Need?

The baseline calculation is simple multiplication:

VRAM (GB) ≈ Parameter Count × Bytes per Parameter

Everything else in this post is a variation on that formula, so it's worth internalizing the precision breakdown:

Precision

Bytes/Param

70B Model Footprint

FP16 / BF16 (16-bit)

2 bytes

~140 GB

INT8 (8-bit)

1 byte

~70 GB

INT4 (4-bit)

0.5 bytes

~35 GB

That table alone explains why quantization isn't optional for local hosting — it's the entire game. Dropping from FP16 to INT4 doesn't just save space, it's the difference between “needs a multi-GPU server rack” and “runs on a single 24GB card.”

But the naive formula understates real-world usage, because model weights aren't the only thing competing for VRAM:

KV Cache. Every token you generate requires storing key/value pairs for attention, and this scales with context length — roughly O(N) per token in the sequence. A model that fits comfortably at a 2K context window can blow your VRAM budget at 32K, even though the weights themselves haven't changed size.
CUDA context and buffer overhead. Budget 1–2 GB just for the runtime itself — kernel launches, workspace buffers, framework overhead — before a single model weight is loaded.

If you're doing back-of-envelope math for a build, don't just compute the weight footprint and call it done. Add 10–20% headroom for KV cache and overhead, more if you're planning long-context work.

To make the KV cache cost concrete: cache size scales with 2 × layers × heads × head_dim × context_length × bytes_per_value (the leading 2 covers storing both keys and values). For a mid-size model at a 32K context window, that can add several gigabytes on top of the weight footprint — often enough to be the difference between a model fitting on your card and not. This is also why quantizing the KV cache itself (more on that in Section 5) is a separate, independent lever from quantizing the model weights. You can run FP16 weights with an INT8 KV cache, or INT4 weights with a full-precision cache — they're orthogonal decisions, and treating them as one knob is a common planning mistake.

3. Quantization Mechanics: Trading Precision for Footprint

Quantization takes weights stored as continuous 16-bit floating-point values and maps them onto a much smaller set of discrete low-bit integers — 8-bit, 4-bit, even lower. Done carelessly, this destroys the fine-grained relationships that attention mechanisms depend on. Done well, modern quantization schemes preserve model behavior remarkably closely while cutting memory requirements by 2–4x.

The major formats

GGUF (llama.cpp ecosystem). The most flexible format for hybrid CPU/GPU offloading and unified memory architectures — this is what you want on Apple Silicon or any setup where you're falling back to system RAM. GGUF's whole design philosophy is “run anywhere,” and it's usually the right default if you're not committed to pure-GPU inference.
EXL2 (ExLlamaV2). Built purely for NVIDIA GPUs. Supports variable bit-rate precision per layer, which lets you squeeze out maximum tokens/sec when every layer fits in VRAM. If you have a dedicated GPU setup and no CPU-offload requirement, EXL2 generally outperforms GGUF on raw throughput.
AWQ and GPTQ. The standard 4-bit formats for dedicated VRAM inference engines like vLLM and TGI. These are the formats you'll see in production serving contexts more than hobbyist local setups.

Quant levels vs. perplexity loss

Not all “4-bit” quantization is equal, and the naming conventions (particularly in the GGUF ecosystem) encode meaningful differences:

Q8_0 — near-lossless, minimal memory savings relative to FP16. Use when quality matters more than footprint and you have the VRAM to spare.
Q4_K_M — the sweet spot for most local deployments. At roughly 4.5 bits per weight, perplexity increase versus full precision is negligible for the vast majority of use cases.
Q2_K — extreme compression, reserved for genuinely tight memory constraints. Degradation here is noticeable, not theoretical — expect the model to make more reasoning errors and lose coherence on longer outputs.

If you're unsure where to start: Q4_K_M is the default worth reaching for first. Only drop lower if you've confirmed the model won't fit any other way.

4. Hardware Architectures & Offloading Strategies

Single GPU (RTX 3090 / 4090, 24GB)

This is the most common consumer setup, and it has real boundaries worth knowing before you pick a model:

34B models at Q4_K_M fit comfortably with room for reasonable context length.
70B models are possible, but only with aggressive quantization and a restricted context window. You're trading context length for parameter count — there's no way around that tradeoff on a single 24GB card.

Multi-GPU strategies

Once you're running two or more cards, the question becomes how work gets split between them:

Tensor parallelism splits individual layers across GPUs, requiring fast interconnect since every forward pass needs cross-GPU communication.
Pipeline parallelism assigns different layers to different GPUs entirely, trading some latency for much lower interconnect bandwidth requirements.

For consumer builds, PCIe lane bandwidth is usually the limiting factor. Dual RTX 3090s over standard PCIe will work, but NVLink (where supported) meaningfully reduces the tensor-parallelism penalty. If you're stuck on PCIe-only, pipeline parallelism is often the more forgiving choice — it doesn't demand the same round-trip communication on every layer, so a slower interconnect hurts you less.

Worth naming explicitly: none of this is exclusive to matched hardware. Mixing consumer cards — say, a 3090 and a 4090 in the same box — works fine for pipeline parallelism, since each GPU just needs to hold and process its assigned layers. Tensor parallelism is far less forgiving of mismatched cards, since it depends on synchronized, near-identical compute timing across GPUs to avoid one card sitting idle waiting on another.

CPU + GPU layer offloading

This is where llama.cpp and Ollama earn their reputation as the flexible option. The --gpu-layers (or -ngl) parameter lets you explicitly decide how many of the model's layers live in VRAM versus system RAM.

The catch: tokens-per-second drops sharply once you cross meaningful weight volume over into system RAM, because you're now bottlenecked by PCIe transfer speed rather than GPU compute. Offloading a few layers to RAM to fit a model that's just barely too big is a great trade. Offloading half the model is a much worse one — expect a steep throughput cliff, not a linear slowdown.

The unified memory alternative

Apple Silicon — M-series Max/Ultra chips with 64GB to 192GB of unified memory — sidesteps the VRAM/RAM split entirely. Because CPU and GPU share the same physical memory pool at high bandwidth, you can load much larger models than a discrete GPU with equivalent nominal memory would allow, without the PCIe offload penalty. For anyone building a dedicated local inference box rather than repurposing a gaming rig, this is worth serious consideration.

5. Software Stack & Runtime Optimization

Choosing an inference engine

llama.cpp / Ollama — maximum hardware flexibility, native CPU offloading, lowest barrier to entry. The right starting point for almost anyone.
ExLlamaV2 — maximum throughput for pure-GPU setups where every layer fits in VRAM.
vLLM — built for batch processing and multi-user serving, with PagedAttention enabling dynamic, efficient KV cache management. Overkill for a single-user local setup, essential if you're serving multiple concurrent requests.

Context management

FlashAttention-2 / FlashDecoding — enable these where supported. They reduce memory overhead and improve throughput for attention computation without changing model outputs.
Context shift vs. context compression. RoPE scaling and YaRN let you extend a model's effective context window beyond what it was trained on, at some cost to coherence at the extremes.
Quantizing the KV cache itself. Running an 8-bit or 4-bit KV cache reclaims meaningful VRAM during long conversations — this is often the difference between a context window that fits and one that doesn't, independent of the model weight quantization you've already chosen.

6. Step-by-Step Optimization Walkthrough

Scenario: Running a 70B parameter model on a single 24GB VRAM GPU plus 64GB DDR5 system RAM.

1. Select format. Q4_K_M GGUF (or the EXL2 equivalent bit-rate if you're pure-GPU).
2. Calculate GPU vs. CPU layer allocation. For a 70B model at this quant level, something like 40 layers on GPU and the remainder on system RAM is a reasonable starting split — exact numbers depend on the model's layer count and your available VRAM after overhead.
3. Enable FlashAttention and 8-bit KV cache in your launch configuration to reclaim additional headroom.
4. Benchmark. Watch tokens/sec output alongside nvidia-smi VRAM allocation, and adjust the GPU/CPU layer split iteratively until you find the point where you're not leaving VRAM unused but also not spilling into a throughput cliff.

This is iterative by nature — the “right” layer split is specific to your exact model, quant level, and context length, and small adjustments can produce outsized throughput differences.

A practical note on reading the benchmark: if throughput drops sharply after increasing GPU layer count by even one or two layers, you've likely crossed into VRAM overflow territory, where the driver is quietly falling back to slower memory paths rather than failing outright. That cliff is diagnostic — back off one or two layers and re-benchmark before concluding your hardware simply can't handle the model. More often than not, the ceiling is a few layers lower than where the first failure appeared to happen.

7. Conclusion & Future Outlook

The checklist, condensed:

Select target model → calculate VRAM + KV cache overhead → pick optimal quantization scheme → tune context settings.

That sequence, worked through deliberately, is what separates “this model won't fit” from “this model runs at a usable speed on hardware you already own.”

What's next for local inference: speculative decoding is already narrowing the throughput gap between quantized and full-precision generation. Sub-4-bit techniques like BitNet b1.58 point toward a future where the precision/footprint tradeoff looks very different than it does today. And Mixture-of-Experts sparsity — where only a fraction of total parameters activate per token — is quietly redefining what “parameter count” even means for memory planning, since a model's listed size and its active footprint per forward pass are increasingly two different numbers. Local inference in a year will probably not be bound by the same walls this post just spent several thousand words explaining how to work around.

https://books.apple.com/us/audiobook/system-restore-urban-re-index/id6771120724

https://books.apple.com/us/audiobook/the-hard-handover-an-owners-manual-for-the-2026-transition/id6771120894

https://books.apple.com/us/audiobook/strike-beirut/id6792928238

https://books.apple.com/us/audiobook/e-drive-the-good-ghost-a-tale-of-choice/id6791204604

https://youtube.com/shorts/nDIJqPg3H5A?is=tkrxq3JO62Mxa8Hy

Ciphergrid.net

Ingramspark.com

Reddit.com