20 Questions: What It Actually Takes to Run AI, From Cloud to Edge

Ciphers

Aug 5

# 20 Questions: What It Actually Takes to Run AI, From Cloud to Edge

*A field guide for anyone deciding whether to rent compute, buy a workstation, or run models at the edge.*

---

### The Big Picture

**1. What are the three tiers of AI deployment?**

Cloud (API access to frontier models), workstation (local GPU running mid-size open models), and edge (lightweight models on consumer or embedded hardware). Each trades capability for control.

**2. Why does this distinction matter now?**

Because open-weight models have closed the capability gap enough that self-hosting is viable — and because data sovereignty, latency, and cost have made "just call the API" a less obvious default than it was two years ago.

**3. What's the single biggest bottleneck across all three tiers?**

Memory bandwidth, not raw compute. A GPU can be fast and still starve if it can't move weights in and out of VRAM quickly enough.

---

### VRAM

**4. What is VRAM actually doing during inference?**

Holding the model's weights and the KV cache (the running memory of the conversation) so the GPU can access them without round-tripping to system RAM.

**5. How much VRAM does a model need?**

Roughly 2GB per billion parameters at FP16, 1GB per billion at 8-bit, and 0.5GB per billion at 4-bit quantization. A 70B model at 4-bit needs about 35GB.

**6. What happens if you run out of VRAM?**

The system offloads layers to system RAM or disk — inference still works, but latency climbs sharply, sometimes 10x or worse.

**7. Does context length affect VRAM separately from model size?**

Yes. Long context windows grow the KV cache, which competes with the model weights for the same VRAM pool — this is often the hidden cost people miss when speccing hardware.

---

### GPU

**8. What GPU tier maps to what use case?**

Consumer cards (RTX 4060–4070, 8–16GB) for edge and small models; prosumer cards (RTX 4090/5090, 24–32GB) for workstation-tier 30B models; data-center cards (A100/H100, 80GB+) for cloud-scale serving.

**9. Is more VRAM always better than more compute (CUDA cores/tensor cores)?**

For inference, VRAM capacity usually matters more — it determines whether the model fits at all. Compute speed determines how fast it runs once it fits.

**10. Can you combine multiple consumer GPUs instead of buying one data-center card?**

Yes, with caveats — multi-GPU setups need model-parallel software support and enough PCIe bandwidth between cards, or you lose much of the benefit to communication overhead.

**11. What about Apple Silicon?**

Unified memory architecture means the GPU and CPU share the same pool, which can make 32–128GB configurations surprisingly competitive for local inference, even without a discrete GPU.

---

### Workstation

**12. What does a serious local-inference workstation look like?**

GPU with 24–48GB VRAM, 64–128GB system RAM, NVMe storage (model weights alone run 20–140GB), and a CPU that isn't a bottleneck for tokenization and orchestration.

**13. Why would someone choose a workstation over cloud API access?**

Operational autonomy — full control over weights, fine-tuning, and inference speed — plus no per-token cost, no rate limits, and no dependency on a provider's uptime or policy changes.

**14. What's the realistic ceiling for a single high-end workstation?**

A 70B parameter model at 4-bit quantization, running at usable but not blazing speed. Beyond that, you're into multi-GPU or cluster territory.

---

### Cloud

**15. What are you actually paying for with cloud/API access?**

Zero hardware maintenance, access to frontier-scale models no consumer setup could run, and elastic scaling — but with recurring token costs and a trust dependency on the provider.

**16. When does cloud stop making financial sense?**

At sustained high-volume usage, where the per-token cost over months exceeds the amortized cost of owning hardware — the crossover point depends heavily on usage pattern, not just volume.

**17. What's the main risk unique to cloud deployment?**

Platform risk — API access can be throttled, deprecated, or repriced, and your product depends on decisions made by someone else's infrastructure team.

---

### Edge

**18. What defines "edge" versus "workstation"?**

Edge means running smaller models (7B–13B) on lower-power, often mobile or embedded hardware — 8–16GB VRAM, optimized for latency and offline operation rather than raw capability.

**19. What's the tradeoff edge deployments make?**

Capability for independence — edge models are less capable than frontier cloud models, but they run without network access, keep data fully local, and have predictable, low latency.

**20. Where is this all heading?**

Toward hybrid architectures — edge for fast, private, always-available tasks; workstation or local cluster for heavier reasoning; cloud reserved for tasks that genuinely need frontier-scale capability. Few organizations will run purely one tier.

---

*Part of the AI & Technological Frontiers series. Next: Shadow Architecture — unpacking the quantization race.*

https://books.apple.com/us/audiobook/portals/id679251155

Ingramspark.com

Reddit.com