Running a Local Agentic Dev Setup on a Single Machine

Disclosure: This post was drafted with the help of a local LLM running on my own hardware — not Claude, not ChatGPT, not any of the usual cloud APIs. The model that produced the first pass is the same one I’m using to write code, answer questions, and drive this blog.

I went down the local LLM rabbit hole because it was cool and because it was cheaper than burning through Claude credits. I started with a single GPU desktop and it snowballed from there.

The Hardware

The inference node is a desktop built around the GPU:

• CPU: AMD Ryzen 9 9950X3D (16 cores, 3D V-Cache)
• GPU: NVIDIA GeForce RTX 5090 (32 GB VRAM)
• OS: CachyOS (Arch-based, kernel 7.1)
• Terminal: Ghostty

The 5090 pulls about 600W under sustained inference load; the whole system sits around 750–800W. That’s non-trivial for a basement build — factor in PSU overhead, cabling, and panel capacity. I have solar panels that defray the electricity, and the rig uses water cooling that routes heat down into the unfinished basement (a more honest way of saying the heat goes somewhere instead of into my living space).

llama-server and Model Choices

llama-swap is the router and process manager. The real work happens in the inference backends, which run llama-server (the serving component of llama.cpp). Each backend loads a GGUF model into VRAM, and the router dispatches requests based on what each model is good at. llama-swap JIT-loads backends on demand, manages TTL lifecycle, and unloads idle models after 30 minutes — reload takes 4–8 seconds, so you don’t waste VRAM sitting idle.

Model Selection

Qwen 3.6 27B from Unsloth’s GGUF repo, specifically the MTP variant. The MTP tuning parameters came from vLLM’s recipes for this model family. It was the best dense model that fit under a 32 GB VRAM constraint with room to spare — dense models (not MoE) benefit from raw memory bandwidth, which is the 5090’s strength. In practice it seemed to out-code the other options at this size.

Second Model: Gemma 4 31B

For chat and creative writing I use Gemma 4 31B. It consistently produces better prose than Qwen — more varied, less repetitive, more disciplined. I A/B tested them on creative writing tasks and the difference was clear enough that I dedicated a separate backend for it.

Gemma handles chat, tutoring, and long-form creative writing. Qwen handles coding. llama-swap routes between them based on which profile is active.

Quantization Math

With 32 GB of VRAM, quantization is about fitting the model while keeping quality acceptable. The NVIDIA driver and OS reserve some VRAM, so you’ve got roughly 28–29 GB available. GGUF files also carry metadata overhead (50–200 MB depending on the model), not reflected in these calculations.

Model	FP16	Q8_0	Q6_K	Q5_K_M	Q4_K_M
7B	~14 GB	~7 GB	~5.6 GB	~4.7 GB	~4 GB
14B	~28 GB	~14 GB	~11.2 GB	~9.4 GB	~7.6 GB
27B	~54 GB	~27 GB	~21.6 GB	~18 GB	~14.6 GB
32B	~64 GB	~32 GB	~25.6 GB	~21.4 GB	~17.3 GB

The VRAM–Quant–Context Compromise

Context size is the factor that ties everything together. You have a fixed amount of VRAM. The model weights take some. The KV cache takes the rest. There’s nothing left for anything else.

On this hardware, 128K is the practical ceiling for Qwen 3.6 27B. There are 256K GGUF variants floating around, but I couldn’t get them to run reliably — 256K OOMs at startup, and 150K loads but crashes inside flash attention during inference (KV + MTP + parallel slot leaves only ~270 MB headroom). 128K is the verified ceiling: it frees enough VRAM for flash attention scratch space.

My Qwen setup runs at Q6_K (~21.6 GB weights), 128K context, q8_0 KV cache, flash attention — estimated 26 GB total. The trick that makes it fit is MTP speculative decoding: a multi-token prediction model drafts the next few tokens and the main model verifies them in one forward pass. Output-identical, roughly 2x throughput. The MTP tuning (max 5 draft tokens, min acceptance probability 0.75) is the sweet spot for this model family.

Gemma 4 31B takes a different approach: Q5_K_XL (~21.9 GB) with a separate MTP drafter model (~0.5 GB in Q8_0), also q8_0 KV cache, at 48K context — estimated 28 GB. Tighter fit, but Gemma’s stronger creative writing makes the tradeoff worth it for chat.

Runtime Tuning

llama-server runtime parameters matter more than most posts acknowledge:

• Thread count: I run with -t 16 to match the CPU core count. More threads don’t help if the bottleneck is GPU memory bandwidth.
• Batch size: Default batch size works for single-request inference. Larger batches help if you’re serving multiple concurrent requests.
• GPU layers: -ngl 999 pushes all layers to the GPU. If you run out of VRAM, lowering this offloads layers to CPU — throughput drops fast, but it’s better than not running at all.
• Flash attention: Enables faster attention computation at the cost of more VRAM for scratch space. On a 5090 with 32 GB, it’s worth it — the throughput gain is significant.
• Quantized KV cache: q8_0 for both K and V caches. Halves the VRAM cost of the KV cache compared to f16 with minimal quality loss. Requires llama.cpp PR #23398 (hadamard rotation for quantized K) or draft token acceptance rate drops to near zero.

For agentic dev work, context fills fast. The harness accumulates conversation history, tool outputs, file contents — easily 16–32K tokens in a single session. When the context fills, the model starts dropping early instructions. A bigger context window is more valuable than a higher quantization level, because the alternative is the model forgetting what you asked it to do.

Vibe

Vibe is the orchestration layer. It manages backend lifecycles, co-starts sidecar services (SearXNG, Qdrant, TTS), renders per-profile config files into frontend directories, and routes everything through llama-swap. It’s not a wrapper — it’s the glue holding the whole stack together.

It’s not production-ready and probably never will be. But it works well enough that I forget it’s there.

llama-swap as Router

I run llama-swap as a persistent service on my Unraid server. The Unraid box handles general duties — file storage, media, Docker containers — and llama-swap runs alongside as a routing layer.

The main routing is between Qwen (coding) and Gemma (chat), but the actual backend fleet is more nuanced — variants with reasoning on/off for different frontends, a multimodal Gemma for OpenWebUI, and a 7B fallback model for fast drafting.

The Tools

I use different interfaces for different workflows:

• Oh My Pi — an agentic coding harness. The model gets read/write access to my repo, runs build commands, and iterates on changes. It’s not a REPL or chat window — it’s a tool that drives the terminal.
• OpenWebUI — backed by Gemma 4 31B with multimodal support. I can upload images, documents, and screenshots; it handles OCR and visual understanding. For general chat, brainstorming, and drafting text, this is the interface. It replaces the Claude web UI for my day-to-day usage.

The Hybrid Workflow

The most interesting part of the setup is how I combine local and cloud models in a single workflow.

For complex tasks, I use Claude (via the web UI or API) to write a detailed plan — architecture decisions, file-by-file breakdown, acceptance criteria — and write it to a markdown file in the repo. Then I switch to Oh My Pi with the local Qwen model and have it pick up the plan. Qwen reads the plan file, understands the scope, and executes: creating files, writing code, running builds, fixing errors, and iterating until the work is done.

This gives me the best of both: Claude’s superior planning and reasoning for the “what” and “why”, and Qwen’s free local execution for the “how”. Claude spends its credits on the thinking; Qwen does the heavy lifting for free. The plan file is the contract between them — clear enough that a less capable model can follow it faithfully.

The Plans

The desktop with the 5090 was the first node, but parts of the roadmap are already deployed:

• 7B fallback model. A Qwen 2.5 Coder 7B in Q4_K_M (~6 GB) already runs as a fast drafting and OOM fallback. When the big model can’t load or context is too long, the small model picks up the slack.
• Old RTX 3080 for small models. A 3080 (10 GB VRAM) running 7B and 14B models. A 7B runs comfortably. A 14B in Q4_K_M (~7.6 GB) fits but eats most of the VRAM — context length is limited unless you accept some CPU offloading.
• DGX Spark. The DGX Spark I’m looking at has 48 GB VRAM. That opens up 32B models in high quantization, or even a 70B in Q4_K_M (~38 GB) if you squeeze it. Q2 quantization of a 70B fits too, but the quality loss is significant — useful for rough classification or summarization, not for anything precision-sensitive. Check the exact SKU before assuming the specs.
• Multi-machine orchestration. The vibe config already points to a separate network peer for the fleet catalog. The infrastructure for adding nodes is in place; it’s a matter of hardware arriving.

4. Fine-tuning pipeline. Fine-tuning is a different beast from inference — even with LoRA/QLoRA and gradient checkpointing, you need more VRAM and compute. The DGX Spark or a multi-GPU setup would be needed for that. The idea is fine-tuning on domain-specific data: code patterns, internal APIs, project conventions.

The Tradeoffs

Running local isn’t free, and it’s not simpler.

What you gain:

• Inference happens locally. My prompts and responses don’t go through a cloud API. The models are downloaded once and run offline. For work involving financial data, medical information, or proprietary code, that matters.
• No rate limits or token budgets. I do a lot of experiments — trying different models, testing prompts, iterating on agent workflows — that would cost a fortune on cloud APIs. Locally they cost maybe 5–10 cents in electricity per hour. Over a $2,000 GPU that’s a few thousand tokens per dollar, but the hardware is a sunk cost that also does other things.
• No surprises, for the local models at least. My setup still uses Claude for planning tasks, so I’m not fully off the cloud. But the execution layer — where most of the tokens are spent — is local and immune to pricing changes.

What you give up:

• Convenience. Setting this up took a weekend of getting CUDA, llama.cpp, GGUF models, and frontend tools to talk to each other. Driver updates will break things. You will get OOM errors. Finding a GGUF that actually works (many on HuggingFace are mislabeled or corrupted) is its own rabbit hole.
• Model quality. The best cloud models are larger and better-trained than anything that fits in 32 GB of VRAM. You’re trading raw capability for control.
• Time. Maintenance is real. Debugging CUDA errors, tuning llama-server parameters, hunting down the right quantization for a new model — none of that disappears with a $20/month subscription.

The honest tradeoff: you trade convenience and model quality for control and cost-at-scale. If you’re not already doing hundreds of tokens per day, the cloud is easier. If you are, and you want to run experiments without counting costs, the local setup pays for itself.

TL;DR

Local LLMs are just wiring together a GPU, a model at the right quantization, a server, and tools that give the model hands. The pieces all exist. The bar to entry is a good GPU and a weekend of setup. Expect OOM errors, driver issues, and time spent finding models that actually work. Once it’s running, the loop is tight enough and cheap enough that you stop thinking about the tool.