A Local LLM Exam Drill System — Textbook to Audiobook to Retrieval Practice

Disclosure: Like the previous post, this was drafted with the help of the local Qwen model running on my own hardware. The system described here is the one I actually use to study.

I’m studying for a distilling diploma exam — short-answer and long-answer questions, no multiple choice. The traditional study loop is: read the material, close the book, try to reproduce it from memory, check what you got wrong, repeat. The bottleneck is the checking step. You need an answer key, a grading standard, and a system that resurfaces your weak spots.

So I built one.

The Architecture

The exam drill runs entirely on a single local GPU — my RTX 5090 with 32 GB VRAM. No cloud APIs, no external services. The stack:

Model: Qwen 3.6 27B (Q6_K, 128K context) acts as the coach. There’s also a Gemma 4 31B variant for comparison.
Frontend: qwen-code, an agentic terminal harness. The model gets tool access and drives a structured loop.
Tools: MCP (Model Context Protocol) tools provide questions, reveal answers, record results, and check coverage.
RAG backing: Qdrant vector store + BGE-M3 embeddings for retrieving source material when the model needs to re-teach a concept.
Lifecycle: Docker Compose brings up Qdrant and embeddings when the session starts, tears them down when it ends.

The entire configuration lives in a single YAML profile. The system prompt alone is 255 lines of carefully crafted guardrails.

The Core Insight

The system is built around one learning principle: retrieval practice. The struggle to recall something from memory — before you see the answer — is what builds durable memory. Reading or highlighting is cheap but ineffective. Making yourself produce the answer from scratch is where learning happens.

The LLM is the perfect medium for this because it can pose questions, wait for your answer, reveal the correct answer, grade your response, and schedule what comes next — all in a single conversation loop. But the model needs strict guardrails, because without them it does what models do: hallucinates, skips steps, or gives answers away.

The Drill Loop

Each round follows the same sequence:

Fetch the next question. The model calls study_next_item, which returns a question from the bank. The question comes from the official self-assessment questions — authentic exam prompts tagged by difficulty: short (single fact), progressive (multi-step), or long (integrative, closest to a real exam question). The answer is not included.
Student answers and commits confidence. You write your answer from memory, then rate how confident you are on a 0–3 scale. Confidence is critical because it identifies the dangerous case: confident-but-wrong. That’s a blindspot — the kind of thing that fails exams. You think you know it, so you don’t study it, and then you miss it on the test.
Reveal and grade. The model calls study_reveal to fetch the official grading key, then decomposes it into discrete required points. It checks your answer against each point — hit, partial, or missed. A fluent answer that misses a key fact doesn’t earn it. A point you get wrong (not just omitted) caps the score regardless of fluency.
Record the result. The model calls study_record_result with your quality score (0–5), stated confidence (0–3), and a note on the gap. The scheduling system uses this to decide what comes next.
Advance. The loop repeats with the next question.

The key constraint: the model cannot see the answer until after you’ve answered. The tools enforce this mechanically. study_next_item returns the question but not the answer. study_reveal is a separate call that happens after your attempt. There’s no way for the model to accidentally show you the answer early, because it literally doesn’t have it.

Blindspot Detection

The confidence score is the engine that makes the system effective. When you answer with high confidence (3) but get a low quality score, the system flags it as a blindspot and resurfaces it harder and faster than a simple miss.

The scheduling works like this:

Diagnostic sweep first. A fresh question from each unit, breadth-first, until you’ve seen everything once.
Blindspots interrupt the sweep. A confident-but-wrong answer comes straight back — a confident error is best corrected promptly.
Spaced re-drilling of gaps. Missed items resurface on a schedule; mastered items leave rotation but come back a day later for a verification check.
Coverage tracking. The study_coverage tool finds units you haven’t drilled, so nothing goes untested.

The model is instructed to trust the tools over its own sense of what to ask. The scheduling logic lives in the tool, not in the model’s judgment.

RAG Grounding

When you miss something badly, the model doesn’t just tell you the right answer — it re-teaches the concept from the source material. This is where the RAG stack kicks in.

The system has a vector store (Qdrant) backed by BGE-M3 embeddings, containing all the course material. When the model needs to explain a concept, it calls retrieve_cibd with a module scope, gets the relevant source text, and builds the explanation from what it returned — not from its own training data.

This is important for two reasons. First, the model’s own knowledge about distilling might be wrong or outdated. Second, the grading key is the source of truth — if the key doesn’t mention something, it’s not part of the answer, even if it’s factually correct in the real world. Exams test what the material says, not what the internet says.

The embeddings are BGE-M3, served by Hugging Face’s text-embeddings-inference container on CPU and pinned to port 14004. It takes no VRAM at all, which matters when the main model already occupies 26 GB of a 32 GB card. Qdrant runs alongside it, and both come up and down with the drill session.

Textbook to Audiobook

The drill works because you’ve already read the material. But reading is passive. What if you could listen to the entire syllabus as audiobooks — on commutes, walks, anywhere — and then be drilled on what you heard? That’s the other half of the system.

I used vamp, a pipeline engine in the vibe repo, to turn the entire textbook into audiobooks. One command processes a module from raw lesson files into MP3s, an M4B audiobook, and an EPUB study guide. The pipeline definition is a single YAML file — 550 lines of stage definitions, prompts, and routing rules.

Multi-Phase, Multi-Role

The pipeline runs through distinct phases, each with its own persona and sometimes its own model:

Vision pass. Gemma 3 27B with its vision projector describes every diagram in the source material — phase diagrams, still schematics, VLE curves. Each SVG is rasterized to 896x896 (single tile, ~256 image tokens) and described as structured JSON: labels, values, flow directions. The model’s persona here is technical illustrator, not lecturer.
Lesson processing. Qwen 3.6 27B reads each lesson’s text alongside the compacted diagram descriptions, folding visual content into prose. Output: structured JSON with lecture content, key numbers, processes, definitions, common mistakes, and exam focus areas. Temperature 0.3 for fidelity.
Unit extraction. Qwen identifies the thematic units across all lessons. Each unit becomes its own podcast downstream. The model judges appropriate duration (30-120 minutes) from content density. A parent unit sized over 90 minutes splits into three passes; 61 to 90 minutes splits into two.
Web search enrichment. SearXNG fetches real-world context — actual distillery practices, brand names, numerical specifications — for each topic the model identified. The search results are compacted from roughly 200 KB of JSON noise per module to between 22 and 104 KB of usable context.
Script writing. This is where the persona matters most. The prompt tells the model: “You are a senior university lecturer recording the audio for ONE lecture in a CIBD distilling certification course.” Each segment follows four moves — frame the concept, teach the mechanism, anchor with a worked example or real practice, tie back to the exam. Temperature 0.7 for natural speech. Each unit script targets 4,200-16,800 words depending on duration.
Chunking. The LLM splits each segment into TTS-sized chunks (100-200 tokens, 1-3 sentences), cutting at natural pause boundaries. Kokoro sounds best short; a 500-word block rushes.
Text-to-speech. Kokoro-FastAPI converts each chunk to WAV at 24kHz. Voice: af_bella (A-grade en-US female, audiobook pick). ~82M parameters, StyleTTS2, 2 to 3 GB of VRAM. Vibe activates the TTS container before this stage and tears it down after, so it doesn’t leak VRAM during the text stages.
Assembly. ffmpeg concatenates per-unit WAVs into MP3s, then bundles them into an M4B audiobook with chapters, cover art (SDXL-Turbo via ComfyUI), and metadata. The EPUB is pandoc-converted from the markdown study guide.

The whole pipeline is a directed acyclic graph of stages. Each stage declares its inputs, its capability, its output format, and retry policies. Vamp resolves the DAG, routes stages to the right backend, and handles failures. A 502 from llama-server mid-restart? Retry with exponential backoff. Kokoro returns an empty body on warm-up? 4 attempts, 10s backoff. The pipeline keeps going.

Why Different Personas Per Stage

The script-writing prompt is dramatically different from the lesson-processing prompt. One is a university lecturer writing a 90-minute lecture. The other is a technical processor extracting structured data. The same model, different personas, different temperatures, different output formats.

This is the pattern that makes pipelines powerful: you decompose a complex task into stages, and each stage gets the persona, temperature, and output contract it needs. The vision pass is deterministic (temperature 0.3). The script writer is creative (0.7). The chunker is mechanical (0.1). The drill’s grader runs with reasoning off, for reasons covered below. One model wearing many hats, or different models for different hats — the pipeline routes.

Model Candidates and Fallbacks

The pipeline config defines capability candidates. long_form (used by script writing, lesson processing, chunking) tries Qwen 3.6 27B first; if that VRAM-rejects because another session holds the GPU, it falls back to a Qwen 2.5 Coder 7B drafter. The prompt is short enough and the output small enough that the fallback works for some stages. This is the three-tier fleet in action: 27B for quality, 7B for resilience.

vibe also defines a separate creative-writing capability with Gemma 4 31B first (A/B tested for better prose) and Qwen as fallback. The audiobook pipeline doesn’t use it; the fiction pipeline in the same fleet does. Different pipelines, different model preferences, same engine.

The System Prompt

The 255-line system prompt is the most important part of the system. It defines every constraint:

Relay, not author. The model delivers questions verbatim from the tool. It never invents, paraphrases, or “improves” a question.
Grade structurally, not holistically. Break the grading key into discrete points, check each one. No gut feelings.
Feedback scaled to the gap. A clean answer gets one line. A real gap gets a full mini-lesson: the missed point, the mechanism (why it’s true), the misconception (if it was a blindspot), and a retention hook.
Teaching escape hatch. The drill is the default, but not a cage. If you ask the model to teach you something explicitly, it does — grounded in the source material, with worked examples. Then it offers to lock it in with recall questions.
Iron rule. Every answer you give gets visible feedback before the model advances. The feedback travels with the study_record_result call as that message’s text. An empty-content record call, or a new question before the prior answer was graded in writing, is the one failure this drill cannot have.

The prompt also handles edge cases: diagram questions (the model can’t see images, so it points you to the figure and grades from the text description), numbers drills (cloze cards built from official answers, graded on whether your value matches), and bank exhaustion (when all questions in scope are done, the model composes a fresh recall prompt from RAG retrieval).

Two Frontends, Same Drill

The system runs in two configurations:

MCP frontend (primary): qwen-code with MCP tools (study_next_item, study_reveal, study_record_result, etc.). This is the standard setup — the model calls tools natively.
CLI frontend (Pi variant): the Pi terminal agent with only bash enabled, calling /home/pequalsnp/.local/bin/distillery-rag study subcommands. Each prints a single JSON object to stdout. No MCP at all — pure bash. This exists because Pi has no MCP by design, and it’s a useful fallback.

Both frontends use the same system prompt and the same backend model. The tool interface is the only difference.

Why Reasoning Is Off

The drill uses the Qwen backend with --reasoning off. This is intentional. With reasoning mode enabled, the model buries tool calls in the hidden reasoning channel about 63% of the time, emitting empty-content tool calls that break the loop. Since the drill runs tool calls every round, reasoning mode is a liability here — the output must stay in visible content so the tool loop works.

The Tradeoffs

What works:

The tool-gated answers eliminate the #1 failure mode of LLM study helpers: showing the answer before you try. The model literally can’t, because it doesn’t have it yet.
Confidence tracking catches blindspots that passive review misses. You can read something and feel like you know it, until you’re asked to produce it from scratch and realize you don’t.
RAG grounding means the model teaches from the source, not from its training data. If the course says one thing and the model’s pretraining says another, the course wins.
Everything runs locally. No API costs, no rate limits, no privacy concerns.

What’s fragile:

The system prompt is 255 lines of carefully tested guardrails. It’s not portable — it’s tuned for this specific exam, this specific tool set, this specific model. Porting it to a different subject means rewriting the prompt.
The model can still fail to follow instructions. The iron rule (feedback before advancing) is the hardest constraint to enforce — the model sometimes races ahead. The system prompt tries to prevent it, but it’s not foolproof.
It’s tied to one GPU. You can’t run this on a phone or a laptop. The main model alone uses 26 GB of VRAM.

TL;DR

A local GPU can do more than run chat. A pipeline engine (vamp) can process a textbook through vision, lesson processing, web search enrichment, script writing, chunking, TTS, and assembly — each stage with its own persona and temperature — producing audiobooks and study guides. Then a model with tool access drives a spaced-repetition drill on top, enforcing that you answer before seeing the answer, grading against an official key, and resurfacing blindspots. The guardrails are the hard part: 255 lines of system prompt for the drill, 550 lines of pipeline YAML for the audiobooks. But once it’s working, it’s a study system that doesn’t sleep, doesn’t charge per token, and actually knows what you forgot.