Choose a local language model that fits the job and the machine.

Size for memory first

Everything else is negotiable; memory is not. Model weights, the key-value cache that grows with your context window, and the runtime’s working buffers all have to fit in the memory pool the model runs from — GPU VRAM, Apple silicon unified memory, or system RAM on CPU. A model that does not fit either fails to load or spills to slower memory and crawls.

Leave 3–4 GB for the operating system and other applications on a laptop. When a model only just fits, the next quantisation down is usually faster and more stable than running at the edge of memory.

Quantisation, explained briefly

Models are trained in 16-bit or higher precision. Quantisation stores the weights with fewer bits, shrinking the file and speeding up memory-bound inference. The naming on Hugging Face follows llama.cpp: Q8_0 keeps almost all of the original quality; Q6_K and Q5_K_M are excellent compromises; Q4_K_M is the common default and where most people should start; Q3 and Q2 variants exist to squeeze a model onto hardware that cannot otherwise run it, at a visible cost in coherence. The “K_M” suffix denotes a mixed scheme that protects the most sensitive layers.

A useful rule: a larger model at Q4 usually beats a smaller model at Q8 on reasoning, while the smaller model at Q8 is faster and often better at narrow tasks like classification. Test both when they fit.

Context windows and what they cost

The context window is how much text the model can consider at once — your conversation, retrieved documents, tool output. Larger windows are convenient but consume memory for the KV cache and slow down the first token. In Glyph you can set the working context per model; for chat and retrieval over a few documents, 8K–16K tokens is usually plenty, and reserving 32K or more only makes sense when you regularly feed long material and have the memory for it.

Choosing by task

Model families differ more by training focus than by parameter count. Start with the family that matches the job, then pick the largest size that fits.

• General assistant and writing: instruction-tuned general models in the 7–14B class are the everyday choice; step up to 27–35B when nuance matters.
• Coding: code-specialised models of the same size usually outperform general ones on completion and refactoring; test on your own codebase.
• Retrieval over documents (RAG): favour models that follow instructions tightly and cite the supplied context; the private RAG page covers the pipeline.
• Vision: choose a model that ships a projector (mmproj) file; Glyph detects it and enables image input.
• Agents and tool use: pick models with native tool-calling support; Glyph’s agent tools work best with them.
• Phones: Glyph mobile runs models up to roughly 10B parameters on device; smaller is faster and cooler.

Licences and provenance

Open-weight models ship under licences that range from permissive to restrictive on commercial use, redistribution, or user counts. Read the licence on the model card before a team adopts a model, prefer official or well-known quantisers on Hugging Face, and keep a note of which repository and revision you deployed. Glyph’s library shows the source repository for every downloaded model so that record is easy to keep.

Runtimes: what Glyph ships

Glyph bundles a llama.cpp runtime for GGUF models on every platform — Metal on Apple silicon, CUDA or Vulkan on machines with a GPU, and CPU everywhere — and supports MLX models on Apple silicon. You do not install Ollama, LM Studio, or a Python environment; you choose a model and the app manages the runtime, GPU layers, and vision projectors. Glyph also exposes its own OpenAI-compatible local API for other tools, and a self-hosted Glyph Private server can front Ollama, vLLM, TGI, or LM Studio for a whole team.

A testing routine that takes an hour

Prepare five to ten prompts from your real work, including two that are hard and one that should be refused or flagged. Run them on each candidate model, note answer quality, first-token latency, and tokens per second, and keep the outputs. Then try the same prompts with a long document attached to see how the model handles context. Public benchmark scores are useful context; your own transcripts are the decision.