Run useful AI on hardware you control.

What actually runs where

A local AI setup has four moving parts: the application you type into, the runtime that executes the model, the model weights themselves, and any indexes built from your documents. In Glyph all four live on the device. The application is the Glyph desktop app; the runtime is llama.cpp (Metal on macOS, CUDA, Vulkan, or CPU on Windows and Linux) or MLX on Apple silicon; models are files in your models folder; the knowledge base index is a FAISS store on disk.

That matters because “local” claims often stop at the application. A desktop app that forwards prompts to a hosted API is a local app, not local AI. In Glyph the model picker distinguishes the two, and a cloud model only exists in the list after you add your own key for it.

Hardware: memory first, then everything else

The single number that determines what you can run is available memory for the model — unified memory on Apple silicon, GPU VRAM on a machine with a discrete NVIDIA or AMD card, or system RAM when running on CPU. Weights, the key-value cache for your context window, and the application itself all draw from it. The table gives working estimates for 4-bit quantised models; treat them as starting points and test with your own prompts.

Leave headroom. A 16 GB machine cannot give 16 GB to a model; the operating system, browser, and Glyph itself need several gigabytes. When a model barely fits, a smaller quantisation or a smaller model is usually faster and more reliable than swapping.

Model formats, runtimes, and quantisation

GGUF is the portable format for llama.cpp-family runtimes and the one you will find most often on Hugging Face. It packages weights, tokenizer, and metadata in a single file and comes in quantisation levels — Q8 is close to full precision, Q4 variants trade a little quality for half the memory, and Q2/Q3 are for squeezing a model onto small hardware. MLX is Apple’s array framework; MLX-converted models can be faster on Apple silicon and Glyph supports them alongside GGUF.

Glyph searches Hugging Face from inside the app, shows what fits your hardware, and downloads the file into your models folder. You can also place GGUF files there yourself; the app reads the model name from the GGUF header, the same way LM Studio handles sideloaded models. Vision-capable models that ship a separate projector file are detected and run with vision enabled.

The local LLM guide goes deeper on choosing between models, licences, and context windows.

What stays local in Glyph — and what does not

With a local model selected, chat, projects, personas, memories, image generation, and transcription run entirely on the device. The knowledge base parses documents, embeds them with a bundled sentence-transformer model, and stores the index locally; retrieval and the final answer never leave the machine unless you have chosen a cloud model for the answer.

Some features exist precisely to reach outside the device and say so when you enable them: web search and the embedded browser fetch pages from the internet; cloud providers send prompts to that provider under your key; connectors talk to the services they connect; GlyphNet opens a connection between your own devices; account sync and subscription checks talk to Fortaify. Crash reporting is scrubbed and can be turned off. None of these routes local-model chat content to Fortaify.

A workable first setup

Install Glyph from the download page and verify the checksum. On first run, pick a model in the 7–9B class that fits your memory and let it download. Ask it the kind of question you actually need answered — a real document summary, a real piece of code — rather than a demo prompt, and note the speed. If it is too slow, try a smaller quantisation before a smaller model; if quality is not there, step up a size class or consider a cloud model for that task only.

Then turn on what you need. Add a folder to the knowledge base if you want answers grounded in your files. Pair your phone over GlyphNet if you want the desktop model on the go. Add a provider key only when a task genuinely needs a hosted model, and keep it labelled.

Local AI across your devices

A single desktop is often the best place for the big model, and a phone is where you actually are. Glyph mobile runs phone-sized GGUF models on the device for quick tasks, and GlyphNet pairs it with your desktop so heavier requests run on your own computer over your network. Pairing is explicit, per device, and revocable. For a team that wants one shared machine with a large model, Glyph Private provides a server-side gateway with identity, per-user model access, and audit.