← All projects
Active October 2026

A Local AI at Home

An AI assistant that runs on my own PC, not in the cloud: a web interface for the whole family and a terminal agent that works on code — on a 12 GB graphics card, without the official support you'd expect.

llama.cppVulkanDockerWSL2Open WebUIOpenCodeSelf-hosting

The idea

The best AI assistants live in the cloud, and every question leaves the house. I wanted to try the alternative: an AI that runs in my own home, on a PC I already have, for two uses. One is a chat-style web interface, for me and the family, with documents to consult. The other is a terminal agent that works on a project — reads files, edits them, compiles, runs tests — in C, C++, Java, .NET, Python and Node, this very website included.

The problem: the “wrong” graphics card

The PC has a Ryzen 9, 128 GB of RAM and a Radeon RX 6700 XT with 12 GB. The RAM is plenty; the 12 GB of video memory is the real limit. And AMD’s compute software (ROCm) doesn’t officially support this card on Windows, and under WSL it only works on newer cards. So the inference engine runs natively on Windows using Vulkan, through llama.cpp, and everything else lives in WSL.

Choosing models follows the same constraint: MoE (“mixture of experts”) models activate only a small part of their weights for each word, so the “experts” can sit in RAM and only the core on the card. A llama.cpp option decides precisely what goes where, which is why I preferred it over simpler tools.

How it’s built

Claude Code or OpenCode? A real test

The interesting question was which agent to use with a local model. Claude Code has the experience I already know, and the engine speaks the Anthropic protocol too, so no translator is even needed. The alternative is OpenCode, an open agent built to work with any model. Instead of trusting reputation, I gave both the same two tasks, with the same model, on isolated copies, with an hour’s limit:

TaskClaude CodeOpenCode
A script to find broken links on the site24 minutes, works but with one false positive5.5 minutes, correct
A C++ program with CMake and tests, checked by a script the models can’t seenot finished in an hour24 minutes, correct

The difference wasn’t down to configuration — prompt caching worked in both — but to how the agent works: Claude Code, designed for Claude models, makes more turns and more rewrites, and with a slower local model that costs dearly. OpenCode is the main client. I also use it from the Mac on local projects, and hand it the memory I’d already prepared for Claude (global and per-project instruction files), most of which it already reads on its own.

A caveat about the test. While Claude Code was working on the C++ task, the graphics card was reset twice by the Windows watchdog (see below): that result is therefore at least partly skewed, and isn’t a final verdict. The rest of the picture holds: OpenCode completed both tasks correctly, including afterwards.

A graphics card that kept resetting

Windows checks that the graphics card answers within two seconds, and if it doesn’t, restarts it. With models split between the card and RAM, some Vulkan operations took longer: the card was reset (seven times in one day) and the request in progress failed with a “device lost” error. The fix was to lengthen those timeouts in the Windows registry, and after a reboot it hasn’t reset again.

Double the speed, with a limit

With multi-token prediction (MTP) the model proposes two words at a time and the model checks them: about 85% are accepted, and writing goes from 16 to 33-37 tokens a second. The price is that it handles one request at a time, so the others queue. In the comparison, with a single run, the agent’s variability (how many attempts it makes) weighed more than generation speed: the C++ task with OpenCode succeeded in 36 minutes instead of 24. I chose MTP as the main configuration, keeping the non-MTP version as a fallback. The bottleneck that remains is prompt reading, and there it would take a card with more memory.

Searching documents, and writing them

Open WebUI can answer questions about uploaded documents, and to do that it must “understand” the text: it needs an embedding model. The default one is built for English, so I replaced it with a small multilingual one that runs on the processor next to the chat model without unloading it; search combines meaning with word matching. A catch: this version stores all settings in its database and environment variables only count at first start, so I wrote a script that changes them with the container stopped, after a safety copy. That’s also how I found the interface’s address was still the old one, without HTTPS.

To test document writing I asked both models for the same usage guide for the family. The result is a single guide, converted to PDF and Word with pandoc, where I corrected the places the models had invented details — a useful reminder: they write well, but the facts need checking.

Backups for a PC that sleeps at night

A fixed-time backup makes no sense on a machine that’s off most of the time. Here a timer fires fifteen minutes after boot and then every four hours: it copies the accounts, history, uploaded documents and agent configuration to the NAS (with a safe copy of the SQLite databases, keeping the last 14), and skips the round if the NAS doesn’t answer. If a backup fails, an email arrives — I tested it with a simulated fault. Then I actually verify it: complete contents and an integrity check on the databases. Only the models stay out, since they can be re-downloaded.

Access to handle with care

OpenCode also has a web interface, handy, but it grants access to a shell, files and the GitHub key. So I keep it listening only on the PC itself, with a password, and open it from the Mac through an SSH tunnel: it’s never exposed to the network.

Realistic expectations

These models are fine for contained edits, explanations, documentation, tests and scripts. On long jobs across many files they’re clearly below cloud models: they get tool calls wrong more often and lose the thread after many steps. It’s not a replacement, it’s a complement that costs only electricity.

What I learned

Status

Running since 17 October 2026 (and already tweaked the same day): engine, models, web interface with HTTPS, development environment, two agents (one on the Mac too) and backups with alerts are up, and the comparison is done. The family usage guide exists; automatic updates remain. The configuration is all in a private repository, without the models, which can be re-downloaded.

Stack

llama.cpp (Vulkan) · llama-swap · Open WebUI · OpenCode · Claude Code · WSL2 · Docker · Windows 11 · PowerShell · Bash