Local LLM@Heggli

My own LLM, at home

I have a GPU box in the homelab. An RTX 3080 with 10 GB of VRAM and a Core Ultra 7 265KF. For a long time it was mostly there to be a GPU box.

Now it runs a local LLM, and it is live right now. The model and the chats run on my own hardware.

It is all in one Git repo, if you want to run it yourself: github.com/JEA666/local_LLM

What is running

It is all Docker Compose. One box, so no Kubernetes.

  • llama.cpp serves the model
  • Open WebUI is the chat page
  • SearXNG does the web search
  • Dashy is the front page
  • Caddy takes care of HTTPS
  • Prometheus and Grafana keep an eye on the GPU and the containers
The Dashy portal for the local LLM stack, with tiles for OpenWebUI, Anonymous Search, Docs, Drone Guide, System Overview, Grafana and the Admin Panel

Local LLM dashy portal

You can’t get a real certificate for a .home name, so a script makes my own certificate authority instead. Chat accounts are created by hand and signup is turned off.

Grafana has one System Overview dashboard. The golden signals are on top, then RED for the LLM and Caddy, and USE for the host, the GPU and the containers.

The System Overview dashboard in Grafana, with the golden signals on top, then panels for the LLM and Caddy, then panels for the host, the GPU and each container

System Overview in Grafana (click for full size)

The model

The model that is live is Qwen3.6-35B-A3B, with a 128K context window. Ten gigabytes of VRAM does not hold 35 billion parameters, that is obvious. But it is a mixture-of-experts model, and only a few of the experts are active for each token. So 39 layers of experts sit in RAM and the GPU does the rest.

I tried a dense 32B model on it. 2.23 tokens per second. Unusable! The mixture-of-experts model that is live now does about 53 tokens per second on a short prompt. When an agent has filled up the context it is closer to 25.

Same machine, same GPU. Just a different kind of model.

Swapping the model is one line in a config file, and there is a small admin page for it too. It also runs a hardware scan and the benchmark.

The admin panel with three cards. Switch model, with the current model and a list of model files. Hardware scan. Benchmark, with a choice of iterations

The admin panel

Kimi uses it as a swarm node

Kimi Code is one of the coding agents I use. The local model is set as its secondary model, so when Kimi spins up sub-agents, or a whole agent swarm, they run on the local model by default. The main model stays in the cloud and does the thinking.

The local one gets the simple, mechanical jobs. Searching the code for a string, reading a file that is already known, that kind of thing.

What is next

I am looking at upgrading the homelab with an R9700 card. It has 32 GB of VRAM instead of 10, so a lot less of the model would have to live in RAM.

But I am a bit distracted by a drone build project at the moment!