My own LLM, at home

I have a GPU box in the homelab. An RTX 3080 with 10 GB of VRAM and a Core Ultra 7 265KF. For a long time it was mostly there to be a GPU box.

Now it runs a local LLM, and it is live right now. The model and the chats run on my own hardware.

It is all in one Git repo, if you want to run it yourself: github.com/JEA666/local_LLM