Local AI Lab
Overview
A machine of my own instead of a subscription in the cloud. The lab runs on two GeForce RTX 4090s with 24 GB of VRAM each under Debian 13 and serves open-weight models for everyday work: smart-home automation, code, research and email.
The appeal is less about raw throughput than about control. Every request stays on my own network, every model can be swapped out, and after the hardware the running cost is electricity rather than a licence fee.
Hardware and operations
- 2× GeForce RTX 4090, 24 GB VRAM each, 48 GB combined
- Debian 13 as the base, services in Docker
- Linux server administration and self-hosting on the local network
- Reverse proxy with TLS in front of the web interfaces
Models and serving
- Ollama for switching between models quickly
- vLLM where throughput and concurrent requests matter
- llama.cpp for quantised models with a small memory footprint
- Open WebUI as the shared interface
- Models in use: Hermes, DeepSeek, Qwen
The agent
- Hermes Agent by Nous Research, self-hosted
- Persistent memory across individual conversations
- Scheduled tasks that run unattended
- Tool calling, so it can actually operate services on the network
What I use it for
- Smart-home automation
- Writing, reading and explaining code
- Research against sources on my own network
- Email and organisation
- Testing models and prompts without paying per token
Highlights
- Two GeForce RTX 4090s, 48 GB of VRAM combined, on Debian 13
- Open-weight models served locally: Hermes, DeepSeek and Qwen
- Three serving paths side by side: Ollama, vLLM and llama.cpp
- Open WebUI as the interface for chat and switching models
- Hermes Agent with persistent memory, scheduled tasks and tool calling
- All data stays on the local network, no third-party API involved
