At the beginning of this year I really got into the whole AI topic. I consider myself fairly late in the game. Once I finally saw some practical use-cases that help me in in daily tasks and make my day more efficient, I really started to like the topic.
In February I got my hands on OpenClaw and the bot - Crabby - has been a helpful agent since. Especially for routine tasks such as log or performance analysis on my managed server infrastructures.
At the beginning I wanted to run a self-hosted LLM (using Ollama) and then connect OpenClaw to it. Obviously I got aware of the token burning and the costs involved and wanted to have the LLM running locally. Besides the costs, there is always the data privacy topic in the back of my head. How can I anonymize data before sending it to an external AI Provider? Unfortunately I had to abandon this idea when my attempt with Ollama in a Docker container just didn't work as expected and I simply failed to connect the dots and fully understand how the self hosted LLM works (or what the right approach would be).
Fast forward a couple of months. We're now in September and I just learned about Atomic Chat. Although this sounds like a simple Chat user interface, it's far more than that. It's basically a configuration tool that lets you run LLMs locally. Without the technical overhead and Docker image tango. Let's get into this!
If you're a reader of my blog, then you know how much I care about Open Source Software. And Atomic Chat is OSS (using the Apache 2.0 license). But besides that what I really like is the fact you can download Atomic Chat for all three major Operating Systems: Windows, macOS and Linux. As I am a Linux user (surprise surprise LOL) I am obviously downloading Atomic Chat for Linux.
To make this easier for all the different Linux distributions, Atomic Chat have decided to use the "global" AppImage package. This is a single binary package that can be run on all distributions (or is at least supposed to). I have made good experiences with other software shipping as AppImage, such as RustDesk.
You can download the latest release straight from the GitHub releases page. As of writing this, the currently latest release is 2.0.37.
Once you have downloaded Atomic.Chat_$VERSION_amd64.AppImage, make sure to give it execution permission and then run it from the Terminal.
ck@linux ~/Downloads/AppImages $ wget https://github.com/AtomicBot-ai/Atomic-Chat/releases/download/v2.0.37/Atomic.Chat_2.0.37_amd64.AppImage
ck@linux ~/Downloads/AppImages $ chmod 755 Atomic.Chat_2.0.37_amd64.AppImage
ck@linux ~/Downloads/AppImages $ ./Atomic.Chat_2.0.37_amd64.AppImage
Note: If the startup shows an error about a missing libfuse.so.2, then install the required library sudo using apt install libfuse2t64. On older distribution releases, such as Ubuntu 22.04, install the libfuse2 package.
This starts up the GUI.
Atomic Chat detected my AMD Radeon RX 580 graphics card as compatible GPU, cool!
Now to the complicated part... Large Language Models!
I'm joking. Choosing a model is actually so easy, I couldn't believe my eyes. I remembered the pain with trying to get a LLM running on my machine back at the beginning of the year. Now here we are where Atomic Chat recommends me models that are compatible for my local machine (so it fits the GPU capacities). Once click - and the model is downloaded and installed.
How this works? The built-in Hugging Face browser. You might have heard of Hugging Face recently. It was just acquired by Nvidia a few days ago for a staggering amount (holy cow!). You could compare Hugging Face to an APT repository for Debian packages or to Docker Hub for Docker container images - but its a repository for Large Language Models.
I bought my current GPU (a MSI Radeon RX 580 ARMOR OC) many years ago. I actually looked it up, it was in January 2021. So this GPU is meanwhile more than 5 years old and is for today's LLMs probably the bare minimum to get an LLM running. Hence Atomic Chat recommended one of the very basic (and small) LLMs to run locally: Qwen3.5 4B.
A click on the Download button - that's it. The LLM will now be downloaded, locally started and then integrated into Atomic Chat.
Once the download is completed and started, the UI will now show the model in the chat prompt.
In the background you can see the LLM has been started using llama.ccp:
ck 18185 0.0 0.0 29652 22760 pts/1 Ss 06:45 0:00 | \_ bash
ck 18248 7.6 0.6 76252860 227256 pts/1 Sl+ 06:45 0:25 | | \_ Atomic-Chat
ck 18312 0.3 0.2 73706132 94384 pts/1 SLl+ 06:45 0:01 | | \_ ././/lib/x86_64-linux-gnu/webkit2gtk-4.1/WebKitNetworkProcess 1 25 28
ck 18316 10.7 2.0 75206320 663724 pts/1 SLl+ 06:45 0:35 | | \_ ././/lib/x86_64-linux-gnu/webkit2gtk-4.1/WebKitWebProcess 4 25 30
ck 19028 13.0 2.7 4202148 905064 pts/1 Sl+ 06:47 0:23 | | \_ /home/ck/.local/share/Atomic Chat/data/llamacpp-upstream/backends/b10809/linux-vulkan-x64/build/bin/llama-server
When you now launch a prompt in the chat window, it will use the locally running LLM. And boy, did I hear my GPU spin up its fans!
OK, great, we got a local LLM and we can use it inside the Atomic Chat user interface and prompt there. That's it?
No, that is not it! And this is where it gets really interesting!
Atomic Chat has its own OpenAI compatible API. It exposes the API (by default) on localhost on port 1337 (geeks understand...). You can find the API and its settings in the left navigation menu with a click on API.
Note: If you want to edit the API settings, they are greyed out as long as the API server is started. To modify the settings, stop the API server.
You can now talk to the LLM using this API:
ck@linux ~ $ curl -s http://localhost:1337/v1/chat/completions -H "Content-Type: application/json" -d '{ "model": "AtomicChat/qwen35-4b-Q4_K_M", "messages": [{"role": "user", "content": "Reply with the single word: Skynet"}] }' | jq
{
"choices": [
{
"finish_reason": "stop",
"index": 0,
"message": {
"role": "assistant",
"content": "Skynet",
"reasoning_content": "Thinking Process:\n\n1. **Analyze the Request:** The user wants me to reply with a single word. The word provided in the prompt is \"Skynet\".\n\n2. **Identify the Constraint:** \"Reply with the single word: Skynet\". This means the output should *only* be the word \"Skynet\".\n\n3. **Verify the Word:** Is \"Skynet\" a single word? Yes.\n\n4. **Formulate Output:** The output should be exactly \"Skynet\".\n\n5. **Final Check:** Does this meet the user's instruction? Yes.\n\n6. **Construct Response:** Skynet.cw\n"
}
}
],
"created": 1789194238,
"model": "AtomicChat/qwen35-4b-Q4_K_M",
"system_fingerprint": "b10809-5266f24da",
"object": "chat.completion",
"usage": {
"completion_tokens": 144,
"prompt_tokens": 18,
"total_tokens": 162,
"prompt_tokens_details": {
"cached_tokens": 14
}
},
"id": "chatcmpl-7byYEx2lnlSlVl3XE7Uus8gndJTT1UvX",
"timings": {
"cache_n": 14,
"prompt_n": 4,
"prompt_ms": 131.894,
"prompt_per_token_ms": 32.9735,
"prompt_per_second": 30.32738411148346,
"predicted_n": 144,
"predicted_ms": 6392.071,
"predicted_per_token_ms": 44.6997972027972,
"predicted_per_second": 22.37146614923395
}
}
Inside Atomic Chat's UI you can see the requests, too:
With the OpenAI compatible API in place, this makes it possible to connect agents to Atomic Chat - which then uses the locally running LLM in the background!
Let's prepare a reusable Python script that sends input to the local AI API:
ck@linux ~ $ sudo apt install python3-openai
ck@linux ~ $ vi scripts/ai_log_analyze.py
ck@linux ~ $ cat scripts/ai_log_analyze.py
#!/usr/bin/env python3
import sys
from openai import OpenAI
client = OpenAI(base_url="http://localhost:1337/v1", api_key="not-needed")
log = sys.stdin.read()
resp = client.chat.completions.create(
model="AtomicChat/qwen35-4b-Q4_K_M",
messages=[
{"role": "system", "content": "You are a Linux sysadmin. Explain the log briefly and suggest one next step."},
{"role": "user", "content": log},
],
)
print(resp.choices[0].message.content)
ck@linux ~ $ chmod 755 scripts/ai_log_analyze.py
What this Python script does is:
Now I can feed some local logs to the Python script, which sends it over to the LLM via the Atomic Chat API (and you'll hear the GPU fans spinning up again!):
ck@linux ~ $ sudo tail -n 50 /var/log/auth.log | ~/scripts/ai_log_analyze.py
**Explanation:**
The logs show a Linux Mint system (`linux`) with routine activity. Most entries are standard system cron jobs and a harmless GNOME keyring warning. The primary activity involves the user `ck` (often a Linux Mint update service account) executing `sudo` commands to refresh system caches, install the `python3-openai` package via `apt`, and repeatedly check the authentication log (`/var/log/auth.log`).
**Next Step:**
Verify the successful installation of `python3-openai` by running `sudo dpkg -l python3-openai` to confirm the package is correctly registered in the system.
Was this fast? No, this took several minutes, unfortunately.
Compared to quick responses that you're used to from tools/agents such as Claude, Grok, ChatGPT and others really shows what kind of large scale AI infrastructure you need to do very quick inference. On the other hand my GPU is not the newest one, so there's definitely also room for speed improvements. However, all the (sensitive) data from /var/log/auth.log remained on my local machine, it was not sent to any external AI provider. This means full data privacy and sovereignty - with the compromise of speed.
Now I wanted to go one step further and connect my already installed Claude Code via Claude Code Router (ccr) to the local LLM. I currently have Claude Code Router connected to an external AI Provider (Venice.ai) but this should also be possible with a local API. I guess. Let's try and find out!
First I enabled the API key in the Atomic Chat API:
Note: Do not forget to stop the API server to change the settings, then start again.
Now I started Claude Code Router (ccr) and then went into the settings using its UI:
ck@linux ~ $ ccr start &
ck@linux ~ $ ccr ui
This opens up the browser and launches http://127.0.0.1:3458/, where the user interface for Claude Code Router runs.
Under Providers you can add a new AI Provider (using the Add button). I chose "Other / custom API endpoint" and called the Provider "Atomic". For the API endpoint I am using http://127.0.0.1:1337/v1:
In the next step you enter the API key, that you defined in the Atomic Chat API settings:
In the background CCR connects to the API and detects the models of this AI Provider. It correctly identified the currently installed Qwen35 model:
In the final step, verify the connection (using Check Connection). Once this is done, you've successfully added a new AI Provider in CCR.
Now there's just the Agent profile missing. I created a new profile named "Atomic". In "Default model" I selected the qwen35 model from the Atomic provider:
Now with this in place, I can run ccr Atomic, and the "Atomic" profile will be loaded. Which uses the local LLM via Atomic Chat's API.
ck@linux ~ $ cd Git/check_smart/
ck@linux ~/Git/check_smart $ ccr Atomic
Again, this was not nearly as fast as working with an external AI Provider (and their large AI infrastructure), but everything was done locally. And didn't cost a dime.
I wish I came across Atomic Chat earlier. It would have helped me tremendously on my AI learning journey which started late 2025 and picked up speed in early 2026. I really love it that local LLM hosting is now made simple and the OpenAI compatible API is integrated in Atomic Chat.
But this only makes sense (and fun) if you have a powerful GPU under the hood. With my dated GPU things worked, but very slowly. I might have to invest in a newer and much more powerful GPU to integrate Atomic Chat further, especially to test interaction with our Crabby (OpenClaw) agent. But right now it's a pretty bad moment to buy GPUs ;-).
No comments yet.
AI AWS Android Ansible Apache Apple Atlassian BSD Backup Bash Bluecoat CMS Chef Cloud Coding Consul Containers CouchDB DB DNS Databases Docker ELK Elasticsearch Filebeat FreeBSD Galera Git GlusterFS Grafana Graphics HAProxy HTML Hacks Hardware Icinga Influx Internet Java KVM Kibana Kodi Kubernetes LVM LXC Linux Logstash Mac Macintosh Mail MariaDB Minio MongoDB Monitoring Multimedia MySQL NFS Nagios Network Nginx OSSEC OTRS Observability Office OpenSearch PHP Perl Personal PostgreSQL PowerDNS Proxmox Proxy Python Rancher Rant Redis Roundcube SSL Samba Seafile Security Shell SmartOS Solaris Surveillance Systemd TLS Tomcat Ubuntu Unix VMware Varnish Virtualization Windows Wireless Wordpress Wyse ZFS Znuny Zoneminder