So, I wanted to jump on the AI-dev-assistant bandwagon. Preferably with an open-source self-hosted system so I don't have to worry about usage limits or the system going away, etc.

A bit of research later, and my dev-assistant of choice is currently Tabby, which is happy to run under docker and then talk to neovim via a socket. So now I just have to set it up to auto-start somehow.

Systemd has socket activation, which is almost the same as inetd, but docker isn't nice enough to accept filehandles via a socket or pipe (which is what inetd - and thus systemd - wants)... but the systemd guys realized that lots of things wouldn't do so and thus wrote systemd-socket-proxyd to bridge the gap.

I mostly followed the method and details as (well) written up in a blog post by an atlassian dev, resulting in:

tabby-docker.sh

#!/bin/bash
exec docker run -t --rm \
  --name tabby \
  --pull always \
  --gpus all \
  -p 8151:8080 \
  -v $HOME/.tabby:/data \
  tabbyml/tabby \
  serve --model TabbyML/StarCoder-1B --device cuda

~/.config/systemd/user/tabby-docker.service

[Unit]
Description=TabbyML/tabby container

[Service]
ExecStart=/home/pj/bin/tabby-docker.sh
ExecStartPost=/home/pj/bin/waitforport localhost 8151
ExecStop=/usr/bin/docker stop tabby

~/.config/systemd/user/tabby-docker-proxy.service

[Unit]
Description=unix socket proxy to tabby-docker service

BindsTo=tabby-docker.service
After=tabby-docker.service

[Service]
ExecStart=/lib/systemd/systemd-socket-proxyd 127.0.0.1:8151

~/.config/systemd/user/tabby-docker-proxy.socket

[Socket]
ListenStream=8150

[Install]
WantedBy=sockets.target

The only tweak I found necessary was to change the Requires= in the myservice-proxy.service file to be BindsTo= so that if I manually stop tabby (via docker stop or systemctl --user stop tabb-docker), it will be restarted when a connection is next made.

The only annoyance factor is the use of two sockets: one for systemd to listen on and one for the proxy to talk to. This has two potential fixes: 1. Docker should be able to accept a connection from systemd somehow: a unix socket, a file descriptor... something. 2. Systemd should have a shortcut syntax to do all of the above. inetd and xinetd had redirect; something similar in the .socket config could obviate the need for the proxy service.

Hmm. Now that I think about it, I wonder if there's a way to wrap the proxy inside the docker container so that the unix socket could be passed via a filesystem mapping. Maybe I'll experiment another time.