So, I wanted to jump on the AI-dev-assistant bandwagon. Preferably with an open-source self-hosted system so I don't have to worry about usage limits or the system going away, etc.
A bit of research later, and my dev-assistant of choice is currently Tabby, which is happy to run under docker and then talk to neovim via a socket. So now I just have to set it up to auto-start somehow.
Systemd has socket activation, which is almost the same as inetd, but docker isn't nice enough to accept filehandles via a socket or pipe (which is what inetd - and thus systemd - wants)... but the systemd guys realized that lots of things wouldn't do so and thus wrote systemd-socket-proxyd to bridge the gap.
I mostly followed the method and details as (well) written up in a blog post by an atlassian dev, resulting in:
tabby-docker.sh
#!/bin/bash
exec docker run -t --rm \
--name tabby \
--pull always \
--gpus all \
-p 8151:8080 \
-v $HOME/.tabby:/data \
tabbyml/tabby \
serve --model TabbyML/StarCoder-1B --device cuda
~/.config/systemd/user/tabby-docker.service
[Unit]
Description=TabbyML/tabby container
[Service]
ExecStart=/home/pj/bin/tabby-docker.sh
ExecStartPost=/home/pj/bin/waitforport localhost 8151
ExecStop=/usr/bin/docker stop tabby
~/.config/systemd/user/tabby-docker-proxy.service
[Unit]
Description=unix socket proxy to tabby-docker service
BindsTo=tabby-docker.service
After=tabby-docker.service
[Service]
ExecStart=/lib/systemd/systemd-socket-proxyd 127.0.0.1:8151
~/.config/systemd/user/tabby-docker-proxy.socket
[Socket]
ListenStream=8150
[Install]
WantedBy=sockets.target
The only tweak I found necessary was to change the Requires= in the
myservice-proxy.service file to be BindsTo= so that if I manually stop tabby
(via docker stop or systemctl --user stop tabb-docker), it will be restarted
when a connection is next made.
The only annoyance factor is the use of two sockets: one for systemd to listen on
and one for the proxy to talk to. This has two potential fixes:
1. Docker should be able to accept a connection from systemd somehow: a unix socket,
a file descriptor... something.
2. Systemd should have a shortcut syntax to do all of the above. inetd and xinetd
had redirect; something similar in the .socket config could obviate the need for
the proxy service.
Hmm. Now that I think about it, I wonder if there's a way to wrap the proxy inside the docker container so that the unix socket could be passed via a filesystem mapping. Maybe I'll experiment another time.
