Architecture

Two artifacts, independent pipelines, one dashboard

Agents initiate outbound WSS — no inbound ports on Proxmox hosts.

Roadmap for this deck

1. Preflight — confirm prerequisites
2. Local build — produce the two artifacts
3. Server deploy — one-time LXC bring-up
4. First agent — prove the pipeline end-to-end
5. Test tier — 2–3 hosts for 24h
6. Full rollout — the remaining fleet
7. Rollback — because things go wrong
8. Ongoing operations — upgrades, backups, rotation
9. Go / No-Go — final sign-off

Hardware & network

SSH root access: hypervisor + every Proxmox host.

Versions & tools

Secrets plan

Three values — keep in a password manager, never in git

Tests first

If either suite is red, stop

cd server && mix deps.get && mix test
cd ../agent && mix deps.get && mix test

Never build a release from a branch with failing tests.

Hash the password

cd server
mix run -e 'IO.puts(Argon2.hash_pwd_salt("your-password"))'

Output looks like:

$argon2id$v=19$m=65536,t=3,p=4$dSB9...$x0OQ...

Server release

MIX_ENV=prod DASHBOARD_PASSWORD_HASH='placeholder' \
  mix release --overwrite

tar -czf /tmp/server_release.tgz -C _build/prod/rel server
ls -lh /tmp/server_release.tgz

Expected: ~30–60 MB tarball.

Agent binaries

cd ../agent
./scripts/build-linux.sh

Expected output:

Binaries written to /.../agent/dist:
  proxmox-monitor-agent_linux_amd64
  proxmox-monitor-agent_linux_arm64

Sanity check:

file dist/proxmox-monitor-agent_linux_amd64 | grep 'ELF 64-bit'

First build: 5–10 min. Subsequent builds: seconds (Docker layer cache).

Create the LXC

On the hypervisor:

pct create 200 \
  /var/lib/vz/template/cache/debian-12-standard_12.7-1_amd64.tar.zst \
  --hostname proxmox-monitor \
  --memory 1024 --cores 2 \
  --rootfs local-zfs:10 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp \
  --unprivileged 1 --features nesting=0 --onboot 1

pct start 200
pct exec 200 -- ip -4 addr show eth0 | grep -Po 'inet \K[\d.]+'

Save the IP as LXC_IP. Typos here cost hours.

Base packages

pct enter 200 then:

apt-get update
apt-get install -y ca-certificates curl gnupg \
  debian-keyring debian-archive-keyring apt-transport-https sqlite3

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | \
  gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' \
  > /etc/apt/sources.list.d/caddy-stable.list

apt-get update && apt-get install -y caddy
caddy version

Upload and extract the release

From your workstation:

scp /tmp/server_release.tgz root@$LXC_IP:/tmp/

Inside the LXC:

mkdir -p /opt/proxmox-monitor
tar -xzf /tmp/server_release.tgz -C /opt/proxmox-monitor
ls /opt/proxmox-monitor/server/bin/
# server  migrate  server.bat  migrate.bat

Environment file

install -d -m 0700 /var/lib/proxmox-monitor

cat > /etc/default/proxmox-monitor <<'EOF'
DATABASE_PATH=/var/lib/proxmox-monitor/monitor.db
SECRET_KEY_BASE=<paste-mix-phx.gen.secret-output>
DASHBOARD_PASSWORD_HASH=<paste-$argon2id$-hash>
PHX_SERVER=true
PHX_HOST=monitor.example.com
PORT=4000
EOF
chmod 0600 /etc/default/proxmox-monitor

Migrate & systemd

set -a; . /etc/default/proxmox-monitor; set +a
/opt/proxmox-monitor/server/bin/server eval 'Server.Release.migrate()'

sqlite3 /var/lib/proxmox-monitor/monitor.db '.tables'
# hosts  metrics  schema_migrations

Then install the systemd unit (see runbook §3.6) with:

ExecStartPre=/opt/proxmox-monitor/server/bin/server eval 'Server.Release.migrate()'
ExecStart=/opt/proxmox-monitor/server/bin/server start
Restart=always
RestartSec=5

systemctl daemon-reload && systemctl enable --now proxmox-monitor

Caddy: TLS + WSS reverse-proxy

monitor.example.com {
    reverse_proxy 127.0.0.1:4000 {
        header_up X-Forwarded-Proto {scheme}
        header_up X-Forwarded-For {remote_host}
        transport http {
            read_timeout 90s
            dial_timeout 10s
        }
    }
}

caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

Server smoke test

From anywhere on the internet:

curl -s https://monitor.example.com/health

Expected:

{"db":"ok","status":"ok","version":"0.1.0"}

Then browser:

• Open https://monitor.example.com/ → redirects to /login
• Enter your dashboard password → lands on empty overview
• "No hosts registered yet." is the expected empty state

Register in the admin UI

1. Browser → /admin/hosts
2. "Register a new host" → enter short name (e.g. pve-host-01)
3. Click Add
4. The page reveals a token — copy it now

Deploy binary + config

export HOST=pve-host-01

scp agent/dist/proxmox-monitor-agent_linux_amd64 \
    root@$HOST:/usr/local/bin/proxmox-monitor-agent
ssh root@$HOST 'chmod 0755 /usr/local/bin/proxmox-monitor-agent'

scp agent/rel/proxmox-monitor-agent.service \
    root@$HOST:/etc/systemd/system/

On the host — write the TOML config:

install -d -m 0700 /etc/proxmox-monitor /var/cache/proxmox-monitor-agent

cat > /etc/proxmox-monitor/agent.toml <<'EOF'
server_url = "wss://monitor.example.com/socket/websocket"
token = "<paste-token-from-dashboard>"
host_id = "pve-host-01"

[intervals]
fast_seconds = 30
medium_seconds = 300
slow_seconds = 1800
EOF
chmod 0600 /etc/proxmox-monitor/agent.toml

Enable and verify

systemctl daemon-reload
systemctl enable --now proxmox-monitor-agent
journalctl -u proxmox-monitor-agent -f

Expected within 10s:

agent: starting with host_id=pve-host-01
reporter: connected, joining host:pve-host-01
reporter: joined host:pve-host-01

Reload the dashboard — the card should be online (green border) with Load / RAM / Pools / VMs populated.

Verify the offline flip

ssh root@$HOST \
  'systemctl stop proxmox-monitor-agent'

ssh root@$HOST \
  'systemctl start proxmox-monitor-agent'

2–3 hosts for 24h

Pick non-critical hosts, or hosts with independent monitoring to fall back on.

What to look for overnight:

• All cards remain online
• No repeating [error] lines in server log
• Retention log line appears: retention: pruned N stale samples (starts firing after 48h)

Tests to actively run

• Reboot a Proxmox host → card goes offline, returns without intervention
• systemctl restart proxmox-monitor on the server → all agents flip offline, then green within 30s. No stuck agents.

Go / No-Go gate

Do NOT proceed to full rollout unless ALL are true for 24h

Batch loop

After 3–4 hosts by hand, batch:

for HOST in pve-host-04 pve-host-05 pve-host-06; do
  echo "Register $HOST in admin UI, paste token:"
  read -s TOKEN

  scp agent/dist/proxmox-monitor-agent_linux_amd64 \
      root@$HOST:/usr/local/bin/proxmox-monitor-agent
  scp agent/rel/proxmox-monitor-agent.service \
      root@$HOST:/etc/systemd/system/

  ssh root@$HOST "chmod 0755 /usr/local/bin/proxmox-monitor-agent &&
    install -d -m 0700 /etc/proxmox-monitor /var/cache/proxmox-monitor-agent &&
    cat > /etc/proxmox-monitor/agent.toml <<EOF
server_url = \"wss://monitor.example.com/socket/websocket\"
token = \"$TOKEN\"
host_id = \"$HOST\"
EOF
    chmod 0600 /etc/proxmox-monitor/agent.toml &&
    systemctl daemon-reload &&
    systemctl enable --now proxmox-monitor-agent"
done

After each batch of ~5: spot-check cards, filter for offline, open a random host detail.

Four escape hatches

ssh root@$HOST \
  'systemctl disable --now proxmox-monitor-agent'

systemctl stop proxmox-monitor
systemctl stop caddy

systemctl stop proxmox-monitor
rm -rf /opt/proxmox-monitor/server
tar -xzf /tmp/server_release_PREV.tgz \
    -C /opt/proxmox-monitor
systemctl start proxmox-monitor

systemctl stop proxmox-monitor
cp /var/backups/proxmox-monitor/monitor-YYYY-MM-DD.db \
   /var/lib/proxmox-monitor/monitor.db
systemctl start proxmox-monitor

Tokens survive DB restores. Metrics post-backup are lost (48h max by retention policy).

Upgrades

cd server
MIX_ENV=prod DASHBOARD_PASSWORD_HASH='placeholder' \
  mix release --overwrite
tar -czf /tmp/server_release.tgz -C _build/prod/rel server
scp /tmp/server_release.tgz root@$LXC:/tmp/

ssh root@$LXC '
  systemctl stop proxmox-monitor
  mv /opt/proxmox-monitor/server{,.old}
  tar -xzf /tmp/server_release.tgz -C /opt/proxmox-monitor
  systemctl start proxmox-monitor   # ExecStartPre runs migrate
'

scp agent/dist/proxmox-monitor-agent_linux_amd64 \
    root@$HOST:/usr/local/bin/proxmox-monitor-agent.new

ssh root@$HOST '
  mv /usr/local/bin/proxmox-monitor-agent{.new,}
  systemctl restart proxmox-monitor-agent
'

SQLite backups

Install as a cron inside the LXC — keeps 30 daily snapshots:

cat > /etc/cron.d/proxmox-monitor-backup <<'EOF'
30 3 * * * root install -d -m 0700 /var/backups/proxmox-monitor && \
  sqlite3 /var/lib/proxmox-monitor/monitor.db \
    ".backup /var/backups/proxmox-monitor/monitor-$(date +\%Y-\%m-\%d).db" && \
  find /var/backups/proxmox-monitor -name 'monitor-*.db' -mtime +30 -delete
EOF

SQLite's online-backup command is safe while the server is running.

Verify at least one run before declaring the rollout complete.

Production readiness

Secrets hygiene

Common errors

File & port cheat sheet

/opt/proxmox-monitor/server/       release tree
/etc/default/proxmox-monitor       env secrets, 0600
/etc/systemd/system/proxmox-monitor.service
/etc/caddy/Caddyfile
/var/lib/proxmox-monitor/monitor.db
/var/backups/proxmox-monitor/      daily backups

tcp 443 (caddy) → tcp 127.0.0.1:4000 (phoenix)

/usr/local/bin/proxmox-monitor-agent
/etc/proxmox-monitor/agent.toml    token, 0600
/etc/systemd/system/proxmox-monitor-agent.service
/var/cache/proxmox-monitor-agent/  Burrito unpack

no listening ports