Known Gotchas & Fixes

These are the things that will waste your time if you don't know them.

Caddy APT repo GPG key expires / goes missing If sudo apt update throws NO_PUBKEY ABA1F9B8875A6661 for the Caddy repo, the signing key needs to be re-fetched:

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' \
  | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
sudo apt update

Docker APT repo generates spurious i386 warnings The Docker repo source at /etc/apt/sources.list.d/docker.sources doesn't specify an architecture, so apt tries i386 and warns that it's unsupported. Fix by pinning to amd64:

sudo sed -i '/^Components:/a Architectures: amd64' /etc/apt/sources.list.d/docker.sources
sudo apt update

mainsail.yooshtek.com not accessible externally The Caddyfile proxies mainsail.yooshtek.com to 192.168.4.200:4409 but external access is currently broken — cause unknown, likely a Mainsail/Klipper config issue restricting connections to LAN only. Mainsail is accessible directly at http://ender.lan:4409 from the LAN. Needs investigation.

Prometheus reload doesn't work via systemctl or HTTP systemctl reload prometheus and curl -X POST http://localhost:9090/-/reload both silently do nothing (the latter requires --web.enable-lifecycle in ExecStart). Always use systemctl restart prometheus, and always validate the config first with promtool check config.

Prometheus 3.x tarball is missing directories As of 3.10.0, the tarball no longer includes consoles/ or console_libraries/. If your install script does mv prometheus-3.x.x.linux-amd64/{consoles,console_libraries} /etc/prometheus/, those two mv commands will fail. This is harmless — just ignore the errors.

Graphite Exporter download URL is broken The /releases/latest/download/graphite_exporter-linux-amd64.tar.gz URL redirects incorrectly. Always use the explicit versioned URL:

https://github.com/prometheus/graphite_exporter/releases/download/v0.16.0/graphite_exporter-0.16.0.linux-amd64.tar.gz

Grafana APT install on Ubuntu 25.10 apt-key is removed in Ubuntu 25.10. Use the modern keyring method:

sudo mkdir -p /etc/apt/keyrings
wget -q -O /tmp/grafana.gpg https://apt.grafana.com/gpg.key
gpg --dearmor < /tmp/grafana.gpg | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" \
  | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt update && sudo apt install -y grafana

Blackbox Exporter YAML indentation The config is strict YAML. Modules must be indented exactly 2 spaces under modules:. A one-space difference silently breaks module lookup — probe results come back empty with no error message. Tedious to debug.

Blackbox duplicate targets in Grafana after config changes After changing target URLs in prometheus.yml, old URLs persist in Prometheus's TSDB for ~5 minutes. The Grafana variable dropdown will show duplicates. Wait 5 minutes, or restart Prometheus to flush them immediately.

qBittorrent is on port 8081, not 8080 Every piece of documentation says 8080. It's 8081. The root path (/) returns 404 unauthenticated and 403 authenticated. The correct health endpoint is /api/v2/app/version.

Immich is on port 30041 with a different API path Official docs say port 2283 and /api/server-info/ping. This installation uses port 30041 and /server-info/ping. The documented API path returns 404.

immichbox memory leak from iOS monitoring app If an iOS monitoring app (e.g. one that initiates a remote desktop session) is left connected, gnome-system-monitor can leak to 8+ GB over several days, eventually exhausting swap (4 GB) and threatening an OOM event. The Prometheus alert node_memory_MemAvailable_bytes{job="immichbox"} < 500000000 will catch this before it becomes critical. Kill gnome-system-monitor to recover the memory. Swap drains back to normal gradually after recovery — no action needed.