Ansible Overview¶
Ansible manages everything below the Kubernetes layer: OS configuration, package installation, K3s installation, Tailscale enrollment, and ongoing server management.
Setup¶
This project uses uv for Python dependency management. All Ansible dependencies (including the ansible package itself) are declared in pyproject.toml.
# Install all dependencies into a local venv
uv sync
# Run any ansible command through uv
uv run ansible <args>
uv run ansible-playbook <args>
uv run ansible-lint
Configuration (ansible.cfg)¶
[defaults]
stdout_callback = ansible.builtin.default
result_format = yaml # YAML output for readability
inventory = inventory/hosts.yaml
roles_path = roles
# SOPS vars plugin — decrypts *.sops.yaml inventory vars on the fly
vars_plugins_enabled = host_group_vars,community.sops.sops
host_key_checking = False
interpreter_python = auto_silent
The community.sops.sops vars plugin is what makes SOPS-encrypted inventory variables (hosts_secrets.sops.yaml) transparent — Ansible decrypts them automatically before each run.
Inventory Structure¶
ansible/inventory/
├── hosts.yaml # Host and group definitions
├── hosts_secrets.sops.yaml # Encrypted: ansible_host, ansible_user, SSH keys
└── group_vars/
├── k3s_cluster.sops.yml # k3s version, cluster token, node config
└── all/
└── vars.yml # Non-secret group variables
Host Groups¶
| Group | Members |
|---|---|
k3s_cluster |
all K3s nodes (masters + nodes) |
masters |
k3s-m1, k3s-m2, k3s-oci-m3 |
nodes |
(K3s agent nodes — currently empty) |
raspberries |
rpi-4b, rpi-z2w-hyperion |
docker |
rpi-4b |
unraid |
hoarder |
hyperion |
rpi-z2w-hyperion |
Collections¶
Declared in ansible/requirements.yml:
| Collection | Version | Purpose |
|---|---|---|
community.sops |
2.2.7 | SOPS vars plugin + tasks |
community.docker |
5.1.0 | Docker management on rpi-4b |
ansible.posix |
2.1.0 | POSIX utilities |
community.general |
12.5.0 | General-purpose modules |
k3s-io/k3s-ansible |
1.2.0 | K3s installation and upgrade |
artis3n.tailscale |
1.2.1 | Tailscale server enrollment |
Install or update:
Roles¶
Galaxy roles install to ~/.local/share/ansible/roles (outside the repo). Custom roles live in ansible/roles/ and are committed. The roles_path in ansible.cfg covers both:
maintenance Role¶
Runs on every server via all playbooks (no tags required). Handles:
SSH Hardening (sshd_config.j2)¶
Deployed to /etc/ssh/sshd_config on every run. Key settings:
| Setting | Value | Purpose |
|---|---|---|
PasswordAuthentication |
no |
Keys only |
PermitRootLogin |
no |
No direct root access |
AllowUsers |
{{ ansible_user }} |
Only the inventory-defined user per host |
MaxAuthTries |
3 |
Disconnect after 3 failed attempts |
MaxStartups |
3:50:10 |
Rate-limit pre-auth connections: allow 3, drop 50% above that, hard cap 10 |
ClientAliveInterval |
300 |
Keepalive every 5 min |
ClientAliveCountMax |
3 |
Disconnect after ~15 min idle |
AllowTcpForwarding |
yes |
Port forwarding (tunnels) |
AllowAgentForwarding |
no |
Block agent forwarding (hijack risk) |
LogLevel |
VERBOSE |
Log key fingerprints on auth |
Config is validated with sshd -t before the service restarts.
inotify Limits¶
Sets fs.inotify.max_user_watches and fs.inotify.max_user_instances via sysctl — required for K3s and file-watching tools.
Package Management (tag: update)¶
Only runs with --tags update. Runs apt dist-upgrade, installs base/group/host packages, cleans up, and reboots if required.
hyperion Role¶
Installs and manages Hyperion.ng on rpi-z2w-hyperion. Installs via apt, disables the per-user systemd unit, and runs the service as root (hyperion@root.service). Validates port 8090 is reachable after start.
docker_compose Role¶
Deploys Docker Compose stacks to hosts in the docker group (rpi-4b) and unraid group (hoarder). Stack definitions live under docker/<inventory_hostname>/ at the repo root (e.g. docker/rpi-4b/adguard/, docker/hoarder/decide/). The role syncs each stack directory to the host, ensures volume directories exist, and manages the compose lifecycle.
Full guide, including all the Unraid-specific gotchas below: .agents/skills/unraid-docker/SKILL.md.
Unraid DockerMan Templates¶
For hosts in the unraid group, the role additionally renders one DockerMan XML template per compose service into /boot/config/plugins/dockerMan/templates-user/, so containers deployed via docker_compose_v2 show up as native, editable apps in the Unraid GUI (icon, WebUI link, port/volume/env fields) instead of "unmanaged" containers.
Compose stays the single source of truth for the container spec (image, ports, volumes, environment, labels). An optional unraid.yml sidecar file per stack supplies fields the compose spec has no equivalent for — icon, WebUI URL, category, overview, per-field descriptions/display-tier, and which env vars to mask as secrets in the GUI:
# docker/hoarder/<stack>/unraid.yml
<service-name>: # keyed by compose service name
icon: https://example.com/icon.png
webui: "http://[IP]:[PORT:8081]/"
category: "Tools: Network:Web"
overview: "Short description shown in the Unraid GUI."
project_url: https://example.com/repo # optional, -> <Project>
support_url: https://example.com/issues # optional, -> <Support>
license: MIT # optional, -> <License>
env_descriptions:
SOME_API_KEY: "Shown under the field in the edit form."
env_display:
SOME_API_KEY: advanced # 'always' (default) or 'advanced'
secret_env:
- SOME_API_KEY # masked in the DockerMan edit form, value never shown
Templates use the resolved runtime configuration and the native my-<container>.xml filename. They preserve network mode, privileges, sysctls, healthchecks, restart policy, tmpfs, devices, entrypoint, and command. network_mode: service:privoxyvpn renders as <Network>container:privoxyvpn</Network> with no second network flag. Duplicate templates for managed apps are removed. This also prevents Unraid from restoring an incompatible extra none/bridge network onto a container that shares the VPN namespace.
Variables listed in secret_env, and variables interpolated from .env, are masked in the edit form. Their current values remain in the root-readable template (mode 0600) so GUI Apply/Update preserves credentials. These resolved templates stay on the server; Git contains only encrypted secrets and template code.
Unraid startup and VPN replacement¶
Unraid's native autostart list starts independent containers before shared-network dependents. docker_compose_unraid_start_delays controls native delays after individual containers start; the default gives privoxyvpn 30 seconds. No Compose process runs at boot. When testing a Docker service restart over SSH, unset XDG_RUNTIME_DIR before invoking /etc/rc.d/rc.docker restart; otherwise the daemon can inherit a session directory that disappears on logout and breaks docker exec/healthchecks. The previous User Scripts start-containers workaround is replaced with a no-op to avoid racing DockerMan.
Docker saves a container ID for --net=container:privoxyvpn. Ansible therefore deploys namespace providers first, checks dependents against the current provider ID, and recreates dependents with stale IDs or invalid extra networks. Apps moved between stacks are reconciled by their configured container names and managed ownership. Interrupted Compose replacement containers are removed only for the affected service/project, preserving volumes.
Use Ansible for VPN image/config changes so it can rebind dependents after replacing the VPN container. Restarting an existing VPN container keeps its ID and needs no recreation. A standalone GUI replacement of the VPN cannot repair Docker's saved namespace IDs in other containers; run the Unraid playbook afterwards.
For apps sharing the VPN namespace, use IPv4 loopback for upstream connections when the VPN firewall drops IPv6; resolving localhost to IPv6 can otherwise stall application initialization.
Containers with restart: "no" are created stopped and omitted from autostart. Global container/network pruning is disabled so manual tools survive.
Gotchas learned the hard way (full detail in the skill file):
- Containers need the
net.unraid.docker.managed=dockermanlabel to show as editable in the GUI — the role injects it automatically into every service on unraid hosts, but a manually-rundocker compose upoutside Ansible won't have it. - Fresh Unraid boxes may have no Python at all — bootstrap with
un-get install python3(see skill file) before the first run. - Unraid's own management nginx squats
:8080on every interface — checkss -tlnpon the host before picking a host port for a new stack. - PUID/PGID convention on Unraid is
99:100(nobody:users), not an arbitrary app-specific UID — match the host's own data ownership.
To add a new native-looking container on Unraid: create docker/hoarder/<stack>/docker-compose.yaml (+ optional secrets.sops.yaml and unraid.yml), then run the unraid.yml playbook.
Linting¶
Pre-commit hook runs ansible-lint automatically on staged playbooks.