ansible-refactoring #45

Merged
mixel merged 14 commits from ansible-refactoring into main 2026-10-17 09:03:22 +00:00
Owner
No description provided.
Root causes of the broken/hanging ansible:check, from the refactoring
audit:

- LXC guests never received the `disabled` tag (only VMs did), so
  stopped guests like forgejo-runner were still targeted and made the
  play fail UNREACHABLE. Expose the LXC `started` attribute through the
  module/stack outputs and tag stopped LXCs as disabled in
  inventory.tf, mirroring the VM ha_state logic.
- All Taskfile ansible tasks used --ask-vault-pass, which blocks or
  crashes any non-TTY run (agents, CI, background tasks). Use the
  vault-password-file per AGENTS.md convention instead.
- ansible.cfg now sets forks=20 (tag_common fits in one batch instead
  of four) and timeout=15 (LAN-only fleet).
- Add check-mode guards to roles whose command/get_url/unarchive/
  systemd tasks register variables consumed by later tasks; without
  them --check crashes with undefined-variable errors. Guards added in
  podman-host, mosquitto, zigbee2mqtt, llamacpp, pelican-panel,
  pelican-wings, forgejo-runner.
- ansible:check now declares deps: [tofu:gen-inventory] like
  ansible:site so it runs against a fresh inventory.

Note: until `task tofu:apply` is run, the state output still lacks the
LXC `started` attribute, so gen-inventory keeps producing the previous
inventory. Apply once to activate the disabled tagging.
Phase 2 quick wins from the refactoring audit:

- Delete dead code: empty ddns-updater and proxmox-lxc-trim role
  skeletons, unused handlers in jellyfin (restarts handled by
  util/podman-quadlet), mathe-trainer, and util/docker, never-notified
  "Clear config cache" handler in pelican-panel, and the duplicate
  "Restart SSH" listener in openclaw (fired via ssh-hardening's notify
  and shadowed the base/common handler).
- zigbee2mqtt: configuration.yaml carries the MQTT password and the raw
  Zigbee network key; deploy it as 0640 (was world-readable 0644) and
  mark the template task no_log.
- SSH service conflict: base/common used the `sshd` service name while
  util/ssh-hardening used `ssh`. Standardize both roles on the
  canonical `ssh` name and a single shared "Restart SSH" handler name.
  base/common keeps its lineinfile hardening because ssh-hardening
  only covers a subset of hosts.
Convert 304 bare module names across 58 files to fully qualified
ansible.builtin.* / ansible.posix.* / community.general.* names,
verified against the installed ansible-core 2.21 resolution table:

- systemd -> ansible.builtin.systemd_service (modern canonical name)
- mount -> ansible.posix.mount
- sysctl -> ansible.posix.sysctl
- filesystem -> community.general.filesystem
- deb822_repository -> ansible.builtin.deb822_repository (promoted to
  core; not present in installed community.general)
- all remaining names -> ansible.builtin.*

Zero short module names remain (YAML-aware audit of every task,
handler, block, and play in roles/ and playbooks); ansible-playbook
--syntax-check passes for site.yml and all other playbooks.
Phase 3 consistency work from the refactoring audit:

- Variable prefixes: rename role variables to <role_name>_* so every
  role follows one convention:
  * uptime-kuma: kuma_* -> uptime_kuma_*
  * pelican-wings: wings_* -> pelican_wings_* (iscsi_* stays as-is; it
    is the util/iscsi-client interface, like swapfile_* or firewall_*)
  * pelican-panel: pelican_* -> pelican_panel_*, php_version ->
    pelican_panel_php_version
  * zigbee2mqtt: mqtt_server -> zigbee2mqtt_mqtt_server,
    zigbee_adapter_usb_path -> zigbee2mqtt_adapter_usb_path
  Renames cover roles, group_vars, Jinja templates, the healthchecks
  filter plugin, kuma-validate.yml, service SKILL.md files, and tests.
- mosquitto: add defaults/main.yml (previously the only service role
  without one) and extract the hardcoded vault/config/passwd paths
  into mosquitto_* variables.
- Handler naming: convert all remaining lowercase handler names to the
  Title Case convention used by the majority of roles (e.g.
  "restart zigbee2mqtt" -> "Restart Zigbee2MQTT"). All 27 notify targets
  audited to resolve against a defined handler or listen.
- Quote style: all file modes now use double quotes.
- Booleans: unquoted yes/no values converted to true/false.
- Flush handlers: add meta: flush_handlers to forgejo-runner (before
  service verification), llamacpp, mosquitto, openclaw, paperclip, and
  zigbee2mqtt so service restarts apply before follow-up roles run
  (e.g. zigbee2mqtt now sees a restarted mosquitto in the same play).
- pelican-panel: move role-local Jinja templates (env.j2,
  pelican.conf.j2) to services/pelican/panel/ where every other role
  stores its templates.
- Also convert remaining ansible.builtin.systemd references to the
  canonical ansible.builtin.systemd_service name.
Phase 4 architecture work from the refactoring audit:

- Merge util/proxmox-swap into util/swapfile. The role gains a
  swapfile_btrfs_detect option that finds the root filesystem type
  and creates the swapfile with `btrfs filesystem mkswapfile` on
  Btrfs roots (fallocate swapfiles are unswappable there) while
  keeping the fallocate path for other filesystems. The Proxmox node
  play in site.yml now uses util/swapfile with the 4GB node sizing;
  the separate role is removed.
- pelican-wings: convert the meta/main.yml dependency list to inline
  include_role tasks (majority pattern across roles) so dependency
  handling is visible and conditional-ready.
- paperless: split the 42-task monolithic tasks/main.yml into
  validate.yml, storage.yml, configure.yml, service.yml, and
  backup.yml, with main.yml orchestrating. Task order is preserved.
- paperless: fix a latent check-mode failure the audit missed - the
  getent UID/GID probes are command tasks that check mode skips, and
  on ansible-core 2.21 the subsequent assert then evaluates its
  `register.rc != 0 or ...` guard as false, failing every --check
  run that reaches it. Mark the read-only probes check_mode: false
  (hermes/openclaw pattern).
- AGENTS.md: list the util/app-deploy and util/cleanup roles.
- Docs: new docs/TODO_ansible.md records the deferred forgejo and
  uptime-kuma quadlet consolidation (with the multi-service design
  util/podman-quadlet would need), the SSH/assh channel wedge and
  mini1 memory-pressure operational notes, and the apply-before-check
  inventory caveat. ANSIBLE_DEPLOYMENT.md now documents forks=20,
  the 15s SSH timeout, check-mode guard expectations, and the
  gen-inventory dependency. SKILL.md updates for openclaw/paperless
  (vault password file, no --ask-vault-pass), zigbee (0640 config
  deployment), and pelican (moved templates).
docker-main idles at a ~4GB working set (Stirling-PDF, Home Assistant,
Open WebUI, SearXNG, and small containers; load average 0.00 with 4GB
available inside the guest), but its 8GB allocation is half of mini1's
physical RAM. The host ran chronically overcommitted: 4GB of swap
permanently occupied by qemu/Ceph anonymous pages, sustained ~160MB/s
swap-in thrashing, 40-85% iowait, and the 128MB LXCs (pihole, garage)
developing D-state processes with unresponsive sshd - which also hung
Ansible check runs and the tofu state backend (garage lives on mini1).

Applied via Proxmox (qm set + graceful restart of VM 100) and synced
through tofu apply. mini1 immediately recovered to 6.7GB free with
swap-in back to ~22MB/s and falling; pihole and garage sshd respond
again.
Two references the Phase 3 rename pass missed because they live outside
the scanned file types:

- The uptime-kuma Quadlet templates (services/uptime-kuma/*.container)
  still referenced kuma_image/kuma_config_dir/kuma_data_dir/
  kuma_bind_address/kuma_http_port/kuma_autokuma_image/
  kuma_autokuma_data_dir, so every uptime-kuma run failed with
  "'kuma_image' is undefined". Rename to uptime_kuma_*.
- The generated inventory's zigbee adapter host var was still emitted as
  zigbee_adapter_usb_path while the role now consumes
  zigbee2mqtt_adapter_usb_path, which would silently fall back to the
  role default and break the USB passthrough path. Rename the key in
  opentofu/inventory.tf; the tofu input variable stays unchanged, so
  .env.local and the Taskfile mapping are untouched.

Verified with a --check run against uptime-kuma and zigbee2mqtt: both
green. Run `task tofu:apply` to persist the renamed inventory key.
- Convert the last three bare `user:` module references to
  ansible.builtin.user (llamacpp, pelican-wings, zigbee2mqtt). The FQCN
  audit walker had `user` in its play-keyword set (it is also a legacy
  play-level remote_user alias), which shadowed the module name; the
  keyword list no longer suppresses it and the audit is clean again.
- services/frigabun/SKILL.md: point the host swapfile description at
  util/swapfile (the proxmox-swap role was merged in Phase 4); the
  proxmox-swap play tag in site.yml is unchanged.
The paperless LXC rootfs is a 16GB RBD volume that carries all Podman
image storage (documents live on NFS). Every version upgrade left the
previous paperless-ngx generation behind - custom plus upstream base,
~2.5GB each - until four generations (7.7GB reclaimable) filled the
rootfs to 99%. Scheduled classifier training then failed every hour
with "[Errno 28] No space left on device"; each training also writes a
~370MB model to /var/lib/paperless/data on the same rootfs.

- Cleared the guest manually (removed the 3.0.5 and 3.1.3 image
  generations, vacuumed journald by 454MB): rootfs back to 62% with
  5.7GB free, and document_create_classifier completes again
  (verified: "Classifier training complete (105.6s)").
- The role now prunes superseded paperless-ngx images after each
  custom-image build, keeping only the pinned custom image and its
  pinned base (new paperless_image_base default). Guards the failure
  mode at the source: the next upgrade cannot re-fill the rootfs.
- SKILL.md gains a rootfs-capacity section with the failure signature,
  manual verification command, and the disk_size growth path.

Verified with a --check run against paperless (ok=91, failed=0).
Drop the locally built paperless-ngx image (localhost/paperless-ngx:
3.2.1-deepseek-openrouter-embedding) and move to the pinned stock
ghcr.io/paperless-ngx/paperless-ngx:3.3.1 with pull policy "missing".
Both patches the image carried are natively supported by 3.3.x:

- Separate embedding API key: PAPERLESS_AI_LLM_EMBEDDING_API_KEY has
  been an upstream setting since 3.3.0 (falls back to the main LLM key
  only when unset), replacing the embedding.py patch that read the
  variable via os.environ.
- DeepSeek thinking disable: 3.3.0 added PAPERLESS_AI_LLM_EXTRA_PARAMS,
  a JSON object merged into every LLM request via additional_kwargs -
  the exact injection point of the former client.py patch. The new
  paperless_ai_llm_extra_params default renders
  {"extra_body": {"thinking": {"type": "disabled"}}}, so DeepSeek
  V4.1-Flash still receives thinking=disabled for suggestions and
  chat. Upstream itself still sends tool_choice=required and treats
  the provider rejection as an error, so the extra parameter remains
  required; it is now plain configuration instead of an image patch.

Role changes: remove the build context, Dockerfile install, image
existence check, and build tasks; prune task now keeps only the single
pinned stock image (and will drop the old localhost/* custom images
on deploy, freeing another ~2.6GB on the 16GB rootfs). services/
paperless/Dockerfile deleted; SKILL.md rewritten for the stock image
and native settings.

Verified with --check against paperless (ok=89, changed=4, failed=0).
Deployment (image pull, webserver restart, 3.3.x DB migrations) is a
reviewed apply: run `task ansible:site-limit:paperless`.
The env template used Ansible's quote filter on the EXTRA_PARAMS JSON,
which wrapped the value in literal quotes. Paperless' JSON parser then
rejects it ("Expecting value (line 1, column 1)") and the webserver
container crashes during settings import on boot - before migrations,
so no data is affected. Env files are consumed whole-line; the raw
JSON object needs no quoting.
docs(zigbee): record outage recovery and deployment follow-ups
Some checks failed
Kuma healthcheck contracts / contracts (pull_request) Has been cancelled
6db1e38f2a
mixel merged commit 2350083be8 into main 2026-10-17 09:03:22 +00:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
mixel/infrastructure!45
No description provided.