Merge pull request 'ISO: flip [breadway] to the signed dl.breadway.dev/arch repo' (#5) from fix/breadway-repo-signing into main

Reviewed-on: #5

DESIGN.md

@@ -16,7 +16,7 @@ taken as current:
 | Work on `dev`; origin = GitHub | Single-trunk `main`; `stable` is a CI marker. `origin` = Forgejo, `github` = GitHub. |
 | `[breadway]` provides bakery/breadbar/bos-settings | `[breadway]` is breadlock + AUR republishes. Desktop apps are bakery. **Not shipped:** breadcast, breadarr. |
 | NVIDIA / A/B / Secure Boot / LUKS2 | NVIDIA proprietary is **unsupported**. A/B root swapping is **not implemented**. Secure Boot is **Setup Mode only** (self-signed `sbctl`). Disk encryption is **LUKS1** because GRUB cannot unlock LUKS2+Argon2id. |
-| `SigLevel = Required` on `[breadway]` | **No.** Forgejo's Arch registry has no pacman-compatible db signatures. `SigLevel = Never` is TLS only; flipping Required without a signed db breaks installs. `KEYS.asc` signs ISO SHA256SUMS, not the pacman repo. |
+| `SigLevel = Required` on `[breadway]` | **Yes, as of the signed repo.** `[breadway]` points at `https://dl.breadway.dev/arch` where `scripts/ci-publish-signed-repo.sh` detach-signs every `.pkg.tar.zst` and the db with the BOS release key (`56203B86…`, `KEYS.asc`). That key is trusted in the pacman keyring at build time, on the live medium, and on the installed target. |
 
 ---
 

README.md

@@ -174,10 +174,10 @@ dedicated release-signing key (not reused from anything else):
 5620 3B86 A110 695A E7F3  1093 4AF3 323D 678E B5E2
 ```
 
-The public half is committed at [`KEYS.asc`](KEYS.asc). That key signs
+The public half is committed at [`KEYS.asc`](KEYS.asc). The same key signs
-**ISO checksums only** — it does not sign the `[breadway]` pacman repo
+the ISO checksums **and** the `[breadway]` pacman repo — every package and
-(Forgejo's Arch registry has no pacman-compatible db signatures; that
+the db at `https://dl.breadway.dev/arch` carry a `.sig` from it, and that
-section stays `SigLevel = Never` until a signed repo exists — see
+section is `SigLevel = Required` (see
 [docs/signed-repo.md](docs/signed-repo.md)). To verify a download:
 
 ```sh
@@ -381,8 +381,9 @@ until `dl.breadway.dev/arch` exists).
 - **Snapshots assume btrfs**: the snapper/grub-btrfs tooling expects the default
   btrfs subvolume layout the installer creates. Recovery is the GRUB
   snapshots submenu, not `snapper rollback` — [docs/hardware.md](docs/hardware.md).
-- **`[breadway]` signatures**: `SigLevel = Never` until a signed repo is
+- **`[breadway]` signatures**: `SigLevel = Required` — the signed repo at
-  stood up at `dl.breadway.dev/arch`. See [docs/signed-repo.md](docs/signed-repo.md).
+  `dl.breadway.dev/arch` is live (db + every package `.sig`ned with the BOS
+  release key). See [docs/signed-repo.md](docs/signed-repo.md).
 
 ## Recovery
 

build-local.sh

@@ -25,14 +25,19 @@ OUT="${OUT:-$REPO/out}"
 STAGE=/tmp/bos-iso-stage
 rm -rf "$STAGE" && cp -a "$REPO/iso" "$STAGE"
 
-# Rewrite the [breadway] pacman repo URL to the fastest reachable address.
+# [breadway] now points at the signed public repo https://dl.breadway.dev/arch
-#   CI_BUILD=1  — container runs on hestia with --network=host; localhost:3002 is direct
+# (SigLevel = Required) — no Forgejo-registry URL rewrite needed anymore.
-#   default     — building on hermes; git.breadway.dev is flaky from there, use Tailscale
+#
-# Only ever rewrites the staged copy, never the committed pacman.conf.
+# Trust the [breadway] repo key in *this* build host's pacman keyring so
-if [ "${CI_BUILD:-0}" = "1" ]; then
+# `pacstrap` can verify [breadway] packages while assembling the airootfs.
-  sed -i 's#https://git.breadway.dev/api/packages/Breadway/arch/os#http://localhost:3002/api/packages/Breadway/arch/os#' "$STAGE/pacman.conf"
+# The same key is baked into the image at etc/pacman.d/breadway-repo.asc and
-else
+# re-trusted on the live medium / installed target (calamares/post-install.sh).
-  sed -i 's#https://git.breadway.dev/api/packages/Breadway/arch/os#http://100.66.238.26:3002/api/packages/Breadway/arch/os#' "$STAGE/pacman.conf"
+BREADWAY_KEY_FPR="56203B86A110695AE7F310934AF3323D678EB5E2"
+BREADWAY_KEY_SRC="$REPO/iso/airootfs/etc/pacman.d/breadway-repo.asc"
+if ! pacman-key --list-keys "$BREADWAY_KEY_FPR" &>/dev/null; then
+  echo "=== trusting [breadway] repo key ($BREADWAY_KEY_FPR) in the host pacman keyring ==="
+  pacman-key --add "$BREADWAY_KEY_SRC"
+  pacman-key --lsign-key "$BREADWAY_KEY_FPR"
 fi
 
 if [ "${FAST_BUILD:-0}" = "1" ]; then

docs/signed-repo.md

@@ -1,18 +1,20 @@
 # Signed `[breadway]` repo
 
-Today the ISO's `[Breadway.os.git.breadway.dev]` section is
+**Status: live.** The ISO's `[breadway]` section is `SigLevel = Required`
-`SigLevel = Never`. That is TLS-only integrity: packages come from Forgejo's
+and points at `https://dl.breadway.dev/arch/$arch`, where every
-Arch registry, which does **not** serve pacman-compatible database
+`.pkg.tar.zst` and the db carry a detached `.sig` from the BOS release key
-signatures. `KEYS.asc` signs **ISO `SHA256SUMS`** and, once published, the
+(`56203B86…`, `KEYS.asc`, `releases@breadway.dev`). That key is trusted in
-`dl.breadway.dev/arch` database. It is **not** imported as a pacman repo key
+the pacman keyring at build time (`build-local.sh`), on the live medium
-on the ISO yet. Do not flip `SigLevel` to `Required` on that section until
+(`iso/airootfs/root/customize_airootfs.sh`), and on the installed target
-a signed repo exists and has been verified; Required without signatures
+(`iso/airootfs/etc/calamares/post-install.sh`).
-breaks the ISO and every installed system.
 
-The signed repo belongs at `https://dl.breadway.dev/arch`, not on Forgejo's
+Forgejo publishing is unchanged: `package.yml` / packaging workflows still
-registry. Forgejo publishing stays as it is (`package.yml` / packaging
+PUT unsigned `.pkg.tar.zst` to Forgejo's Arch registry. The signed tree at
-workflows PUT unsigned `.pkg.tar.zst` so existing Never installs keep
+`dl.breadway.dev/arch` is rebuilt from that registry by
-working).
+`.forgejo/workflows/signed-repo.yml` + `scripts/ci-publish-signed-repo.sh`.
+
+The rest of this doc is the original stand-up / verification procedure,
+kept for reference and for re-verifying after key rotation.
 
 ## Stand up `dl.breadway.dev/arch`
 
@@ -139,15 +141,29 @@ stays; Never installs keep working. The signed tree is rebuilt by the bos
 workflow above (registry fetch + sign + `repo-add -s`), not by writing
 `/srv` from breadlock's container.
 
-## After the signed repo exists
+## The ISO flip (done)
 
-Only after `https://dl.breadway.dev/arch/x86_64/breadway.db.sig` HEADs 200
+All three steps have landed:
-and the verify commands above succeed:
 
-1. Import `KEYS.asc` into the ISO keyring (`pacman-key --add` + `--lsign-key`).
+1. **Key trusted.** The public key is committed at
-2. Point `[breadway]` `Server` at `https://dl.breadway.dev/arch/$arch`.
+   `iso/airootfs/etc/pacman.d/breadway-repo.asc`. `build-local.sh`
-3. Only then flip that section to `SigLevel = Required`.
+   `pacman-key --add` + `--lsign-key`s it into the build host keyring;
+   `customize_airootfs.sh` does the same in the airootfs;
+   `calamares/post-install.sh` re-does it in the target chroot.
+2. **`Server`** in `iso/pacman.conf` and `iso/airootfs/etc/pacman.conf`
+   points at `https://dl.breadway.dev/arch/$arch`, section renamed to
+   `[breadway]` (matching `breadway.db`).
+3. **`SigLevel = Required`** on that section.
 
-Do not do those three steps against Forgejo's registry. See
+### Re-verify after any build
-`iso/pacman.conf` and `iso/airootfs/etc/pacman.conf`. This tree does
+
-**not** change either file.
+In a VM booted from a fresh ISO:
+
+```sh
+sudo pacman -Sy            # must fetch breadway.db + .sig, no signature error
+sudo pacman -Si breadlock  # lists the [breadway] section
+sudo pacman -S --noconfirm yay-bin   # installs with no key prompt
+```
+
+Then run the installer and, on the installed system, `sudo pacman -Sy`
+again — the target keyring must already trust `56203B86…`.

iso/airootfs/etc/calamares/post-install.sh

@@ -73,15 +73,23 @@ passwd -l root || true
 # over to the target (unpackfs may skip it / perms differ), leaving the installed
 # system unable to verify package signatures — the first `pacman -Syu` then dies
 # with "keyring is not writable / required key missing". Initialise it here so a
-# fresh install can update out of the box. archlinux-keyring is already present
+# fresh install can update out of the box. archlinux-keyring verifies official
-# and is the only keyring populated — it verifies official Arch packages.
+# Arch packages; the BOS release key (56203B86…, shipped at
-# [breadway] stays SigLevel=Never (Forgejo does not serve pacman-compatible
+# /etc/pacman.d/breadway-repo.asc) verifies the signed [breadway] repo at
-# db signatures). Do not import KEYS.asc here: that key signs ISO SHA256SUMS,
+# dl.breadway.dev/arch — SigLevel = Required there, every package and the db
-# not the pacman repo; treating it as a repo key would be a lie.
+# carry a .sig from it.
 # ---------------------------------------------------------------------------
+BREADWAY_KEY_FPR="56203B86A110695AE7F310934AF3323D678EB5E2"
 if command -v pacman-key &>/dev/null; then
     pacman-key --init || echo "WARN: pacman-key --init failed"
     pacman-key --populate archlinux || echo "WARN: pacman-key --populate failed"
+    if [[ -f /etc/pacman.d/breadway-repo.asc ]]; then
+        pacman-key --add /etc/pacman.d/breadway-repo.asc \
+            && pacman-key --lsign-key "$BREADWAY_KEY_FPR" \
+            || echo "WARN: could not trust the [breadway] repo key — pacman -Sy will fail on [breadway]"
+    else
+        echo "WARN: /etc/pacman.d/breadway-repo.asc missing — [breadway] (SigLevel=Required) will not verify"
+    fi
 fi
 
 # ---------------------------------------------------------------------------

iso/airootfs/etc/pacman.conf

@@ -32,18 +32,17 @@ Include = /etc/pacman.d/mirrorlist
 # are NOT here; they are bakery-baked into /usr/local at ISO build time.
 #
 # Packages are published to the Forgejo Arch registry (group "os") by the
-# .forgejo/workflows/*.yml workflows in this repo (and breadlock's).
+# .forgejo/workflows/*.yml workflows; scripts/ci-publish-signed-repo.sh then
+# collects them, detach-signs each .pkg.tar.zst with the BOS release key
+# (releases@breadway.dev), runs `repo-add -s`, and publishes the signed db
+# at https://dl.breadway.dev/arch/$arch (signed-repo.yml).
 #
-# Forgejo's Arch package registry does not serve pacman-compatible db
+# SigLevel = Required: every package AND the db carry a .sig from key
-# signatures. SigLevel = Never is TLS-only integrity: the connection is
+# 56203B86A110695AE7F310934AF3323D678EB5E2 — the same key committed as
-# HTTPS (or rewritten to hestia's localhost:3002 in CI). breadlock (PAM)
+# KEYS.asc / etc/pacman.d/breadway-repo.asc, imported into the pacman
-# rides this repo. Do NOT flip to SigLevel = Required unless a signed db
+# keyring at build time (build-local.sh), on the live medium, and on the
-# has been verified to work — Required without signatures breaks the ISO
+# installed target (calamares/post-install.sh).
-# and every install that uses [breadway]. KEYS.asc is the ISO SHA256SUMS
-# signing key, not a pacman repo key.
 # -----------------------------------------------------------------------
-# The section name must match Forgejo's served db filename
+[breadway]
-# ({owner}.{group}.{domain}.db) — pacman fetches "<section>.db" from Server.
+SigLevel = Required
-[Breadway.os.git.breadway.dev]
+Server = https://dl.breadway.dev/arch/$arch
-SigLevel = Never
-Server = https://git.breadway.dev/api/packages/Breadway/arch/os/$arch

iso/airootfs/etc/pacman.d/breadway-repo.asc

@@ -0,0 +1,15 @@
+-----BEGIN PGP PUBLIC KEY BLOCK-----
+
+mDMEakhwGhYJKwYBBAHaRw8BAQdA/sZ/GYec5M2MD+w20mVF5tMUhGji210Dg7zL
+TAhNsg60WUJPUyBSZWxlYXNlIFNpZ25pbmcgKGdpdC5icmVhZHdheS5kZXYvQnJl
+YWR3YXkvYm9zIHJlbGVhc2VzIG9ubHkpIDxyZWxlYXNlc0BicmVhZHdheS5kZXY+
+iJYEExYKAD4WIQRWIDuGoRBpWufzEJNK8zI9Z4614gUCakhwGgIbIwUJA8JnAAUL
+CQgHAgYVCgkICwIEFgIDAQIeAQIXgAAKCRBK8zI9Z4614ggYAQDP8FTZ14i9YPKD
+ARvZuP5QaYOUFhQ8uyG0CowXKy9O0AEAqYfjnvyJI3N651pVFSNUXyP16w1kMPSs
+K0g3CLsztQ+4OARqSHAaEgorBgEEAZdVAQUBAQdAuJFuy2GHz5m9wXTm/PdSpLE9
+gERwHOLyM1OFuttrJW4DAQgHiH4EGBYKACYWIQRWIDuGoRBpWufzEJNK8zI9Z461
+4gUCakhwGgIbDAUJA8JnAAAKCRBK8zI9Z4614nzLAP9grcIFsAAeCyVKhziHmpXq
+E0Hm6FfIr4sdEf63HZkyfwD/XeKeWfb3EWvVsloJrZZ9tDmR67iK52Hwl82wfFAU
+cAo=
+=Mrh1
+-----END PGP PUBLIC KEY BLOCK-----

iso/airootfs/root/customize_airootfs.sh

@@ -0,0 +1,27 @@
+#!/usr/bin/env bash
+# Run by mkarchiso inside the airootfs chroot, after packages are installed
+# and before the squashfs is built. (archiso prints a deprecation warning for
+# this hook, but there is no non-deprecated replacement for "trust an extra
+# pacman repo key in the image keyring", and BOS ships no pacman-init.service.)
+#
+# Purpose: trust the BOS release key (56203B86…) in the image's pacman
+# keyring so the signed [breadway] repo (SigLevel = Required,
+# https://dl.breadway.dev/arch) verifies both on the live medium and — via
+# calamares' unpackfs, which copies this squashfs to the target — on the
+# installed system. calamares/post-install.sh re-does this in the target
+# chroot as a fallback (unpackfs can skip /etc/pacman.d/gnupg).
+set -euo pipefail
+
+BREADWAY_KEY_FPR="56203B86A110695AE7F310934AF3323D678EB5E2"
+KEY_FILE="/etc/pacman.d/breadway-repo.asc"
+
+pacman-key --init
+pacman-key --populate archlinux
+
+if [[ -f "$KEY_FILE" ]]; then
+    pacman-key --add "$KEY_FILE"
+    pacman-key --lsign-key "$BREADWAY_KEY_FPR"
+    echo "customize_airootfs: trusted [breadway] repo key $BREADWAY_KEY_FPR"
+else
+    echo "customize_airootfs: WARNING $KEY_FILE missing; [breadway] will not verify" >&2
+fi

iso/pacman.conf

@@ -49,18 +49,17 @@ Include = /etc/pacman.d/mirrorlist
 # are NOT here; they are bakery-baked into /usr/local at ISO build time.
 #
 # Packages are published to the Forgejo Arch registry (group "os") by the
-# .forgejo/workflows/*.yml workflows in this repo (and breadlock's).
+# .forgejo/workflows/*.yml workflows; scripts/ci-publish-signed-repo.sh then
+# collects them, detach-signs each .pkg.tar.zst with the BOS release key
+# (releases@breadway.dev), runs `repo-add -s`, and publishes the signed db
+# at https://dl.breadway.dev/arch/$arch (signed-repo.yml).
 #
-# Forgejo's Arch package registry does not serve pacman-compatible db
+# SigLevel = Required: every package AND the db carry a .sig from key
-# signatures. SigLevel = Never is TLS-only integrity: the connection is
+# 56203B86A110695AE7F310934AF3323D678EB5E2 — the same key committed as
-# HTTPS (or rewritten to hestia's localhost:3002 in CI). breadlock (PAM)
+# KEYS.asc / airootfs/etc/pacman.d/breadway-repo.asc, imported into the
-# rides this repo. Do NOT flip to SigLevel = Required unless a signed db
+# pacman keyring at build time (build-local.sh), on the live medium, and
-# has been verified to work — Required without signatures breaks the ISO
+# on the installed target (calamares/post-install.sh).
-# and every install that uses [breadway]. KEYS.asc is the ISO SHA256SUMS
-# signing key, not a pacman repo key.
 # -----------------------------------------------------------------------
-# The section name must match Forgejo's served db filename
+[breadway]
-# ({owner}.{group}.{domain}.db) — pacman fetches "<section>.db" from Server.
+SigLevel = Required
-[Breadway.os.git.breadway.dev]
+Server = https://dl.breadway.dev/arch/$arch
-SigLevel = Never
-Server = https://git.breadway.dev/api/packages/Breadway/arch/os/$arch