Booting a snapshot from the GRUB menu leaves a read-only, half-broken system

Breadway commented

2026-09-26 07:27:41 +08:00

Owner

Problem

grub-btrfs boots snapper snapshots, which are read-only. Without grub-btrfs' overlayfs initramfs hook, anything that writes to /, /etc or /var/lib fails (NetworkManager, greetd, systemd state). That makes the advertised recovery path ("reboot, pick a snapshot") unreliable. There's also no documented way to make a snapshot permanent, because snapper rollback is unsupported.

Where

iso/airootfs/etc/calamares/post-install.sh initramfs HOOKS section (lines 95-149).

Fix

Add the grub-btrfs-overlayfs hook to mkinitcpio HOOKS (after filesystems); it ships in the grub-btrfs package. It needs the udev base hook; under systemd it doesn't apply, so check which base hook is in use and document or handle both.
Add a bos-restore-snapshot <n> helper (and a Settings > Snapshots button). It should mount the top-level subvolume, rename @ to @.broken-<date>, run btrfs subvolume snapshot from @snapshots/<n>/snapshot to @, then reboot. This keeps rootflags=subvol=@ valid.
Update bos-update, bos-rescue and the site text to describe the real flow.

Acceptance

Boot a snapshot from GRUB: you reach a working, writable (overlay) desktop.
bos-restore-snapshot followed by a reboot boots the restored state from the default GRUB entry.

Source: BOS v0.6.2 source review (tag dated 2026-09-19); paths and line numbers refer to that tree. Read AGENTS.md first. One issue per branch/PR, match existing style. Verify with FAST_BUILD=1 sudo ./build-local.sh plus a VM install. Add a CHANGELOG [Unreleased] entry.

## Problem grub-btrfs boots snapper snapshots, which are read-only. Without grub-btrfs' overlayfs initramfs hook, anything that writes to `/`, `/etc` or `/var/lib` fails (NetworkManager, greetd, systemd state). That makes the advertised recovery path ("reboot, pick a snapshot") unreliable. There's also no documented way to make a snapshot permanent, because `snapper rollback` is unsupported. ## Where `iso/airootfs/etc/calamares/post-install.sh` initramfs HOOKS section (lines 95-149). ## Fix - Add the `grub-btrfs-overlayfs` hook to mkinitcpio HOOKS (after `filesystems`); it ships in the `grub-btrfs` package. It needs the udev base hook; under `systemd` it doesn't apply, so check which base hook is in use and document or handle both. - Add a `bos-restore-snapshot <n>` helper (and a Settings > Snapshots button). It should mount the top-level subvolume, rename `@` to `@.broken-<date>`, run `btrfs subvolume snapshot` from `@snapshots/<n>/snapshot` to `@`, then reboot. This keeps `rootflags=subvol=@` valid. - Update `bos-update`, `bos-rescue` and the site text to describe the real flow. ## Acceptance - Boot a snapshot from GRUB: you reach a working, writable (overlay) desktop. - `bos-restore-snapshot` followed by a reboot boots the restored state from the default GRUB entry. --- Source: BOS v0.6.2 source review (tag dated 2026-09-19); paths and line numbers refer to that tree. Read AGENTS.md first. One issue per branch/PR, match existing style. Verify with `FAST_BUILD=1 sudo ./build-local.sh` plus a VM install. Add a CHANGELOG `[Unreleased]` entry.

Breadway added the

P1

recovery

snapshots

labels

2026-09-26 07:27:41 +08:00

Breadway commented

2026-09-30 01:25:03 +08:00

Author

Owner

Investigation found a design blocker before implementation. The installed grub-btrfs package provides grub-btrfs-overlayfs as an mkinitcpio run_latehook and its own docs show it with the udev base hook. The package docs and Arch documentation state it does not work with a systemd based initramfs. BOS post-install currently keeps the systemd/sd-encrypt hook set on current images, so simply appending grub-btrfs-overlayfs would leave the default snapshot boot read-only and fail the acceptance test.

Which direction should the installer take?

Convert installed images to the udev/encrypt hook set so the packaged overlay hook works. This changes the installed initramfs base and needs validation for encrypted installs.
Keep systemd initramfs and add a BOS-maintained systemd initrd overlay implementation. This is more code and maintenance.
Limit this issue to udev systems and explicitly document that snapshot boots remain read-only under systemd. This does not meet the current writable-desktop acceptance test on BOS defaults.

The requested Settings > Snapshots restore button belongs to the separate bos-settings repo. The BOS queue scope is limited to this repo, so that UI change needs a separate BSET issue. Also, a restore helper must resolve the Btrfs device from the installed fstab or kernel command line when booted via overlay, since findmnt / then reports overlayfs rather than the underlying Btrfs device. I left the issue in TODO pending a choice on the initramfs approach and split scope.

Investigation found a design blocker before implementation. The installed `grub-btrfs` package provides `grub-btrfs-overlayfs` as an mkinitcpio `run_latehook` and its own docs show it with the `udev` base hook. The package docs and Arch documentation state it does not work with a `systemd` based initramfs. BOS post-install currently keeps the `systemd`/`sd-encrypt` hook set on current images, so simply appending `grub-btrfs-overlayfs` would leave the default snapshot boot read-only and fail the acceptance test. Which direction should the installer take? 1. Convert installed images to the `udev`/`encrypt` hook set so the packaged overlay hook works. This changes the installed initramfs base and needs validation for encrypted installs. 2. Keep `systemd` initramfs and add a BOS-maintained systemd initrd overlay implementation. This is more code and maintenance. 3. Limit this issue to `udev` systems and explicitly document that snapshot boots remain read-only under `systemd`. This does not meet the current writable-desktop acceptance test on BOS defaults. The requested Settings > Snapshots restore button belongs to the separate `bos-settings` repo. The BOS queue scope is limited to this repo, so that UI change needs a separate BSET issue. Also, a restore helper must resolve the Btrfs device from the installed fstab or kernel command line when booted via overlay, since `findmnt /` then reports overlayfs rather than the underlying Btrfs device. I left the issue in TODO pending a choice on the initramfs approach and split scope.

Breadway commented

2026-09-30 10:29:19 +08:00

Author

Owner

The snapshot-menu boot now reaches the greeter with a writable overlay, and a write under /etc succeeds. The default boot after restore still stopped in initrd; the agent was rebuilding initramfs to retest when it hit the usage limit. No active verification is running, so moving to TODO with the default-boot acceptance check outstanding.

The snapshot-menu boot now reaches the greeter with a writable overlay, and a write under `/etc` succeeds. The default boot after restore still stopped in initrd; the agent was rebuilding initramfs to retest when it hit the usage limit. No active verification is running, so moving to TODO with the default-boot acceptance check outstanding.

Breadway commented

2026-10-01 15:22:08 +08:00

Author

Owner

Created linked child issue BSET-22 for the Settings > Snapshots restore action in the separate bos-settings repository. The current BOS-26 PR will cover initramfs overlay boot, the restore CLI, and matching BOS documentation; BSET-22 tracks the UI control and confirmation flow.

Created linked child issue BSET-22 for the Settings > Snapshots restore action in the separate `bos-settings` repository. The current BOS-26 PR will cover initramfs overlay boot, the restore CLI, and matching BOS documentation; BSET-22 tracks the UI control and confirmation flow.

Breadway commented

2026-10-01 18:02:50 +08:00

Author

Owner

Isolated handoff check: I copied the issue-scoped snapshot overlay, restore CLI, installer hook, and docs changes into a separate worktree based on origin/main. The shared dirty checkout was left untouched. bash -n passed for the changed shell scripts and git diff --check passed. The VM acceptance remains unresolved: the prior run reached the greeter with a writable overlay, but default boot after bos-restore-snapshot stopped in initrd. I did not launch a VM or ISO, and cannot diagnose that failure from static checks alone. This is not ready for REVIEW; BOS-26 remains in DOING until the default-boot failure is understood and retested in an authorized headless VM.

Isolated handoff check: I copied the issue-scoped snapshot overlay, restore CLI, installer hook, and docs changes into a separate worktree based on origin/main. The shared dirty checkout was left untouched. `bash -n` passed for the changed shell scripts and `git diff --check` passed. The VM acceptance remains unresolved: the prior run reached the greeter with a writable overlay, but default boot after `bos-restore-snapshot` stopped in initrd. I did not launch a VM or ISO, and cannot diagnose that failure from static checks alone. This is not ready for REVIEW; BOS-26 remains in DOING until the default-boot failure is understood and retested in an authorized headless VM.

Breadway commented

2026-10-01 18:48:34 +08:00

Author

Owner

Follow-up investigation: the isolated implementation edits mkinitcpio HOOKS and rebuilds all presets with mkinitcpio -P; the existing static checks pass, but they do not explain why the default boot after restore stopped in initrd. There is no existing headless VM session available for a safe reproduction, and I did not launch a VM or ISO. The snapshot-menu boot had previously reached the greeter with a writable overlay, but default-boot-after-restore acceptance remains unverified and unresolved. Keeping BOS-26 in DOING with the issue-scoped changes staged in the isolated worktree. No PR opened.

Follow-up investigation: the isolated implementation edits mkinitcpio HOOKS and rebuilds all presets with `mkinitcpio -P`; the existing static checks pass, but they do not explain why the default boot after restore stopped in initrd. There is no existing headless VM session available for a safe reproduction, and I did not launch a VM or ISO. The snapshot-menu boot had previously reached the greeter with a writable overlay, but default-boot-after-restore acceptance remains unverified and unresolved. Keeping BOS-26 in DOING with the issue-scoped changes staged in the isolated worktree. No PR opened.

Breadway commented

2026-10-01 19:07:37 +08:00

Author

Owner

Headless VM acceptance attempt: there is no existing test VM or QCOW2 image, and the host has no accessible /dev/kvm. I ran the documented sudo FAST_BUILD=1 ./build-local.sh from the isolated BOS-26 worktree to produce a current ISO, but it stopped before archiso because the builder is missing all 18 required bakery binaries (bakery, bread, BOS apps, etc.). The only available BOS ISO is dated 2026-09-18, before this issue's changes, so booting it would not test the snapshot overlay/restore implementation. No VM or GUI session was launched. BOS-26 remains in DOING with default-boot-after-restore acceptance unresolved; producing a valid current image requires staging the required bakery binaries.

Headless VM acceptance attempt: there is no existing test VM or QCOW2 image, and the host has no accessible /dev/kvm. I ran the documented `sudo FAST_BUILD=1 ./build-local.sh` from the isolated BOS-26 worktree to produce a current ISO, but it stopped before archiso because the builder is missing all 18 required bakery binaries (`bakery`, `bread`, BOS apps, etc.). The only available BOS ISO is dated 2026-09-18, before this issue's changes, so booting it would not test the snapshot overlay/restore implementation. No VM or GUI session was launched. BOS-26 remains in DOING with default-boot-after-restore acceptance unresolved; producing a valid current image requires staging the required bakery binaries.

Breadway commented

2026-10-01 19:15:49 +08:00

Author

Owner

Built a fresh headless-test ISO at /tmp/bos26-out/bos-2026.10.01-x86_64.iso after staging the 18 required binaries and breadhelp data with scripts/ci-stage-bakery.py. The script verified the stable minisign signature and artifact SHA-256 checksums; staging used current stable package versions in a temporary lockfile because several older lockfile pins no longer publish complete artifacts. The resulting ISO is bootable-format and contains the BOS-26 restore helper and initcpio plugin. During mkinitcpio, ArchISO also reported binary not found: nbd-client; mkarchiso still produced the ISO. I attempted to start QEMU headlessly on a disposable qcow2 disk, but automatic approval rejected standalone VM launch under the standing rule that VMs may only run in an already-running headless nested session. No VM started. Runtime snapshot/restore acceptance remains unverified pending explicit permission to run a standalone headless QEMU guest or an existing nested VM session.

Built a fresh headless-test ISO at `/tmp/bos26-out/bos-2026.10.01-x86_64.iso` after staging the 18 required binaries and breadhelp data with `scripts/ci-stage-bakery.py`. The script verified the stable minisign signature and artifact SHA-256 checksums; staging used current stable package versions in a temporary lockfile because several older lockfile pins no longer publish complete artifacts. The resulting ISO is bootable-format and contains the BOS-26 restore helper and initcpio plugin. During mkinitcpio, ArchISO also reported `binary not found: nbd-client`; mkarchiso still produced the ISO. I attempted to start QEMU headlessly on a disposable qcow2 disk, but automatic approval rejected standalone VM launch under the standing rule that VMs may only run in an already-running headless nested session. No VM started. Runtime snapshot/restore acceptance remains unverified pending explicit permission to run a standalone headless QEMU guest or an existing nested VM session.

Breadway commented

2026-10-02 11:12:00 +08:00

Author

Owner

Implementation and runtime acceptance are ready for review in PR #83: #83 (head 0aa5be816ca513820a814521bd82e9bbe1e7482d). A headless UEFI QEMU guest booted the generated GRUB snapshot entry for read-only snapshot 1; / was a writable overlay, /etc writes succeeded, and the source snapshot stayed read-only. sudo bos-restore-snapshot 1 with the exact RESTORE 1 confirmation restored snapshot 1 to @, preserved the old root as @.broken-20261002-030155, rebooted via the default GRUB entry, and verified writable Btrfs @, restored marker, and a successful /etc write. The shipped bos-copy-kernel and post-install script ran in a disposable guest. The interactive Calamares UI path, BIOS boot, encryption, and udev initramfs path were not exercised. Evidence and screenshots are preserved in ignored out/bos26-vm/ in the isolated worktree. Issue moved to REVIEW.

Implementation and runtime acceptance are ready for review in PR #83: https://git.breadway.dev/Breadway/bos/pulls/83 (head `0aa5be816ca513820a814521bd82e9bbe1e7482d`). A headless UEFI QEMU guest booted the generated GRUB snapshot entry for read-only snapshot 1; `/` was a writable overlay, `/etc` writes succeeded, and the source snapshot stayed read-only. `sudo bos-restore-snapshot 1` with the exact `RESTORE 1` confirmation restored snapshot 1 to `@`, preserved the old root as `@.broken-20261002-030155`, rebooted via the default GRUB entry, and verified writable Btrfs `@`, restored marker, and a successful `/etc` write. The shipped `bos-copy-kernel` and post-install script ran in a disposable guest. The interactive Calamares UI path, BIOS boot, encryption, and udev initramfs path were not exercised. Evidence and screenshots are preserved in ignored `out/bos26-vm/` in the isolated worktree. Issue moved to REVIEW.

Breadway closed this issue

2026-10-02 14:36:49 +08:00

Rows
Columns

Booting a snapshot from the GRUB menu leaves a read-only, half-broken system #26

Problem

Where

Fix

Acceptance