research

I'll fan out parallel research agents on this. Quick read of the problem first: the symptom you describe — `/` on one physical drive, `/boot` on a separate physical drive, and a boot-time race where `/boot` fails to mount — is a classic **systemd device-readiness / mount-ordering race**, mostly independent of whether the box runs Xen. I'll have the agents cover the systemd/fstab angle, the bootloader/initramfs angle, the hardware-enumeration angle, and the Xen-specific angle, then synthesize.

Let me launch them concurrently.

• Agent Research solutions to a Linux boot-time race condition where a separately-mounted /boot partition (on a SECOND physical … 166.6s 18.1 KB

• ToolSearch select:WebSearch,WebFetch 5ms 100 B

I'll research this thoroughly using web search to confirm current best practices, then synthesize a technical answer.

• Agent Research the bootloader/GRUB/initramfs angle of a Linux boot problem where /boot lives on a SEPARATE PHYSICAL DRIVE from… 158.0s 14.7 KB

I'll run several web searches in parallel to confirm current best practices from authoritative sources.

• WebSearch systemd fstab nofail noauto x-systemd.device-timeout x-systemd.automount mount options 8.2s 2.7 KB

• WebSearch systemd.mount man page local-fs.target local-fs-pre.target x-systemd.requires before after 5.1s 2.5 KB

• WebSearch Arch wiki fstab usage UUID nofail x-systemd.automount boot partition 7.2s 2.7 KB

I'll research this Linux boot problem. Let me start by loading the web search tools and running parallel searches on the key angles.

• ToolSearch select:WebSearch,WebFetch 3ms 100 B

• WebSearch GRUB /boot separate drive boot order race condition slow drive spin up firmware timeout 10.6s 2.4 KB

• WebSearch rootdelay rootflags kernel cmdline initramfs does not help /boot mount race fstab 9.9s 2.3 KB

Let me run parallel web searches across the four key angles.

• Agent Research the hardware / device-enumeration timing angle of a Linux boot race condition: a server has root (/) on one phy… 173.1s 14.2 KB

Let me fetch the authoritative man page and Arch wiki for precise syntax, plus search for diagnosis specifics.

• WebSearch systemctl status boot.mount journalctl -b emergency mode fstab failed mount diagnose systemd-fsck 12.0s 2.6 KB

• WebFetch https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html 3.3s 215 B

I'll research this Linux boot race condition thoroughly. This is a focused technical research task that benefits from multiple web searches to confirm current facts. Let me use the deep-research harness.

• ToolSearch select:WebSearch,WebFetch 3ms 100 B

• WebSearch does /boot need to be mounted at runtime systemd noauto automount kernel update grub 9.7s 2.5 KB

• Agent The user wrote "xenn" — likely the Xen hypervisor. Research whether running Xen changes anything about a boot problem wh… 103.4s 10.9 KB

• WebSearch EFI System Partition ESP must be on primary boot drive UEFI which disk /boot location best practice 10.4s 2.3 KB

• WebSearch systemd fstab mount /boot race condition x-systemd.device-timeout nofail noauto separate disk 9.1s 2.3 KB

I'll research this. Since "xenn" is ambiguous, let me run several searches in parallel to cover the interpretations and the technical specifics.

• ToolSearch select:WebSearch,WebFetch 4ms 100 B

Let me run parallel web searches across the key topic areas.

• WebSearch systemd x-systemd.device-timeout udev .device unit by-uuid mount timeout slow drive 9.6s 2.1 KB

• WebSearch rootdelay scsi_mod.scan=sync usb-storage.delay_use kernel boot wait for slow drive 8.0s 2.5 KB

• WebSearch /dev/sda /dev/sdb device naming not stable race condition mount by UUID instead of sdX 6.3s 2.1 KB

• WebSearch HDD staggered spin-up PUIS SSU BIOS spin-up delay SATA link training time 12.8s 2.2 KB

• WebSearch Xen dom0 systemd boot /boot separate disk mount race condition fstab nofail 7.1s 2.8 KB

• WebSearch Xen hypervisor GRUB multiboot2 boot xen.gz dom0 kernel initramfs /boot 8.5s 2.5 KB

• WebSearch XCP-ng XenServer boot partition layout /boot conventions 12.1s 2.7 KB

Let me fetch the Debian manpage (which mirrors freedesktop) and the Arch fstab wiki for precise syntax details.

• WebFetch https://manpages.debian.org/trixie/systemd/systemd.mount.5.en.html 6.7s 1.7 KB

• WebFetch https://wiki.archlinux.org/title/Fstab 2.6s 664 B

Let me dig deeper into a few authoritative sources and search for the specific diagnostic angle.

• WebSearch BIOS POST delay HDD spin up secondary drive not detected boot device missing intermittent 10.4s 3.1 KB

• WebSearch separate /boot partition still needed modern best practice reasons against unnecessary 9.8s 2.2 KB

• WebFetch https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html 4.7s 215 B

Good foundation. Let me dig deeper into the systemd/udev mechanics and the diagnostics angle, plus SMART slow-spin detection.

Let me fetch the authoritative "How Xen Boots" doc and search for the remaining items (pvgrub/pygrub, Xenial EOL).

• WebSearch systemd dev-disk-by-uuid.device unit how created udev SYSTEMD_READY device timeout 90 seconds default 14.2s 5.3 KB

I have strong authoritative material from the freedesktop/Debian systemd.mount man page. Let me get a couple more specifics: the device-name ordering non-determinism and a RHEL/Debian confirmation on nofail behavior, plus the fstab man page for option semantics.

• WebSearch /dev/sda /dev/sdb non-deterministic ordering kernel async probe reason persistent block device names by-uuid 10.4s 3.1 KB

• WebSearch dmesg ata link up SATA link timestamps slow drive detection journalctl boot device timeout diagnose 11.0s 2.7 KB

• WebSearch RHEL fstab nofail mount entry not critical boot continue red hat documentation x-systemd.device-timeout 10.0s 2.3 KB

• WebSearch pygrub pvgrub domU bootloader vs dom0 grub Xen wiki difference 11.8s 2.4 KB

• WebSearch SMART spin up time attribute 3 failing drive slow to spin up diagnose smartctl 10.4s 2.5 KB

• WebFetch https://xenbits.xen.org/docs/unstable/hypervisor-guide/x86/how-xen-boots.html 7.4s 1.2 KB

• WebSearch Ubuntu 16.04 Xenial Xerus end of life EOL date ESM 10.1s 3.1 KB

• WebSearch separate /boot partition different drive fails to mount intermittently boot race nofail x-systemd.device-timeout 10.5s 2.6 KB

I have enough to confirm the architecture. Let me verify two precise points: that GRUB reads /boot at firmware stage (no OS), and the systemd udev-based mount-on-device-appearance behavior.

• WebFetch https://wiki.archlinux.org/title/Fstab 6.5s 777 B

• WebFetch https://www.baeldung.com/linux/boot-partition-necessary 5.0s 215 B

I have sufficient confirmation from the search results across all four angles. The man pages and ArchWiki summaries already gave me the precise behavior. Let me do one final targeted search to nail the "how to tell which failure mode" diagnostic and the rootdelay-vs-/boot distinction.

I have strong coverage. Let me fetch two authoritative primary sources to nail down the precise mechanics: the systemd.device man page and the kernel parameter docs, plus confirm libata/HBA scan behavior.

• WebSearch GRUB loads kernel initramfs from /boot before kernel runs rootdelay only affects root not /boot 9.8s 2.9 KB

• WebSearch emergency mode dependency failed for /boot mount cannot find boot device journalctl diagnose 11.4s 2.7 KB

• WebFetch https://www.freedesktop.org/software/systemd/man/latest/systemd.device.html 2.8s 215 B

• WebSearch libata scsi_mod.scan async vs sync HBA RAID controller drives behind expander slow to appear initramfs 9.8s 2.8 KB

I have enough authoritative coverage to synthesize. Let me grab one more confirmation on the kernel-parameters doc for rootdelay and the man page via an alternate mirror.

• WebFetch https://man7.org/linux/man-pages/man5/systemd.device.5.html 13.9s 1.5 KB

I have enough to write the note. Here's the research.

---

# Xen and the "/boot on a separate drive fails to mount at boot" race

## First: "xenn" is ambiguous

"xenn" isn't a real product name. Ranked by likelihood for a boot-problem context:

1. **Xen** (the bare-metal hypervisor) — most likely, and the focus below.
2. **XCP-ng / XenServer / Citrix Hypervisor** (the turnkey Xen appliance distros) — possible if they run a packaged virtualization host rather than rolling their own. Different boot conventions; covered briefly.
3. **Xenial** (Ubuntu 16.04 "Xenial Xerus") — possible if "xenn" is a typo'd distro name. Note it is now fully end-of-life.

The key conclusion up front: **for a `/boot`-fails-to-mount-at-boot race, plain Xen changes essentially nothing about the fix.** It's the same generic Linux/systemd problem with the same generic fix.

---

## 1. How Xen boots (why `/boot` must be readable by the bootloader at the very start)

Xen is itself a kernel that GRUB loads via the **Multiboot** protocol. Xen supports Multiboot 1 and 2 on BIOS, and Multiboot 2 with EFI extensions (or native EFI64) on UEFI. ([How Xen Boots — xenbits](https://xenbits.xen.org/docs/unstable/hypervisor-guide/x86/how-xen-boots.html))

A typical GRUB entry looks like:

```
multiboot2 /boot/xen.gz dom0_mem=2544M ... console=vga
module2   /boot/vmlinuz-...-xen root=... ro ...
module2   /boot/initrd-...-xen.img
```

So GRUB reads **three files out of `/boot`** at the very start of boot:
- `multiboot2` → the Xen hypervisor (`xen.gz`, or `xen.efi` on UEFI),
- `module2` → the **dom0 Linux kernel** (`vmlinuz`),
- `module2` → the **dom0 initramfs**.

(`xen.gz` is a stripped ELF32 image loaded at 2MB and gzip-compressed; `xen.efi` is the PE32+ equivalent. ([xenbits](https://xenbits.xen.org/docs/unstable/hypervisor-guide/x86/how-xen-boots.html)))

**Why this matters for your problem:** GRUB reads these files using its *own* filesystem drivers, before any OS or initramfs exists. GRUB locates `/boot` by partition (`set root=...`/`search --fs-uuid`), not via Linux's `/etc/fstab`. This stage is **identical to bare-metal Linux** — bare metal GRUB also reads `vmlinuz` + `initrd` from `/boot` the same way; Xen just adds `xen.gz` as the first file. As long as GRUB can find the partition that holds `/boot`, the multiboot stage succeeds whether or not Xen is involved.

**Crucial point:** the failure you're describing — `/boot` failing to *mount* due to an fstab/systemd race — happens **much later**, after dom0's kernel and systemd are already running. It is not the GRUB/multiboot stage. GRUB doesn't "mount" anything; it just reads files. So the race is not a Xen-boot-sequence problem at all.

---

## 2. Does dom0 have `/boot` mounting quirks? (No — it's normal systemd)

Once Xen hands control to the dom0 kernel and that kernel pivots from initramfs to the real root, **dom0 is an ordinary Linux system running ordinary systemd.** It processes `/etc/fstab`, generates `*.mount` units (e.g. `boot.mount`), and brings up `local-fs.target` exactly as bare metal does.

Therefore the **`/boot` mount race in dom0 is the same race as on bare metal.** The classic symptom is well documented on systemd systems with a separate `/boot`: systemd's generator-created mount unit and an automount/duplicate path can collide, and `boot.mount` fails with "already mounted" or "/boot is busy," or the device simply isn't ready when `local-fs` runs. ([ArchWiki: fstab](https://wiki.archlinux.org/title/Fstab); [Arch forum: separate /boot causing boot failure](https://bbs.archlinux.org/viewtopic.php?id=147117))

The Xen-specific dom0 fstab gotchas that *do* exist are unrelated to your issue — they involve `xenfs`/`/proc/xen` mounts failing, not block-device `/boot` mounts. ([Arch forum: Xen modules fail to load at boot](https://bbs.archlinux.org/viewtopic.php?id=235539))

Note also: this failure mode is **dom0-only**. domU (guest) kernels are loaded by **pygrub** (a Python script in dom0 that reads the guest's filesystem) or **PV-GRUB / pvgrub** (a GRUB compiled to run paravirtualized inside the domU). Those concern *guest* `/boot` and have nothing to do with the host's separate-`/boot`-drive mount race. ([Xen wiki: PvGrub](https://wiki.xenproject.org/wiki/PvGrub); [Debian wiki: PyGrub](https://wiki.debian.org/PyGrub))

---

## 3. Common Xen + GRUB `/boot` failures, and placement guidance

The Xen-flavored boot failures that show up in the wild are about the **multiboot/GRUB stage**, not the systemd mount race:
- GRUB not built with multiboot2 support / wrong entry type so `xen.gz` won't load. ([help-grub thread on multiboot2 in GRUB2 with Xen](https://lists.gnu.org/archive/html/help-grub/2017-05/msg00023.html))
- BIOS-vs-UEFI mismatch (xen.gz vs xen.efi path differences).
- `grub-mkconfig` not picking up the Xen entry on Debian/Ubuntu (the `20_linux_xen` script).

There is no Xen Project guidance saying "`/boot` must be on the primary disk" beyond the universal bootloader requirement: **the firmware/GRUB must be able to reach the partition holding `xen.gz` + kernel + initramfs at power-on.** That's the same constraint as any GRUB system. If `/boot` is on a second physical disk, the disk simply needs to be in the firmware boot order and reachable by GRUB — again, identical to bare metal.

---

## 4. If they meant XCP-ng / XenServer, or Xenial

**XCP-ng / XenServer / Citrix Hypervisor:** These are appliance distros that **own the whole disk and manage their own partition layout** — you don't hand-roll fstab. XCP-ng 8.3 mandates GPT, won't use the old 4 GB layout, and places GRUB at `efi/EFI/xenserver/grub.cfg` (UEFI) or `grub/grub.cfg` (BIOS), with `xen.gz`, `vmlinuz`, `install.img` in the boot area. ([XCP-ng install docs](https://docs.xcp-ng.org/installation/install-xcp-ng/); [XenServer partition layout](https://docs.xenserver.com/en-us/citrix-hypervisor/install/partition-layout.html)) On these, a user-introduced "separate `/boot` drive that races at mount" is **not a normal scenario** — you generally don't edit their fstab. If they're hitting boot trouble on XCP-ng, it's far more likely BIOS boot order / UEFI-vs-BIOS mode mismatch (XCP-ng explicitly warns never to switch firmware mode after install) than an fstab race. ([XenServer 8 advanced install](https://docs.xenserver.com/en-us/xenserver/8/install/advanced-install.html))

**Ubuntu 16.04 "Xenial":** if "xenn" is really "xenial," note it hit standard EOL on **2021-04-30** and its Ubuntu Pro/ESM coverage ended in **April 2026** — so it is now fully end-of-life as of today (2026-06-17). ([Canonical: 16.04 transitions to ESM](https://canonical.com/blog/ubuntu-16-04-lts-transitions-to-extended-security-maintenance-esm); [Canonical: end of ESM options](https://ubuntu.com/blog/ubuntu-16-04-lts-has-reached-the-end-of-standard-expanded-security-maintenance-with-ubuntu-pro-here-are-your-options)) On Xenial the `/boot` race is just the generic systemd fstab issue.

---

## 5. Do the standard fixes apply under Xen? (Yes — unchanged)

Every standard fix applies to Xen dom0 exactly as to bare metal, because dom0 is a normal systemd Linux once it's up:

- **Mount by UUID** (not `/dev/sdb1`) — immune to disk enumeration order changing between boots, which is itself a common cause of the "race." ([ArchWiki: fstab](https://wiki.archlinux.org/title/Fstab))
- **`nofail`** — boot continues if `/boot` isn't ready. (Caveat: it *silently* ignores the failure, so pair it with the timeout.) ([ArchWiki: fstab](https://wiki.archlinux.org/title/Fstab))
- **`x-systemd.device-timeout=...`** — bound the wait so a slow/absent second disk doesn't hang boot for the default 90 s. ([ArchWiki: fstab](https://wiki.archlinux.org/title/Fstab))
- **Firmware/GRUB boot order** — ensure the disk with the ESP / GRUB / `/boot` is first in boot order and reachable by GRUB. This is the one that actually touches the Xen multiboot stage (GRUB must read `xen.gz`), and it's still the generic fix.
- Optionally drop a duplicate automount or add `x-systemd.requires`/ordering if you're hitting the "already mounted / busy" collision.

None of these need a Xen variant. There is no `xen.gz`-equivalent option, no dom0 fstab flag, nothing hypervisor-specific to add.

---

## Bottom line

- **"xenn" almost certainly means Xen** (or possibly XCP-ng/XenServer; or a typo for Xenial).
- **Xen does not change your `/boot` mount race.** The race occurs in dom0's normal systemd phase, which is identical to bare metal. GRUB's multiboot2 load of `xen.gz` + kernel + initramfs happens earlier and isn't a "mount" at all.
- **Xen adds exactly one wrinkle**, and only to the *GRUB* stage: GRUB must be multiboot2-capable and must be able to read `xen.gz` from the `/boot` partition. If `/boot` is on a second disk, that disk must be in firmware boot order — the same requirement bare-metal GRUB has for `vmlinuz`.
- **All the standard fixes** (UUID mounts, `nofail`, `x-systemd.device-timeout`, fixing BIOS/UEFI boot order, keeping ESP/`/boot` reachable) **apply unchanged under Xen.** There is no Xen-specific fix to add.
- If it's **XCP-ng/XenServer**, the appliance manages its own partitions/fstab, so a hand-rolled separate-`/boot` race shouldn't arise; suspect firmware-mode/boot-order instead.
- If it's **Xenial 16.04**, same generic fix, but flag that it is now fully EOL.

### Sources
- [How Xen Boots — Xen Project hypervisor guide](https://xenbits.xen.org/docs/unstable/hypervisor-guide/x86/how-xen-boots.html)
- [GRUB2 + multiboot2 with Xen 4.9+ (help-grub list)](https://lists.gnu.org/archive/html/help-grub/2017-05/msg00023.html)
- [ArchWiki: fstab (nofail, x-systemd.device-timeout, UUID)](https://wiki.archlinux.org/title/Fstab)
- [Arch forum: systemd + separate /boot causing boot failure](https://bbs.archlinux.org/viewtopic.php?id=147117)
- [Arch forum: Xen modules fail to load at boot (dom0 fstab quirks)](https://bbs.archlinux.org/viewtopic.php?id=235539)
- [Xen wiki: PvGrub (domU bootloader)](https://wiki.xenproject.org/wiki/PvGrub)
- [Debian wiki: PyGrub (domU bootloader)](https://wiki.debian.org/PyGrub)
- [XCP-ng install docs](https://docs.xcp-ng.org/installation/install-xcp-ng/)
- [XenServer partition layout](https://docs.xenserver.com/en-us/citrix-hypervisor/install/partition-layout.html)
- [XenServer 8 advanced/other install scenarios](https://docs.xenserver.com/en-us/xenserver/8/install/advanced-install.html)
- [Canonical: Ubuntu 16.04 transitions to ESM](https://canonical.com/blog/ubuntu-16-04-lts-transitions-to-extended-security-maintenance-esm)
- [Canonical: Ubuntu 16.04 end of standard ESM — options](https://ubuntu.com/blog/ubuntu-16-04-lts-has-reached-the-end-of-standard-expanded-security-maintenance-with-ubuntu-pro-here-are-your-options)

I now have comprehensive, authoritative confirmation from all the requested sources. Let me compile the final technical answer.

---

# Boot-time race: separately-mounted /boot on a second physical drive fails to mount

This is the classic symptom of (a) using `/dev/sdX` names that swap between two drives, and/or (b) treating `/boot` as a *required* mount so a slow/missing device drops you into emergency mode. Below is the systemd + fstab layer explained, then a prioritized set of concrete fixes.

---

## 0. Root-cause summary (read first)

Two independent failure modes are usually conflated here:

1. **Device-name race.** Since Linux 5.3, SCSI/SATA probing is *asynchronous* (`scsi_mod.scan=async`), so the order in which two physical drives are enumerated is non-deterministic. `/dev/sda` and `/dev/sdb` can swap between boots, or when hardware is added/removed. An fstab entry like `/dev/sdb1 /boot ...` then points at the wrong disk or a disk that isn't ready yet. **Fix: never use `/dev/sdX`; use `UUID=`/`PARTUUID=`/`LABEL=`.**

2. **Criticality race.** By default a local fstab entry is *required* by `local-fs.target` and is *ordered before* it. If the second drive shows up late (or never), the mount job fails, `local-fs.target` fails, and systemd drops to emergency mode. **Fix: make `/boot` non-critical with `nofail` (+ a short device timeout), or don't mount it at boot at all (`noauto` ± automount).**

The recommended combination is at the end of §1.

---

## 1. fstab mount options (systemd semantics)

When systemd boots, `systemd-fstab-generator` reads `/etc/fstab` and synthesizes a `.mount` unit per entry at boot. The 4th field (options) is where you control behavior. Exact semantics from the `systemd.mount(5)` man page:

### `auto` / `noauto`
- `auto` (default): the mount unit is pulled in as a dependency of `local-fs.target`, so it mounts during boot.
- `noauto`: *"the mount unit will not be added as a dependency for local-fs.target."* It is not mounted at boot; you mount it on demand (`mount /boot`, `systemctl start boot.mount`) or via an automount trigger. It can still be a dependency of other units.

### `nofail`
From the man page: *"With nofail, this mount will be only **wanted, not required**, by local-fs.target or remote-fs.target. Moreover, the **mount unit is not ordered before** these target units."*

Two distinct effects, both matter:
- **Wanted not required** → if the mount fails, `local-fs.target` still succeeds → boot continues, no emergency mode.
- **Not ordered before local-fs.target** → boot does not *wait* for this mount; it proceeds in parallel and the mount completes whenever the device appears.

This is exactly what you want for a non-critical `/boot`.

### `x-systemd.device-timeout=`
*"Configure how long systemd should wait for a device to show up before giving up on an entry from /etc/fstab."* Accepts `s`, `min`, `h`, `ms`. **Default is 90 s.** Critical caveat (Arch wiki / RHEL): `nofail` alone still makes systemd wait the full 90 s for a missing/slow device before giving up — adding `nofail` without a timeout can make boot 90 s *longer*. Always pair them: `nofail,x-systemd.device-timeout=10`.

### `x-systemd.automount`
*"An automount unit will be created for the file system."* systemd creates a `boot.automount` unit that watches the mount point; the real `boot.mount` fires only on first access. Notes:
- *"If x-systemd.automount is used, neither auto nor noauto have any effect"* — the automount unit (not the mount unit) becomes the boot dependency, and the automount point itself comes up instantly (it's just a kernel autofs watch), so a missing second drive **cannot** block or fail boot.
- Pair with `x-systemd.idle-timeout=` to auto-unmount after a period of inactivity.

### `x-systemd.requires=` / `x-systemd.before=` / `x-systemd.after=`
- `x-systemd.requires=UNIT` → adds a `Requires=` **and** `After=` on `UNIT`. Use to express "mount only after this other thing exists" (e.g. an mdraid/crypt/overlay dependency).
- `x-systemd.before=UNIT` / `x-systemd.after=UNIT` → pure ordering (`Before=`/`After=`), no requirement. Use to slot the mount into the right place in the dependency graph without forcing it to pull the unit in.

These are rarely needed for a plain second-disk `/boot`, but are the right tool if `/boot` lives on top of RAID/LVM/LUKS and must wait for that layer to assemble.

### Recommended combination for a non-critical `/boot` that must still mount eventually

Two valid strategies — pick by §6:

**Strategy A — mount at boot but never block/fail boot (simplest, keeps /boot mounted):**
```
UUID=XXXX-XXXX  /boot  vfat  defaults,nofail,x-systemd.device-timeout=10  0  2
```

**Strategy B — on-demand automount (recommended; /boot literally cannot break boot):**
```
UUID=XXXX-XXXX  /boot  vfat  defaults,noauto,x-systemd.automount,x-systemd.idle-timeout=120  0  0
```
(`fat`/`vfat` ESP-style `/boot` → use `0 2` or `0 0` for fsck pass; `ext4` `/boot` → `0 2`.)

---

## 2. Persistent identifiers instead of `/dev/sdX`

**Why `/dev/sdX` is unreliable:** kernel block-device names are assigned at probe time. Asynchronous SCSI/SATA scanning (default since kernel 5.3) means probe order between two drives is **non-deterministic across boots**; it also shifts when you add/remove disks or change SATA ports. An fstab line keyed on `/dev/sdb1` can resolve to the wrong disk or to a device that isn't enumerated yet → the `/boot` race. (Red Hat RHEL 9 "Overview of persistent naming attributes" documents this; the udev `/dev/disk/by-*/` symlinks exist precisely to give stable references.)

**Stable alternatives** (find them with `lsblk -f`, `blkid`, or `ls -l /dev/disk/by-*`):

| Identifier | What it ties to | fstab syntax | Notes |
|---|---|---|---|
| `UUID=` | the **filesystem** UUID | `UUID=1234-ABCD /boot ...` | Most common. Changes if you reformat. For FAT ESPs it's the short `XXXX-XXXX` form. |
| `PARTUUID=` | the **GPT partition** UUID (or MBR disk-id+offset) | `PARTUUID=abcd-... /boot ...` | Survives reformatting the filesystem; tied to the partition. Good for ESP/`/boot`. |
| `LABEL=` | filesystem label | `LABEL=BOOT /boot ...` | Human-readable but collision-prone if two filesystems share a label — avoid with two drives unless labels are unique. |

Recommendation for a two-drive setup: **`UUID=` (or `PARTUUID=` for the ESP)**. Never `LABEL=` if you can't guarantee uniqueness across both disks.

---

## 3. Writing `.mount` / `.automount` units manually

You normally let `systemd-fstab-generator` create these from fstab, but you can write them directly in `/etc/systemd/system/`. **The unit filename must equal the escaped mount path** (`systemd-escape -p --suffix=mount /boot` → `boot.mount`). A unit named `boot.mount` *must* have `Where=/boot` or systemd rejects it.

**`/etc/systemd/system/boot.mount`:**
```ini
[Unit]
Description=Boot partition on second drive
# Don't block/fail boot; these mirror what nofail does:
DefaultDependencies=no
Conflicts=umount.target
Before=umount.target

[Mount]
What=/dev/disk/by-uuid/1234-ABCD
Where=/boot
Type=vfat
Options=defaults
TimeoutSec=10

[Install]
WantedBy=local-fs.target
```

**`/etc/systemd/system/boot.automount`** (preferred — on-demand, can't break boot):
```ini
[Unit]
Description=Automount /boot on access

[Automount]
Where=/boot
TimeoutIdleSec=120

[Install]
WantedBy=local-fs.target
```
Then `systemctl daemon-reload && systemctl enable --now boot.automount`. (If using the automount unit, you do **not** also enable `boot.mount`; the automount triggers it.)

**Ordering targets:**
- `local-fs-pre.target` — runs *before* any local filesystem is mounted (a hook point for things that must exist before mounts, e.g. device setup). Local mounts auto-gain `After=local-fs-pre.target`.
- `local-fs.target` — reached once all *required* local mounts are up; the rest of userspace (and `sysinit.target` → `basic.target` → services) orders after it.

Per the man page: *"Mount units referring to local file systems automatically gain an `After=` dependency on `local-fs-pre.target`, and a `Before=` dependency on `local-fs.target` **unless one or more mount options among `nofail`, `x-systemd.wanted-by=`, and `x-systemd.required-by=` is set**."* That single sentence is the whole mechanism: `nofail` is what removes the `Before=local-fs.target` ordering edge, so boot stops waiting on `/boot`.

---

## 4. `systemd-fsck@`, `local-fs.target`, and fail vs `nofail`

- **`systemd-fsck@.service`** is a templated unit instantiated per device that has a non-zero fsck pass number (6th fstab field). For `UUID=…/boot…0 2`, systemd runs `systemd-fsck@dev-disk-by\x2duuid-….service` *before* mounting. If the device is missing/late, the fsck instance is part of what the mount job waits on. A `0` in the 6th field disables fsck for that entry (common for FAT ESPs).
- **`local-fs.target`** is the synchronization point for "all required local filesystems mounted." It is a *passive* target pulled together by the mount units.

**What happens on failure:**

| Entry has… | If the device is missing/slow | Net effect |
|---|---|---|
| (default, no `nofail`) | mount job is **required by** and **ordered before** `local-fs.target`; the job blocks until `x-systemd.device-timeout` (90 s default), then **fails** | `local-fs.target` fails → `sysinit`/boot can't proceed → **`emergency.target` (maintenance shell, asks for root password)** |
| `nofail` | mount is only **wanted**, **not ordered before** `local-fs.target`; failure is ignored | boot continues; `/boot` mounts later if/when the device appears (within the timeout window udev sees it) |
| `noauto` | mount not pulled into boot at all | boot continues; `/boot` never auto-mounts |
| `noauto,x-systemd.automount` | automount watch installed instantly | boot continues; `/boot` mounts on first access |

So the difference between a normal entry and `nofail` is *precisely* the difference between dropping to emergency mode and a clean boot.

---

## 5. Diagnosis

Run these (in emergency mode, the root prompt or `journalctl -xb` is available):

- **`systemctl status boot.mount`** — state of the mount unit; shows the exact `mount(8)` error, the `What=`/`Where=`, and which job failed. Use the escaped name if needed: `systemctl status 'boot.mount'`.
- **`systemctl --failed`** — lists failed units; `boot.mount` and/or `systemd-fsck@…` will appear here.
- **`journalctl -b`** (or `-xb` for explanatory hints) — full current-boot log; grep for the mount point / device. Look for `Timed out waiting for device`, `Dependency failed for /boot`, or `Failed to mount`.
- **`journalctl -b -u boot.mount -u 'systemd-fsck@*'`** — narrow to just the mount/fsck units.
- **`systemd-analyze blame`** — per-unit startup time; a `/boot` entry that's eating ~90 s confirms a device-timeout stall (the classic "`nofail` but no `x-systemd.device-timeout`" symptom).
- **`systemd-analyze critical-chain local-fs.target`** — shows what `local-fs.target` is waiting on and the ordering chain.
- **`lsblk -f`** and **`blkid`** — confirm the second drive is actually present and what its real UUID/PARTUUID/LABEL is (compare against fstab — a stale UUID after reformat/disk-swap is the #1 emergency-mode cause).
- **`dmesg | grep -iE 'sd[a-z]|ata|nvme|I/O error'`** — kernel view of the drive: late enumeration, link resets, SMART/I-O errors, or the drive not appearing at all (hardware/cabling vs config problem).
- **`findmnt /boot`** / **`mount -a`** (after fixing fstab) — verify the entry now resolves and mounts cleanly before rebooting.

Workflow: `lsblk -f`/`blkid` to confirm the device + true UUID → compare to `/etc/fstab` → `systemctl status boot.mount` + `journalctl -xb` for the precise failure → `systemd-analyze blame` to catch timeout stalls.

---

## 6. Does `/boot` even need to be mounted at runtime?

**No, not for normal operation.** By the time userspace runs, the bootloader has already read `/boot` (kernel + initramfs), the kernel is in RAM, and `/boot` serves no runtime purpose. It is only needed:
- by the **bootloader at boot** (which reads the raw partition itself, not the OS mount), and
- when you **update the kernel/initramfs or regenerate bootloader config** (`mkinitcpio`/`dracut`/`update-grub`/`grub-mkconfig`, kernel package upgrades write new files into `/boot`).

**Tradeoff:**

| Approach | Pros | Cons |
|---|---|---|
| **`nofail` (mounted normally, Strategy A)** | `/boot` is there when package manager runs a kernel update — no surprises; simplest mental model | still attempts at boot; a fully dead second drive logs a failed mount (harmless with `nofail`) |
| **`noauto` (never auto-mounted)** | `/boot` literally can't affect boot | **danger:** a kernel update with `/boot` unmounted writes the new kernel to the *empty mountpoint on the root fs*, not the real `/boot` partition → next boot uses stale/mismatched kernel. You must remember to `mount /boot` before every kernel update. |
| **`noauto,x-systemd.automount` (Strategy B, recommended)** | best of both: zero boot impact, **and** the package manager touching `/boot` during a kernel update transparently triggers the mount | minor first-access latency; needs the option spelled correctly |

**Recommendation:** Use **Strategy B (`noauto,x-systemd.automount`)** unless you have a specific reason to keep `/boot` always mounted. It removes `/boot` from the boot dependency graph entirely (so a second-drive race can never drop you to emergency mode) while still auto-mounting on first access, which is exactly when `apt`/`pacman`/`dnf` write a new kernel. If you prefer `/boot` always present and visible, use **Strategy A (`nofail,x-systemd.device-timeout=10`)**. Avoid bare `noauto` (kernel-update foot-gun).

---

## Prioritized concrete fixes

1. **Switch the identifier off `/dev/sdX`.** Get the real value (`lsblk -f` / `blkid`) and write `UUID=…` (or `PARTUUID=…` for an ESP). This alone fixes the device-enumeration race. *(Highest priority — it's the actual root cause of two-drive ordering.)*

2. **Make `/boot` non-critical.** Add `nofail` **and** `x-systemd.device-timeout=10` (the timeout is mandatory — `nofail` alone keeps the 90 s stall). Or go to automount per #3.

3. **Prefer on-demand automount** for a true non-critical `/boot`:
   ```
   UUID=XXXX-XXXX  /boot  vfat  defaults,noauto,x-systemd.automount,x-systemd.idle-timeout=120  0  0
   ```
   This takes `/boot` out of the boot graph completely yet auto-mounts on kernel updates.

4. **Validate before rebooting:** `systemctl daemon-reload` (or `systemctl daemon-reexec`) to regenerate units from the new fstab, then `mount -a` and `findmnt /boot` — never reboot on an untested fstab.

5. **If still failing, diagnose with** `systemctl status boot.mount`, `journalctl -xb`, `systemd-analyze blame`, `dmesg | grep -iE 'sd|ata|nvme'` to distinguish a config problem (stale UUID) from a hardware problem (drive enumerating late / I-O errors).

Recommended final fstab line (pick one):
```
# Strategy A — always mounted, never blocks/fails boot:
UUID=XXXX-XXXX  /boot  vfat  defaults,nofail,x-systemd.device-timeout=10  0  2
# Strategy B — recommended: on-demand, out of the boot graph entirely:
UUID=XXXX-XXXX  /boot  vfat  defaults,noauto,x-systemd.automount,x-systemd.idle-timeout=120  0  0
```

---

## Sources

- [systemd.mount(5) — freedesktop.org (latest)](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html) — authoritative option reference (`nofail`, `noauto`, `x-systemd.*`, automatic local-fs dependencies)
- [systemd.mount(5) — Debian trixie manpage](https://manpages.debian.org/trixie/systemd/systemd.mount.5.en.html) — same man page, fetched verbatim for the dependency/option quotes
- [fstab — ArchWiki](https://wiki.archlinux.org/title/Fstab) — UUID/LABEL/PARTUUID recommendations, `nofail` + `x-systemd.device-timeout` pairing, automount guidance
- [systemd.mount(5) — Arch manual pages](https://man.archlinux.org/man/systemd.mount.5.en)
- [Why not mount /boot? — Arch Linux Forums](https://bbs.archlinux.org/viewtopic.php?id=253376) — `/boot` not needed at runtime; only for updates
- [Overview of persistent naming attributes — RHEL 9 docs](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/9/html/managing_file_systems/assembly_overview-of-persistent-naming-attributes_managing-file-systems) — why `/dev/sdX` is non-persistent; udev `by-*` symlinks
- [release-notes: non-deterministic SCSI probing — Debian bug #987906](https://www.mail-archive.com/debian-doc@lists.debian.org/msg19122.html) — async probe since kernel 5.3 → non-deterministic `sda`/`sdb`
- [Use of x-systemd.automount in /etc/fstab — SUSE KB 000020402](https://www.suse.com/support/kb/doc/?id=000020402)
- [Troubleshooting Failed Mounts in Emergency Mode with systemd](https://linuxeries.org/post/2026-05-31-troubleshooting-failed-mounts-in-emergency-mo/) — `systemctl --failed`, `journalctl -xb`, `systemctl status` diagnosis flow
- [Welcome to emergency mode — fsck/fstab boot errors](https://linuxconfig.org/welcome-to-emergency-mode-boot-error-understanding-and-resolving-fsck-issues) — fstab/fsck → emergency mode mechanics
- [systemd, fstab, and nofail — Nelson's log](https://nelsonslog.wordpress.com/2022/08/16/systemd-fstab-and-nofail/) — practical `nofail`/`device-timeout` notes
- [fstab-generator #795 — systemd GitHub](https://github.com/systemd/systemd/issues/795) — how `auto,nofail` interacts with the initial transaction and udev pull-in

A note on the man page: `freedesktop.org` returned HTTP 403 to my fetcher, so I cross-verified every option quote against the identical Debian-hosted `systemd.mount(5)` man page (same upstream source). The Arch `Fstab` wiki page is currently behind an Anubis anti-bot challenge for automated fetches; its specifics here come from the indexed search excerpts plus the corroborating SUSE/RHEL/man-page sources.

I have everything I need across all four angles. Here is the analysis.

---

# /boot on a separate physical drive: the boot-race problem, dissected

## The core architectural fact that resolves the confusion

`/boot` is read at **two completely different times by two completely different actors**, and conflating them is the source of most misdiagnosis:

1. **At power-on, by the firmware + GRUB** — before any kernel, any initramfs, any systemd exists. GRUB reads `/boot` to load `vmlinuz-*` (the kernel) and `initramfs-*` (the initial ramdisk) into memory. This uses GRUB's own built-in filesystem drivers and the firmware's disk access, not Linux.
2. **After the kernel is running, by systemd** — `/boot` becomes an ordinary `fstab` mount, identical to mounting `/home` or a data disk. At this point the kernel and root filesystem are already up; `/boot` is mounted only so the package manager can write new kernels there.

Because of this split, a "drive-readiness race" on the `/boot` drive produces **two distinct, unrelated failure modes** with different symptoms, different diagnostics, and different fixes. You must determine which one you're hitting before changing anything — the wrong fix (e.g., `rootdelay=`) addresses neither.

---

## How to tell which failure mode you have (do this first)

| Observation | Failure mode |
|---|---|
| You **never see the GRUB menu**, OR you see firmware messages like "No boot device", "Operating system not found", or GRUB drops to a `grub>` / `grub rescue>` prompt saying it can't find a file | **(a) Firmware/GRUB stage** — the `/boot` (or ESP) drive wasn't ready when firmware/GRUB looked for it |
| GRUB menu appears and you select an entry, but boot then hangs or drops into **emergency / maintenance mode** with messages like `Dependency failed for /boot` or `Timed out waiting for device /dev/disk/by-uuid/...` | **(b) systemd fstab stage** — kernel booted fine (so GRUB already read `/boot` successfully), but the later re-mount of `/boot` raced the drive |

The decisive tell: **if the kernel boots at all, GRUB already read `/boot` successfully this cycle** — so any remaining `/boot` problem is failure mode (b), a normal fstab/systemd mount race, *not* a GRUB problem. Confirm with `journalctl -xb` (or `journalctl -xb -1` for the previous failed boot) and look for the `boot.mount` unit and `Timed out waiting for device`.

---

## Failure mode (a): firmware/GRUB can't find /boot at power-on

**Mechanism.** When the firmware (UEFI or legacy BIOS) hands control to the boot disk, a secondary spinning HDD may not have completed spin-up and SATA link negotiation yet. Firmware POST is fast; a cold HDD can take several seconds to become ready. If the firmware enumerates drives before the secondary drive answers, that drive is simply *absent* from the boot device list, and the firmware either skips to the next entry or reports no boot device. This is intermittent precisely because it's a timing race — warm reboots (drive already spinning) succeed, cold boots fail. Forum reports confirm a POST/boot delay of a few seconds reliably fixes secondary-drive non-detection, and that it's effectively a firmware-timing bug unless the drive's spin-up is genuinely slow ([Tom's Hardware](https://forums.tomshardware.com/threads/why-is-secondary-hdd-preventing-bios-boot-up.3761428/), [ASRock Forums](https://forum.asrock.com/forum_posts.asp?TID=478&title=ability-to-skip-waiting-for-hard-drives-to-spin-up), [HP Support](https://h30434.www3.hp.com/t5/Desktop-Hardware-and-Upgrade-Questions/Second-hard-drive-not-recognized-until-reboot/td-p/8346127)).

**Where the ESP must live (UEFI specifically).** On UEFI, the firmware doesn't load GRUB from `/boot` — it loads the bootloader from the **EFI System Partition (ESP)**, a FAT32 partition the firmware reads directly. The ESP is conventionally on the primary drive and ideally the first partition, since it's least disturbed by repartitioning ([Arch Wiki — ESP](https://wiki.archlinux.org/title/EFI_system_partition), [Wikipedia — ESP](https://en.wikipedia.org/wiki/EFI_system_partition)). If your ESP (or a BIOS-mode GRUB stage) lives on the slow secondary drive, the firmware races that drive at the earliest possible moment — the worst place to have the dependency.

**Fixes for (a), best first:**
1. **Move the ESP / bootloader to the primary (fast, always-ready) drive.** The cleanest fix. Keep GRUB and the kernel it loads on the same disk so the firmware never depends on the slow drive to *start* booting. With an NVMe/SSD primary this race essentially disappears.
2. **Add a firmware boot/POST delay.** Many BIOS/UEFI setups expose "POST Delay," "Boot Delay," or staggered-spin-up timing options. Setting 3–5 s gives a cold HDD time to become ready before enumeration ([Tom's Hardware](https://forums.tomshardware.com/threads/why-is-secondary-hdd-preventing-bios-boot-up.3761428/), [ASRock](https://forum.asrock.com/forum_posts.asp?TID=478&title=ability-to-skip-waiting-for-hard-drives-to-spin-up)).
3. **Fix the boot order** so the correct drive is the explicit first boot entry (`efibootmgr` on UEFI), not relying on firmware fallback that may skip a not-yet-ready disk ([Arch Wiki — GRUB](https://wiki.archlinux.org/title/GRUB)).
4. **Rule out power/cabling** — an unstable 5 V rail or marginal SATA cable causes a drive to spin but not enumerate, which looks identical to a spin-up race ([Seagate KB](https://www.seagate.com/support/kb/the-bios-does-not-detect-or-recognize-the-ata-sata-hard-drive-168595en/)).

> Note: `GRUB_TIMEOUT` and the kernel command line are **irrelevant** to mode (a) — by definition you haven't reached a working GRUB/kernel yet.

---

## Failure mode (b): /boot mounts at GRUB stage but fails to re-mount under systemd

**Mechanism.** The kernel and initramfs were already loaded by GRUB (so the *boot* succeeded). Now systemd processes `/etc/fstab` and tries to mount `/boot` as a normal filesystem. systemd doesn't poll — it waits for **udev** to announce the device (by UUID/by-id symlink). If the slow drive hasn't appeared within the device timeout (**default 90 s**), the `boot.mount` unit fails; because `/boot` is pulled into `local-fs.target`, the failure cascades to "Dependency failed for Local File Systems" and drops you to emergency mode ([Arch Wiki — Fstab/mount discussion](https://wiki.archlinux.org/title/Fstab), [systemd.mount](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html), [Arch BBS — Failed to mount /boot](https://bbs.archlinux.org/viewtopic.php?id=298472)).

**Fixes for (b):** tune the `/boot` line in `/etc/fstab`:
- **`nofail`** — makes the mount *wanted* but not *required* by `local-fs.target`, and removes the ordering dependency, so a failed/slow `/boot` no longer blocks boot or throws you into emergency mode ([systemd, fstab, and nofail](https://nelsonslog.wordpress.com/2022/08/16/systemd-fstab-and-nofail/), [Arch Wiki — Fstab](https://wiki.archlinux.org/title/Fstab)). This is the single most important option for a non-essential late mount like `/boot`.
- **`x-systemd.device-timeout=`** — pair with `nofail`. Otherwise the default 90 s device wait makes every boot stall on the slow drive. Set it to a value that comfortably covers the drive's worst-case appearance (e.g. `x-systemd.device-timeout=30`) — long enough to mount the slow drive, short enough that a truly dead drive doesn't hang boot for 90 s ([Arch Wiki — Fstab](https://wiki.archlinux.org/title/Fstab), [systemd issue #795](https://github.com/systemd/systemd/issues/795)).
- Optionally **`noauto` + `x-systemd.automount`** — defers mounting `/boot` until something actually accesses it (i.e., a kernel update), sidestepping the boot-time race entirely. `/boot` is unused at runtime, so this is safe.

Resulting fstab line, e.g.:
```
UUID=<boot-uuid>  /boot  ext4  defaults,nofail,x-systemd.device-timeout=30  0 2
```

This is harmless and effectively always correct for a `/boot` on a separate physical drive — `/boot` genuinely isn't needed at runtime, so it should never be allowed to block boot.

---

## On `rootdelay=` and `rootflags=` — they do NOT help a /boot race

This is the most common misapplied fix, so be precise:

- **`rootdelay=`** tells the **initramfs** how long to wait for the **root (`/`)** device to appear before mounting it ([bootparam(7)](https://man7.org/linux/man-pages/man7/bootparam.7.html), [Launchpad Q#125823](https://answers.launchpad.net/ubuntu/+source/grub2/+question/125823)). It runs *after* GRUB has already loaded the kernel+initramfs from `/boot`, and it concerns `/`, not `/boot`. So:
  - It **cannot** help mode (a) — GRUB already finished reading `/boot` before the kernel/initramfs ran; the delay happens too late.
  - It **does not** help mode (b) — that's a systemd `/boot` mount, not the initramfs root mount.
  - It **would** be the right tool only if your *root* (`/`) device were the one racing — a different problem from the one described.
- **`rootflags=`** just sets mount options for `/` — also unrelated to `/boot`.
- **`/boot` is normally not in the initramfs at all.** The initramfs's job is to find and mount `/`; once `/` is up, systemd handles `/boot` from fstab. There's no initramfs-stage `/boot` mount to delay. (The one nuance: a *ro mount is always attempted first* during initramfs root handling regardless of cmdline/fstab options — see [dracut-ng #2403](https://github.com/dracut-ng/dracut/issues/2403) — but that's about root, not `/boot`.)

The kernel-side analogue to a delay that *would* matter for `/boot` doesn't exist, because `/boot` mounting is owned by systemd/udev — hence `x-systemd.device-timeout`, not `rootdelay`, is the correct knob for mode (b).

---

## Is splitting / and /boot across two separate physical drives a sane layout?

Short answer: **it's defensible for specific reasons, but for a single-OS desktop it mostly adds risk — including exactly this race — for little benefit.**

**When a separate `/boot` is justified:**
- Root on **full-disk encryption (LUSK/LVM/RAID)** or an exotic/new filesystem GRUB can't read — `/boot` must be on a plain filesystem GRUB understands ([Arch Wiki — GRUB](https://wiki.archlinux.org/title/GRUB)).
- **Dual/multi-boot**, where a shared `/boot` survives deleting one OS's root ([Baeldung](https://www.baeldung.com/linux/boot-partition-necessary)).
- Root partition **> 2 TB on UEFI**, which can require a separate sub-2 TB `/boot` ([Baeldung](https://www.baeldung.com/linux/boot-partition-necessary)).

**When it's asking for trouble:**
- A separate `/boot` *partition* is one thing; putting `/boot` (or the ESP) on a **separate slow physical drive** is the specific antipattern causing your race. Modern UEFI has no 1024-cylinder limit, and for a single-OS install a dedicated `/boot` "might be overcomplicating system administration" ([Baeldung](https://www.baeldung.com/linux/boot-partition-necessary)). Distro installers (e.g. Red Hat) default to a small `/boot` *partition on the same disk as root*, never a separate spindle ([RHEL partitioning guide](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/installation_guide/s2-diskpartrecommend-x86)).

**Recommended layout if you don't have a hard reason otherwise:** keep `/boot` (and the ESP) **on the same fast primary drive as `/`** — ideally the SSD/NVMe. This removes the slow-drive dependency from *both* the firmware/GRUB stage and the systemd stage simultaneously, and matches every distro default. Reserve the slow secondary drive for data, with its fstab entry marked `nofail`.

---

## Practical decision tree

1. **No GRUB menu / firmware can't find boot device?** → Mode (a). Move ESP+GRUB+kernel to the primary drive (best), or add a 3–5 s firmware POST/boot delay; fix `efibootmgr` order; check power/cabling. `rootdelay=`/`GRUB_TIMEOUT` are useless here.
2. **GRUB works, kernel boots, then "Dependency failed for /boot" / device timeout?** → Mode (b). Add `nofail,x-systemd.device-timeout=30` (or `noauto,x-systemd.automount`) to the `/boot` fstab line. `rootdelay=` is useless here too.
3. **Best long-term fix for either:** stop splitting `/boot` onto a separate slow physical drive — relocate `/boot` and the ESP to the primary drive alongside `/`.

---

### Sources
- [Linux boot process: GRUB → initramfs → systemd](https://www.commandinline.com/linux-boot-process-grub-systemd/) — GRUB loads kernel+initramfs from /boot before kernel runs
- [bootparam(7) man page](https://man7.org/linux/man-pages/man7/bootparam.7.html) — `rootdelay=`, `rootflags=` semantics (root only)
- [Ubuntu Launchpad Q#125823 — rootdelay scope](https://answers.launchpad.net/ubuntu/+source/grub2/+question/125823)
- [dracut-ng issue #2403 — initramfs ro root mount behavior](https://github.com/dracut-ng/dracut/issues/2403)
- [systemd.mount man page](https://www.freedesktop.org/software/systemd/man/latest/systemd.mount.html)
- [systemd issue #795 — auto,nofail and udev-driven mount pull-in](https://github.com/systemd/systemd/issues/795)
- [Arch Wiki — Fstab (nofail, x-systemd.device-timeout, automount)](https://wiki.archlinux.org/title/Fstab)
- [Nelson's log — systemd, fstab, and nofail](https://nelsonslog.wordpress.com/2022/08/16/systemd-fstab-and-nofail/)
- [Arch BBS — Failed to mount /boot, emergency mode](https://bbs.archlinux.org/viewtopic.php?id=298472)
- [Arch Wiki — EFI system partition](https://wiki.archlinux.org/title/EFI_system_partition)
- [Wikipedia — EFI system partition](https://en.wikipedia.org/wiki/EFI_system_partition)
- [Arch Wiki — GRUB](https://wiki.archlinux.org/title/GRUB)
- [Tom's Hardware — secondary HDD preventing BIOS boot](https://forums.tomshardware.com/threads/why-is-secondary-hdd-preventing-bios-boot-up.3761428/)
- [ASRock Forums — skip waiting for drive spin-up](https://forum.asrock.com/forum_posts.asp?TID=478&title=ability-to-skip-waiting-for-hard-drives-to-spin-up)
- [HP Support — second drive not recognized until reboot](https://h30434.www3.hp.com/t5/Desktop-Hardware-and-Upgrade-Questions/Second-hard-drive-not-recognized-until-reboot/td-p/8346127)
- [Seagate KB — BIOS does not detect SATA drive](https://www.seagate.com/support/kb/the-bios-does-not-detect-or-recognize-the-ata-sata-hard-drive-168595en/)
- [Baeldung — Is the /boot partition always necessary?](https://www.baeldung.com/linux/boot-partition-necessary)
- [RHEL recommended partitioning scheme](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/installation_guide/s2-diskpartrecommend-x86)

I have comprehensive, well-sourced material across all five topic areas. Let me compile the final report.

---

# Linux Boot Race: `/boot` on a Second Drive That Isn't Ready in Time

## The mechanism in one paragraph

systemd-udevd discovers block devices asynchronously as the kernel reports them, and for each one that carries the `systemd` udev tag (all block devices by default) it synthesizes a `.device` unit such as `dev-disk-by\x2duuid-<UUID>.device`. A `.mount` unit for `/boot` automatically gains a dependency on the `.device` unit that backs it ("swap and mount units acquire dependencies on the units encapsulating their backing block devices" — systemd.device(5)). systemd waits for that `.device` unit to appear, but only up to a timeout (default 90 s for fstab-referenced devices; overridable per-entry with `x-systemd.device-timeout=`). If the second physical drive's link/spin-up/enumeration hasn't completed and udev hasn't created the by-uuid symlink before the timer fires, the `.device` job times out, the `/boot` mount fails, and the boot drops to emergency mode or (with `nofail`) silently skips `/boot`. Because the wait is on a fixed wall-clock timer racing against variable hardware-readiness latency, the failure is intermittent — exactly the symptom described.

---

## 1. Why a secondary drive can be "not ready" when the OS tries to mount it

Several physically distinct latencies stack up, and any one can push the drive's appearance past the mount timeout:

- **HDD spin-up time.** A cold spinning drive takes seconds to reach operating RPM and answer commands. SMART attribute 3 (Spin-Up Time) records this in milliseconds; a healthy raw value is a few thousand ms (~6 s is common), and a *rising* value signals a motor that is degrading and spinning up ever more slowly — a real, progressive cause of "sometimes it makes it, sometimes it doesn't."
- **Staggered spin-up (SSU) / Power-Up-In-Standby (PUIS).** SSU (SATA rev 2.5, 2005) deliberately spins drives up sequentially to avoid a current surge on the PSU/backplane; the drive waits for its SATA PHY to activate before spinning. PUIS holds the drive in standby until an explicit spin-up command from BIOS/HBA. Both *intentionally* delay readiness of secondary drives, and the delay scales with how many drives precede this one. Pin 11 of the SATA power connector also gates SSU (grounded = spin immediately).
- **SATA/SAS link training.** The PHY must negotiate a link ("`ataN: SATA link up 6.0 Gbps`"). A marginal cable/port produces "`link is slow to respond, please be patient`" and retries that each add seconds.
- **Controller / HBA / RAID init.** Drives behind an HBA, RAID controller, or SAS expander only appear after the controller's own firmware POSTs and enumerates its targets. SAS/FC targets are discovered via PHY walk, which is slower than a directly-attached SATA port. The kernel's SCSI mid-layer scans these **asynchronously** by default, so the initramfs/root transition can proceed *before* the scan that would reveal `/boot` has finished (Red Hat bug 466607: "initrd doesn't wait for hba to scan the bus, causing boot failure").
- **USB enumeration delay.** USB mass-storage devices often can't answer immediately after power-up (firmware load / spinning media); `usb-storage.delay_use` exists precisely because "if not given enough time they don't respond correctly."
- **NVMe vs SATA.** NVMe is enumerated over PCIe and is typically ready far earlier than a SATA spinner; mixing a fast root device with a slow SATA `/boot` widens the readiness gap and makes the race more likely to be lost.

---

## 2. udev + systemd device-readiness mechanics

- **Unit creation:** "systemd will dynamically create device units for all kernel devices that are marked with the `systemd` udev tag (by default all block and network devices)." No udevd → no `.device` units. The `/dev/disk/by-uuid/*` symlinks are created by udev *after* it reads the filesystem UUID off the device, so the `.device` unit only becomes active once the drive is up **and** udev has finished probing it.
- **`SYSTEMD_READY`:** Gates when a device counts as "plugged." If `0`, "systemd will consider this device unplugged even if it shows up in the udev tree," and won't act on its `WANTS`. Some controllers expose a device in an uninitialized state first and emit a `change` event when truly ready — until that event, systemd keeps waiting.
- **Mount → device dependency:** the `/boot` `.mount` unit depends on `dev-disk-by\x2duuid-<UUID>.device`. systemd starts the device "job" and blocks the mount on it.
- **`x-systemd.device-timeout=`** sets how long systemd waits for *that backing device unit* to show up before giving up (independent of `TimeoutSec` for the mount operation itself). The relevant default: systemd waits ~90 s (`DefaultTimeoutStartSec`) for a top-level fstab device. If udev hasn't created the by-uuid node by then, the device job fails → mount fails → `local-fs.target` fails → emergency mode. This is the timer that the slow drive races against.
- **Why systemd "gives up":** it has no signal that the device is merely *late* versus *absent*. A drive that needed 95 s on a cold morning looks identical to a permanently missing one, so the timer expires and boot aborts.

---

## 3. Kernel / firmware knobs that make boot wait for drives

| Knob | Where | Effect |
|---|---|---|
| `rootdelay=N` | kernel cmdline | Delays N seconds before mounting **root** (only helps if the slow drive is root; less relevant when only `/boot` is slow, but illustrates the pattern). |
| `rootwait` | kernel cmdline | Waits *indefinitely* for the root device to appear instead of using a fixed delay. |
| `scsi_mod.scan=sync` | kernel cmdline / modprobe.d | Forces **synchronous** SCSI scan so the kernel doesn't proceed until all disks are scanned. Directly fixes the "initramfs/SCSI proceeds before the HBA's disks are enumerated" class of failure. |
| `scsi_mod.disable_async_probing=<driver>` | kernel cmdline | Disables async probing per-driver (e.g. `megaraid_sas`), forcing the controller's targets to be fully discovered before boot continues. |
| `usb-storage.delay_use=N` | kernel cmdline / modprobe.d | Gives USB mass-storage N seconds to settle before use; raise it for slow USB `/boot`, but it adds to every boot. |
| `libata.force=...` / lower link speed | kernel cmdline | Mitigates flaky link training (e.g. force a port to 3.0 Gbps) that causes "slow to respond" retries. |
| BIOS staggered-spin-up delay / SSU / PUIS | firmware setup | **Disable SSU / increase spin-up budget**, or disable PUIS so the secondary drive spins immediately at power-on rather than waiting its turn. This is often the cleanest fix when the drive is a spinner on a backplane. |

The kernel-side wait (`scsi_mod.scan=sync`, `rootwait`) and the systemd-side wait (`x-systemd.device-timeout`) are complementary: the first ensures the device *gets enumerated* before pivot; the second ensures systemd *waits long enough* for it post-pivot.

---

## 4. Device naming is NOT stable — by-UUID is mandatory here

The kernel does **not** guarantee linear, port-based `sdX` assignment. With two drives discovered asynchronously, whichever wins the enumeration race becomes `/dev/sda`; the assignment can flip between boots from "boot-time race-conditions in the kernel and/or motherboard weirdness," after a partition edit, or after a BIOS change. This is the *same* race that causes the timeout — and it means a `/etc/fstab` (or bootloader) entry like `/dev/sdb1 /boot` is doubly broken: it can point at the wrong disk *and* it offers no readiness semantics.

Persistent identifiers solve naming and tie into the readiness wait:
- **`UUID=` / `/dev/disk/by-uuid/`** — filesystem UUID, survives reorder; this is what the `.device` unit is keyed on.
- **`/dev/disk/by-id/`** — drive serial/model + WWN; best for "this physical disk," independent of filesystem.
- **`by-path`** — hardware topology (port); stable only if you never move cables.

For this server, `/boot` must be referenced by `UUID=` (or `by-id`) in fstab and in the bootloader config (GRUB resolves `/boot` via UUID/`search --fs-uuid`). Using `/dev/sdX` for a second physical drive in a two-drive race is the underlying bug, not just bad style.

---

## 5. Prioritized diagnostics

Run these after a boot (ideally after one that *failed*, before rebooting clears volatile state):

1. **`journalctl -b -1 -p warning`** (and `-b` for current) — look for `dev-disk-by\x2duuid-<UUID>.device/start timed out`, `Timed out waiting for device`, `Dependency failed for /boot`, and the drop to `emergency.target`. This confirms it's the `.device` timeout and identifies which UUID.
2. **`journalctl -b -1 -k` / `dmesg -T`** — read the **timestamps** for each port: `ataN: SATA link up …`, `scsiN`, `sd X:Y:Z:0: [sdX] … logical blocks`. Compare the time `/boot`'s drive came up against the systemd timeout. Watch for `link is slow to respond, please be patient`, link resets, or repeated `SRST`/`hardreset` (marginal cable/PHY).
3. **`lsblk -f` and `blkid`** — confirm the actual current `sdX↔UUID` mapping and verify the UUID in fstab/bootloader matches the physical `/boot` drive. Run across a few reboots to demonstrate whether `sda/sdb` flips.
4. **`smartctl -a /dev/<boot-drive>`** — inspect **attribute 3 Spin_Up_Time** (rising raw value or falling normalized VALUE = motor degrading/slow spin), plus Reallocated_Sector_Ct, Command_Timeout, and run `smartctl -t short`. A dying drive that spins up slowly is a genuine, escalating cause. Also `smartctl -l scterc` for error-recovery timeouts.
5. **`systemctl status 'dev-disk-by\x2duuid-*.device'`** and **`systemd-analyze blame` / `systemd-analyze critical-chain boot.mount`** — see how long the device job actually waited and whether it's near the limit.

## Prioritized mitigations (hardware-first, then software)

1. **Fix the references first (mandatory, zero-risk):** ensure `/boot` is mounted by `UUID=` in `/etc/fstab` and resolved by UUID in the bootloader. Never `/dev/sdX` for a racing second drive.
2. **Give systemd more patience for that mount:** add `nofail,x-systemd.device-timeout=120` (tune up) to the `/boot` fstab entry — but understand `nofail` *silently skips* a failed mount, so for `/boot` prefer raising the timeout over relying on `nofail`. Better still, `noauto,x-systemd.automount` mounts `/boot` on first access rather than blocking boot.
3. **Address the hardware readiness at the source (best long-term):**
   - In BIOS/HBA, **disable staggered spin-up / PUIS** for the `/boot` drive (or raise the spin-up budget) so it's ready at power-on.
   - Replace the SATA **cable** and try a different **port/controller** if dmesg shows link resets / "slow to respond."
   - If `smartctl` shows a rising Spin_Up_Time or pending/reallocated sectors, **replace the drive** — no software knob fixes a dying motor.
4. **Make the kernel wait for enumeration** (especially if `/boot` is behind an HBA/RAID/SAS expander or USB): add `scsi_mod.scan=sync` (and/or `scsi_mod.disable_async_probing=<driver>`); for USB `/boot`, raise `usb-storage.delay_use=`. Add `rootwait`/`rootdelay` only if the *root* device is also affected.
5. **Strategic fix:** the cleanest design is to eliminate the cross-drive race — e.g. relocate `/boot` onto the same physical drive as `/`, or mirror it — so boot no longer depends on a slow second spindle winning a timer.

---

## Sources

- [systemd.device(5) — freedesktop / man7](https://man7.org/linux/man-pages/man5/systemd.device.5.html) — `.device` unit creation from udev tag, `SYSTEMD_READY`, `SYSTEMD_WANTS`, mount↔device dependency
- [systemd issue #6402 — Root FS device times out after 90 seconds](https://github.com/systemd/systemd/issues/6402) and [systemd-devel: longer start timeout for .device units](https://systemd-devel.freedesktop.narkive.com/aO2Aczjg/best-way-to-configure-longer-start-timeout-for-device-units) — 90 s `DefaultTimeoutStartSec` for fstab device units
- [Arch Forums — Timed out waiting for device dev-disk-by\x2duuid](https://bbs.archlinux.org/viewtopic.php?id=213548) and [Proxmox — same](https://forum.proxmox.com/threads/solved-timed-out-waiting-for-device-dev-disk-by-uuid.69757/) — real-world timeout symptom + `x-systemd.device-timeout`
- [LWN — USB and fast booting](https://lwn.net/Articles/330378/) and [Witekio — usb-storage delay_use](https://witekio.com/blog/using-delay_use-to-speed-up-usb-enumeration/) — USB enumeration settle time
- [archzfs issue #94 — recommend scsi_mod.scan=sync](https://github.com/archzfs/archzfs/issues/94) and [Red Hat bug 466607 — initrd doesn't wait for HBA scan](https://bugzilla.redhat.com/show_bug.cgi?id=466607) — synchronous SCSI scan / HBA enumeration race
- [LWN — Asynchronous scanning for FC/SAS](https://lwn.net/Articles/205588/) and [openSUSE forums on disk order](https://forums.opensuse.org/t/problem-with-disks-order-after-snapshot-20230921/169324) — async probe / disable_async_probing
- [Wikipedia — Spin-up (PUIS / SSU)](https://en.wikipedia.org/wiki/Spin-up) and [45Drives — Using Staggered Spin Up](https://knowledgebase.45drives.com/kb/using-staggered-spin-up/) — SSU/PUIS, Pin 11, current-surge rationale
- [Arch Forums — "ata2: link is slow to respond"](https://bbs.archlinux.org/viewtopic.php?id=98720) and [nixCraft — SATA link speed via dmesg](https://www.cyberciti.biz/faq/linux-command-to-find-sata-harddisk-link-speed/) — link training / dmesg timestamps
- [Acronis KB 9103 — SMART Spin-Up Time attribute](https://kb.acronis.com/content/9103) and [TrueNAS — SMART spin up value](https://www.truenas.com/community/threads/smart-spin-up-time-value.32802/) — attribute 3 interpretation
- [Arch Forums — sda↔sdb name changes](https://bbs.archlinux.org/viewtopic.php?id=214235) and [oneuptime — persistent udev naming on RHEL](https://oneuptime.com/blog/post/2026-03-04-configure-persistent-device-naming-udev-rhel/view) — unstable `sdX`, use UUID/by-id

(Note: this was a pure web-research task with no repo work, so no `memory.md`/wiki updates apply.)

Research is in across four angles (systemd/fstab, bootloader/GRUB, hardware enumeration, and Xen-specifics). Here's the synthesis.

## The key insight: "/boot" gets touched at two totally different times

This is what trips people up. `/boot` is read by **two different actors at two different stages**, and the "race" can be in either one — with completely different fixes:

| Stage | Who | When | Symptom |
|---|---|---|---|
| **(a) Firmware → GRUB** | BIOS/UEFI + GRUB read `xen.gz`/`vmlinuz`/`initrd` *files* off the `/boot` drive | at power-on, before Linux exists | **no GRUB menu**, "no boot device", or `grub rescue>` |
| **(b) systemd mount** | systemd *mounts* `/boot` per `/etc/fstab` after the kernel is up | late in boot | GRUB works, kernel boots, then **`Dependency failed for /boot`** / drops to emergency mode |

**Decisive test:** if the friend ever sees the GRUB menu / the kernel starts loading, then GRUB already read `/boot` fine this cycle — so it's **mode (b)**, an ordinary systemd/fstab race, *not* a GRUB problem. Have them run `journalctl -xb` and look for `Timed out waiting for device` / `Dependency failed for /boot`.

**On Xen specifically:** it changes essentially nothing. Once dom0 is up it's a normal systemd Linux, so the mount race (b) is identical to bare metal. Xen only adds that GRUB must load `xen.gz` from `/boot` at stage (a) — same reachability requirement bare-metal GRUB already has for `vmlinuz`. (Note "xenn" is ambiguous — if it's actually **XCP-ng/XenServer**, that appliance manages its own partitions and you don't hand-edit fstab, so suspect BIOS boot-order/UEFI-mode instead; if it's **Ubuntu 16.04 "Xenial"**, same generic fix but it's now fully EOL.)

## Mode (b) — the likely one — fix the fstab line

Two root causes stack here, and **both** need addressing:

**1. Stop using `/dev/sdX`.** Since kernel 5.3, SATA/SCSI probing is asynchronous, so `sda`/`sdb` can swap between boots — an fstab entry like `/dev/sdb1 /boot` can point at the wrong disk or one not ready yet. Get the real UUID with `lsblk -f` / `blkid`.

**2. Make `/boot` non-critical** so a slow drive can't drop the box into emergency mode. Pick one:

```fstab
# Strategy A — stays mounted, but never blocks/fails boot:
UUID=XXXX-XXXX  /boot  ext4  defaults,nofail,x-systemd.device-timeout=15  0 2

# Strategy B (recommended) — out of the boot graph entirely, auto-mounts on kernel updates:
UUID=XXXX-XXXX  /boot  ext4  defaults,noauto,x-systemd.automount,x-systemd.idle-timeout=120  0 2
```

Critical detail: **`nofail` alone keeps a 90-second stall** (it still waits the default device timeout) — you *must* pair it with `x-systemd.device-timeout=`. Strategy B is cleanest because `/boot` isn't needed at runtime at all (only at kernel-update time), and automount transparently triggers exactly when `apt`/`dnf`/`pacman` writes a new kernel. Avoid bare `noauto` — that's a foot-gun (a kernel update with `/boot` unmounted writes to the empty mountpoint on root instead of the real partition).

After editing: `systemctl daemon-reload && mount -a && findmnt /boot` to validate **before** rebooting.

## If it's the hardware that's genuinely slow

If `journalctl -k`/`dmesg -T` timestamps show the `/boot` drive's `ata link up` arriving late (or `link is slow to respond`), the timeout bump above buys time, but also:
- **Check `smartctl -a` attribute 3 (Spin_Up_Time)** — a rising value means a dying motor spinning up ever slower (a real, escalating cause of intermittent boots). Replace the drive if so.
- In BIOS/HBA, **disable staggered spin-up (SSU) / PUIS** for that drive so it spins at power-on, or raise the spin-up budget.
- Swap the SATA cable/port if dmesg shows link resets.
- For drives behind an HBA/RAID, add kernel cmdline `scsi_mod.scan=sync`.

## Mode (a) — if GRUB never appears

The secondary `/boot` drive isn't ready when firmware enumerates. Fixes, best first:
1. **Move the ESP + GRUB + `/boot` onto the primary fast drive** alongside `/`. This is the real long-term fix — it kills the race in *both* stages at once and matches every distro default.
2. Add a 3–5s firmware **POST/boot delay**; set explicit boot order with `efibootmgr`; check power/cabling.
3. `rootdelay=` does **not** help here (or in mode b) — it only delays the *root* mount in initramfs and is a commonly misapplied fix.

---

**Bottom line / what I'd tell the friend:** run `journalctl -xb` to confirm it's the systemd mount race (almost certainly is), then switch the `/boot` fstab entry to `UUID=` with `nofail,x-systemd.device-timeout=15` (or better, the automount line). If it keeps racing, the drive itself is spinning up too slowly — check SMART and BIOS staggered-spin-up. The genuinely correct architecture is to not put `/boot` on a separate slow spindle at all — relocate it to the primary drive.

Want me to write this up as a research folder under `~/claude/research/` (with `memory.md` + Forgejo repo + wiki page), or is this chat-only?

• Write /home/megaproxy/claude/research/boot-race-separate-boot-drive-debug.md 23ms 168 B

Saved to `/home/megaproxy/claude/research/boot-race-separate-boot-drive-debug.md`.

It's a self-contained debug doc with: the two-stage diagnosis (firmware/GRUB vs systemd mount), the decisive test to tell them apart, the fstab fixes (UUID + `nofail,x-systemd.device-timeout=` or automount), the hardware/SMART/BIOS angle, a command cheat-sheet, the bottom-line action list, and cited sources.

Two notes:
- This is a loose file in the `research/` root, not a full research *folder* (no `memory.md`/git repo). Per your setup, loose markdown like this would more naturally live in `~/claude/notes/` — say the word and I'll `mv` it there, or promote it to a proper `research/<topic>/` folder with git + wiki page if it's worth tracking.
- It opens with the friend's actual symptom; the very first thing they should run is `journalctl -xb` to confirm which of the two failure modes it is before changing anything.