LVM PE migraged to new PV, now Linux boot stops after decryption of LUKS device

Question

I migrated all LV extents from one PV to a new PV. Both old and new PVs were LUKS encrypted devices. root is one of the migrated LVs.

This didn't work as, after migration, boot wouldn't even try to unlock the new LUKS PV. It was asking for password for old LUKS, which doesn't exist anymore, those drives have already been reassigned and overwritten.

However, if I use the kernel parameters rd.auto=1, and remove old kernel parameters which specify the UUID of the old PV, I can get linux to boot and ask for the right password. Yet, it still doesn't work.

Before I give the password it says

dracut-initqueue /usr/bin/crypt-run-generator: line 14: cryptsetup: command not found Unnecessary job for dev-mapper-luks-<UUID of new PV LUKS> was removed

So, it automatically finds the new LUKS partition, and let's me unlock it, but that's all after it has given up on finding usable filesystems on it. Why would it do that?

At this point, the boot process stalls waiting indefinitely for required jobs to complete. I forget all their names, but one is cryptsetup.target and I think one was mounting root. So, that makes sense.

But, I don't know how to configure grub and dracut so that this will work. It's like the initrd has a configuration telling it only to start LVM on the old PV. When I boot a live image all the LVs are there and data looks great.

What grub2 (efi mode) configuration and dracut configuration do I need to fix this?

EDIT Since posting, I've tried many things. I have found kerel params that I think are right, but they don't seem to make any difference. I created a new initrd image with dracut to which I forcefully added cryptsetup and a new, fixed /etc/crypttab. That did seem to help and all the errors are now gone...but it still will not boot! Using the latest options and initrd I've tried, the boot process does not ask for any passphrase. It says "Sending to plymoth for password request" but that just stalls.

Now, one of the kernel options I tried was rd.shell. This gave me a dracut shell after it timed out, unable to find root. From the dracut shell I can start the mdadm RAID PV with --assemble --scan, luksOpen it with my passphrase, and all the LVs are right there. So I still don't understand what it wants.

Answer 1

As I hoped, the "standard" approach to configuring Linux boot applies, but it took me a long time to relearn it, as a lot has changed since I had to use it last.

• boot live image
• assemble MD, unlock LUKS, LVM check, etc.
• build target system in chroot jail
  • mount /newroot ../boot ../boot/efi and ../var, at least
  • mount -t sysfs sysfs /newroot/sys
  • mount -t proc none /newroot/proc
  • mount -t bind /dev /newroot/dev
  • mount -t bind /run /rnewroot/run
• chroot /newroot
• from inside jail, fix stuff as needed (/etc/fstab /etc/crypttab /etc/mdadm.conf /etc/lvm/ etc.)
• and, fix grub2 CMDLINE env var with new kernel parameters
  • dracut --print-cmdline on Fedora/RHEL/CenOS will output a minimal set of parameters. This is super helpful as it's hard to know what your distro needs and to keep up with latest kernel options. Moreover, saves you the trouble of tracking down UUIDs for luks/md devices!
  • for Ubuntu?
  • for Arch?
• build new grub2 config
  • grub2-mkconfig [-o ]
• install grub2, if necessary
  • for BIOS grub2-install
  • for EFI, distro dependent
  • dnf reinstall grub-efi grub-efi-modules shim for Fedora?
  • Ubuntu?
  • Arch?
• regen intird/initramfs boot image
  • dracut for Fedora
  • Ubuntu?
  • Arch?