Fastest way to convert an ext4 formatted disk to LVM with ext4 on it?

Question

I'm currently doing cp -aR to copy data from my (99% full) 1TB ext4 formatted disk to a new LVM-with-ext4-on-it disk. It's taking forever.

Is there any way to attempt to "convert" the disk in place? I'm on EC2 so backing up takes minutes.

Or alternatively, is there any way that might be faster than cp to directly copy the ext4 filesystem onto the LVM disk?

Answer 1

I wrote blocks (née lvmify) which does this conversion in-place. It works by shrinking the filesystem a bit, moving the start of the filesystem to the end of the partition, and copying an LVM superblock (preconfigured with the right PV/LV/VG) in its place.

Answer 2

I'm unsure about how to convert the disk live, but I think rsync will be a better and safer way to copy your data over. It'll allow you to resume and keep the data intact in the event the transfer stops.

I did find a similar process completed by someone adding an external drive to their local system as an LVM. There's not a whole lot of information, but I think it will be enough to get your started:

"So today I discovered the awesome that is LVM. Installing Debian, I selected "LVM - Use entire disk". But the main drive was a slow and small 5200rpm laptop drive. Today I inserted my spare 1.5TB drive and booted up. Wanted the system on this bigger faster drive instead.

LVM approach: add /dev/sdc to the volume group, then run "pvmove /dev/sda". This moves all data from sda to other drives (only sdc available). No need to reboot, no need to unmount. While I'm writing this, the data is being moved.

Later, do "vgreduce megatron /dev/sda" to remove the slow drive from the volume group and voila. Data moved. (megatron is the name of the volume group and of my computer). This might be old news to many but I just thought this was really cool :)"

Granted this was done locally, but I think with additional research, you maybe be able to accomlpish this.

source

Answer 3

No method exists to do such an LVM conversion "on the fly."

LVM actually resides under your filesystem to expand it across multiple physical devices, or stripe or mirror it, etc. The final step in creating a logical volume is to lay a file system on top of it. Here are examples from RHEL6.

What you will need to do is back up or archive the data on that device, then destroy existing filesystem, create a logical volume, and re-lay a filesystem on top. If this is a root filesystem on a linux OS, consider doing a reinstallation. It may be faster.

For faster copies, I'm fond of dd, but I don't use it often. Making a mistake with dd is dangerous. One thing you can try is tar czv <source fs> | (cd <destination fs>; tar x) which will transfer a compressed tar file on the fly.

Good luck!

Answer 4

Manual lvmify:

1. Shrink filesystem by 8 MiB (if the length is a multiple of 4 MiB shrinking by 4 MiB is enough; if not sure shrink a bit more and expand filesystem to maximum size after step 7) or increase the size of the partition to a multiple of 4 MiB and having 1 free 4 MiB block at the end.
2. Copy the first 4 MiB to the location of the last 4 MiB aligned block, e.g. with a 17 MiB disk that's at 12 MiB (toy example for simplicity, see illustration below).
3. Create PV with --dataalignment 4M
4. Create VG
5. Create LV with --extents 1, -Zn and PV specification /dev/sdX:1234 where sdX is your disk (add the partition number if used) and 1234 the last extent number, i.e. number of available extents - 1. In the example, we have 3 extents (4 MiB header, 12 MiB usable space, 1 MiB trailing space) and the index of the last extent is 2.
6. Grow LV (lvextend) to full size, e.g. with -l+100%FREE
7. Check plausibility of the data in the new LV. Steps 5+6 can be repeated with corrected numbers after removing the LV. (Assuming no journal replay damaged the data.)

Illustration (toy example with 17 MiB disk):

Meta| PV usable | trailing data
+---+---+---+---+-+
| 0 | 1 | 2 | 3 | |
+---+---+---+---+-+
  |           ^
   \----------/ copy

Shrinking filesystem to 3 extents (12 MiB).
LV extents: 0 (was 3), 1, 2

Ideas for testing this approach before touching important data:

• Create a sparse file and a loop device (or use a VM), fill the first and last 20 MiB or so with random data (or sequential block numbers), get the md5sum of the relevant data (excluding the area we overwrite in step 2 and the trailing data), follow steps, check that md5sum of LV contents matches earlier md5sum
• Create a device snapshot (100 MiB backing storage should be plenty, this can be on a tmpfs), e.g. following https://gist.github.com/jowagner/b36024636140ddf453c12eaf6e590b5d, and work on the writeable snapshot for testing.

If you are not 100% sure like me whether dd seek= does the right thing for step 2 here a few lines of code that I used:

dev_name = 'dev/sdX'
dev_size = 24000272007168  # put size in bytes here
pe_size = 4096 * 1024
n_extents = dev_size // pe_size
f = open(dev_name, 'r+b')
f.seek(0)
block = f.read(pe_size)
assert len(block) == pe_size
f.seek(pe_size*(n_extents-1))
f.write(block)
f.close()