4

This was my home file storage setup. It does not have backups because the RAID setup was meant to be the redundancy. I did not account for what happened and am paying the price. The setup:

  • Ubuntu 16.04
  • Four-disk RAID 5 array using mdadm (4x2TB): /dev/md0
  • On the array, a PV and LV managed by LVM.
  • On the Logical Volume named vg0, an XFS file system.

Note that the Linux host, including /etc and /boot, are installed on a different disk and are completely accessible (so I have access to /etc/lvm/archive). The RAID array is purely file storage, the boot process has no dependency on it at all other than its entry in /etc/fstab.

For whatever reason I had booted from a FreeDOS installer which I was struggling to understand. I think I may have told it to repartition this volume, although I cannot remember doing so. In any case, when I rebooted into Linux (Ubuntu 16.04), I was dropped into a recovery mode prompt as the root user. It could not mount the UUID of the volume group as defined in /etc/fstab.

It has been sufficiently long enough since I originally setup this RAID array that I completely forgot how LVM worked, or that I even used LVM to create the volume. (10-12 years, replacing hard disks and resizing the array occasionally over the course of that time.) So, first I tried to use testdisk [1] to find and restore the partition information. This never worked, the partition was always the incorrect size (524Gb instead of 4.5TB) and never on a "physical sector boundary." I experimented with various geometries thinking there was a magic combination that would perfectly restore the partition. Here is the current status of the disk according to fdisk:

$ sudo fdisk -l /dev/md0
GPT PMBR size mismatch (1098853631 != 200894463) will be corrected by w(rite).
Disk /dev/md0: 4.1 TiB, 4500904476672 bytes, 8790829056 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 1048576 bytes / 3145728 bytes
Disklabel type: dos
Disk identifier: 0x00000000

Device     Boot Start        End    Sectors  Size Id Type
/dev/md0p1          1 1098853631 1098853631  524G ee GPT

Partition 1 does not start on physical sector boundary.

And parted:

(parted) print list                                                       
Error: /dev/md0: unrecognised disk label
Model: Linux Software RAID Array (md)                                     
Disk /dev/md0: 4501GB
Sector size (logical/physical): 512B/4096B
Partition Table: unknown
Disk Flags: 

In posting a question to the testdisk forum [2], I realized that I had used LVM to manage the RAID array, and that it was possible that these do not use a traditional partitioning tool at all. Researching "recovering lvm physical volumes" dug up http://blog.adamsbros.org/2009/05/30/recover-lvm-volume-groups-and-logical-volumes-without-backups/. pvck tells me the following:

$ sudo pvck /dev/md0
  Incorrect metadata area header checksum on /dev/md0 at offset 4096
  Found label on /dev/md0, sector 1, type=LVM2 001
  Found text metadata area: offset=4096, size=192512
  Incorrect metadata area header checksum on /dev/md0 at offset 4096

I also have several backups of the LVM volume in /etc/lvm/archives, the latest being the following:

crw@bilby:~$ sudo cat /etc/lvm/archive/vg0_00002-935168089.vg
# Generated by LVM2 version 2.02.98(2) (2012-10-15): Sun Jul 19 12:00:04 2015

contents = "Text Format Volume Group"
version = 1

description = "Created *before* executing 'lvextend /dev/vg0/lv0 /dev/md0'"

creation_host = "bilby" # Linux bilby 3.16.0-43-generic #58~14.04.1-Ubuntu SMP Mon Jun 22 10:21:20 UTC 2015 x86_64
creation_time = 1437332404  # Sun Jul 19 12:00:04 2015

vg0 {
    id = "Q4ZRRc-1l0h-FEgu-jrxA-EfW1-tAis-vv0jyL"
    seqno = 5
    format = "lvm2" # informational
    status = ["RESIZEABLE", "READ", "WRITE"]
    flags = []
    extent_size = 262144        # 128 Megabytes
    max_lv = 0
    max_pv = 0
    metadata_copies = 0

    physical_volumes {

        pv0 {
            id = "bKQs0l-zNhs-X4vw-NDfz-IMFs-cJxs-y0k6yG"
            device = "/dev/md0" # Hint only

            status = ["ALLOCATABLE"]
            flags = []
            dev_size = 8790828672   # 4.09355 Terabytes
            pe_start = 384
            pe_count = 33534    # 4.09351 Terabytes
        }
    }

    logical_volumes {

        lv0 {
            id = "pqInOe-ZLpV-t9oK-GQE1-AoIt-mB3M-4ImaV1"
            status = ["READ", "WRITE", "VISIBLE"]
            flags = []
            segment_count = 1

            segment1 {
                start_extent = 0
                extent_count = 22356    # 2.729 Terabytes

                type = "striped"
                stripe_count = 1    # linear

                stripes = [
                    "pv0", 0
                ]
            }
        }
    }
}

If it is helpful, the following is the detail on the RAID array:

$ sudo mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sun Oct 11 13:34:16 2009
     Raid Level : raid5
     Array Size : 4395414528 (4191.79 GiB 4500.90 GB)
  Used Dev Size : 1465138176 (1397.26 GiB 1500.30 GB)
   Raid Devices : 4
  Total Devices : 4
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Mon Oct  3 13:12:51 2016
          State : clean 
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 1024K

           UUID : 9be3b2f7:102e373a:822b5a8f:216da2f7 (local to host bilby)
         Events : 0.103373

    Number   Major   Minor   RaidDevice State
       0       8       64        0      active sync   /dev/sde
       1       8       48        1      active sync   /dev/sdd
       2       8       16        2      active sync   /dev/sdb
       3       8       32        3      active sync   /dev/sdc

Finally, here is the sad trail of testdisk.log that I have left behind: https://dl.dropboxusercontent.com/u/2776730/testdisk.log

edit: output of lsblk:

crw@bilby:~$ sudo lsblk
NAME                 MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                    8:0    0 59.6G  0 disk  
├─sda1                 8:1    0  243M  0 part  /boot
├─sda2                 8:2    0    1K  0 part  
└─sda5                 8:5    0 59.4G  0 part  
  ├─bilby--vg-root   252:0    0 43.4G  0 lvm   /
  └─bilby--vg-swap_1 252:1    0   16G  0 lvm   [SWAP]
sdb                    8:16   0  1.8T  0 disk  
└─md0                  9:0    0  4.1T  0 raid5 
sdc                    8:32   0  1.8T  0 disk  
└─md0                  9:0    0  4.1T  0 raid5 
sdd                    8:48   0  1.8T  0 disk  
└─md0                  9:0    0  4.1T  0 raid5 
sde                    8:64   0  1.8T  0 disk  
└─md0                  9:0    0  4.1T  0 raid5 

I am completely lost and suspect I have made things worse. My questions are:

Do I need to "fix" the partition information before dealing with LVM issues? Should I attempt to "pvcreate --uuid xxx --restorefile yyy"? And then would I need to extend the disk, and run something like the xfs equivalent of fsck? Or is my data lost to me at this point? :'(

Please let me know if there is anything I can add to make debugging this issue easier. Thanks!

  • 2
    Bear with me, I'm reading through this and it's going to take me a while to sort it out. This looks very recoverable. In the meantime, you might get downvoted on here and moved to SuperUser, as you mention this is a home machine. You should move this question to avoid glares. – Spooler Oct 4 '16 at 6:39
  • 1
    Apologies, I am not familiar with the domain specificities of StackExchange. When you say "move," do you mean just copy'n'paste this question over to that forum? Thanks! – Craig Wright Oct 4 '16 at 6:42
  • 2
    Eh, apparently you need 250 reputation to do that yourself. This is getting meta. I'll flag it and things will happen. No worries. – Spooler Oct 4 '16 at 6:46
  • OK. Just let me know, I can just repost the question. I never understood the difference before, but after looking, this question clearly belongs in SuperUser. – Craig Wright Oct 4 '16 at 6:48
  • 1
    Please post the output of lsblk. I need the data in there to isolate problems. Specifically, I'm wanting to see if you have any partition tables on your RAID members (sdb to sde), but it could provide other useful data. – Spooler Oct 4 '16 at 7:05
2

If any of this starts to not work, or stops making sense, STOP and ask a subject matter expert. This is unsafe work. Operate on disk images copied by "dd" to either files on a large storage medium, or directly to new disks of equal or greater size to protect your original dataset from tomfoolery. You may perform these operations on a single live set, but if you mess up that could be it for your data.

Alright. To begin, we need to repair this storage stack methodically, from the base disk level up. You ran a FreeDOS installer, and that messed with your disks by (presumably) creating a partition table on one of them.

Your disks participate in the MD array directly, no partition table to speak of. This is fairly typical. However, this is also a 0.90 revision metadata structure on that array, so putting a partition table on any of those disks directly will mess with the array.

Check as to whether you have a disk (any from sdb to sde) that has a partition table on it, in the form of /dev/sdb1 for example. If you have one like this, you will need to consider it dirty and take it out of your array, placing it back in after getting rid of that table.

Even if we don't see a partition on one of those disk, an integrity check needs to be run on /dev/md0. The command to do this is simple:

# /usr/share/mdadm/checkarray -a /dev/mdX

If that comes back with a mismatch count of greater than zero, then that array will have to be repaired. We'll visit that if need be, as it currently doesn't look like the issue.

On to more concrete problems, testdisk put a GPT on /dev/md0, and a partition on that disk (/dev/md0p1). This was never supposed to be there, and is corrupting your LVM metadata. You volume group is meant to reside directly on /dev/md0, as that's the way you originally created it.

First, we will have to deal with that errant GPT on /dev/md0. It needs to be "zapped". Zapping a GPT will blank all GPT structures, returning it to a disk with no table, as it should be in this case. This article details that excellently: "http://www.rodsbooks.com/gdisk/wipegpt.html". If you don't zap it, you will have a broken GPT structure on that disk that partitioning utilities will try to "correct", causing problems for you down the road all over again.

After doing that, you now can re-create all of your LVM metadata using the archive file you posted in your question. Thankfully, you've given me enough information to just hand you a command that will work. If you want to know more about this process, this is a great resource: "https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Logical_Volume_Manager_Administration/mdatarecover.html".

The command to recreate your physical volume with all of its original metadata:

# pvcreate --uuid "bKQs0l-zNhs-X4vw-NDfz-IMFs-cJxs-y0k6yG" --restorefile /etc/lvm/archive/vg0_00002-935168089.vg

This archive file describes /dev/md0 as being the disk that constitutes your volume group, and will use that, as it should. If you have a later archive file in your LVM archives directory, USE THAT INSTEAD. The goal is to bring the volume group to its latest valid state.

After this, checking your PV, VG, and LV integrity is key. You've already attempted this, but this time it should be more productive. The commands pvck and vgck are what should be used here.

First, perform pvck:

# pvck /dev/md0

After that validates, run vgck:

# vgck vg0

Once you have validated all metadata, it's time to activate your LVs, if they are not already:

# vgchange -ay vg0

And finally, checking the filesystem on /dev/mapper/vg0-lv0 (which in your case is XFS) for potential errors:

# xfs_check /dev/mapper/vg0-lv0

That should return nothing if there are no errors. If something is amiss, then xfs_repair will be necessary (DON'T DO THIS WHILE IT's MOUNTED):

# xfs_repair /dev/mapper/vg0-lv0

  • First hitch: running the "checkarray" command does not seem to do anything, or it happens so quickly that I miss it. I ran it a couple of times while also running the following: "watch cat /proc/mdstat" and "watch -n 1 cat /sys/block/md0/md/sync_action" and did not see any change to either status. Based on the information I found here: thomas-krenn.com/en/wiki/Mdadm_checkarray, I should be seeing a more-lengthy check process. On the plus side, " tail /sys/block/md0/md/mismatch_cnt" returns 0. So... all good? I hope? – Craig Wright Oct 4 '16 at 17:14
  • Additionally, the lsblk I ran above seems to indicate that all of the member disks of the array are partition-free. So another positive indicator. – Craig Wright Oct 4 '16 at 17:16
  • Another interesting tidbit. I am running all the non-destructive commands while I wait for a backup disk to arrive, to perform the recommended "dd" backup. :) This is the output of gdisk: crw@bilby:~$ sudo gdisk /dev/md0 GPT fdisk (gdisk) version 1.0.1 Partition table scan: MBR: protective BSD: not present APM: not present GPT: not present Creating new GPT entries. Command (? for help): So seems that there is no GPT scheme installed? So confusing. Either way, it looks like when it is time, zapping with gdisk will eliminate the GPT and MBR. – Craig Wright Oct 4 '16 at 17:28
  • Worked! A couple of minor notes, possibly needing correction, for your answer: I needed to run # vgcfgrestore vg0 before the volume group could be seen or checked. xfs_check no longer seems to be included with xfsprogs...? I ran xfs_repair -n ... instead, and then just ran xfs_repair for good measure. Otherwise, a miraculous successful data recovery. Thank you very much! – Craig Wright Oct 6 '16 at 23:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.