23

Final update:

I already knew what I needed to do to fix this problem; I just didn't know how to do it. I was hoping there would be some ready-made tool to do that automatically - but couldn't find any. I am accepting Rod's answer because despite not directly solving my issue, it gives a very good background on the sector size issue, and gave me confidence that the issue really was partition alignment and addressing. For those that come to this question having the same issue, read it thoroughly and carefully, including comments, before doing anything.


In the beginning

I had a computer and needing more space I've bought a new 500GB drive and an USB enclosure. Soon I have noticed that if I partitioned the drive on the enclosure and moved it to the computer, it wouldn't recognize the partitions (and vice-versa). I assumed it was a problem with the enclosure and didn't worry about it.

Then, tragedy

A wonderful day, my computer decided to not turn on anymore. Turns out the motherboard (unbranded, just a big MADE IN CHINA printed on it) is dead. I've been using it as a file-server and that 500GB drive is now full of data that I can not afford to lose. I am broke now and can not afford a new computer, so my only hope was the "defective" USB enclosure.

The investigation

Armed with several Linux distributions, a laptop, VirtualBox and the enclosure I did a forensic analysis on the issue. dmesg reported partition size was beyond end-of-drive. So I went through hard drive datasheets, calculated sector counts from scratch, tested drive boundaries manually with dd, and everything looked OK, until I fired up fdisk and it said:

    Note: Sector size is 4096 (not 512).

How modest of fdisk. This "note" was the root of all the issues. After some more fiddling these conclusions were drawn:

  • The USB enclosure is not defective.

  • The SATA controller on the now dead motherboard is the one that was "weird", at least. It did not report 4096-byte sectors to the operating system, so the OS happily created the MBR using 512-byte sector addresses.

  • Now when I try to access the partition, the OS tries to use the 512-byte based addresses on a 4096-byte sector drive, and of course, it's not gonna work.

The question

  • So, how can I correct the addresses in the MBR so they are valid on a 4096-byte sector size, aside from manually editing the MBR on an hex-editor, and

  • The partitions are not aligned for 4096-byte sectors. There is some tool available to align them aside from copying in and out of another drive? (I do not have spare drives), or will I need to create some tool that "shifts" the data to the side a little chunk at a time? Partitions are ext3.

Thanks!

Update:

I found there is a clever way to use dd to shift the partition in-place in this question: How to move a partition in GNU/Linux? But I don't know if it will work on a slice of a sector, though. I can't test it right now but will do when I have some time.

Update 2:

So I have successfully aligned the partition using the method above and hand-edited the MBR on a hex editor. As soon as I re-plugged the HDD, boom partition automatically mounted! I do not recommend this though, there were I/O errors during the process and I could have lost everything, see comment on Rod's answer. For the other partition I will not take risks and will use an old HDD and align chunks at a time by copying the data and then pasting it back on a different position.

  • don't know but a remark- sounds like you could give lessons in how computers work! (and then if it helps solve the problem, buy another hard drive with the cash) – barlop Nov 23 '13 at 16:20
  • @barlop Thanks! But I already have to split my day between my job and college so a second job is a no-go right now ;) I will have to fix these partitions the hard way =) – NothingsImpossible Nov 23 '13 at 17:06
  • 1
    MAN IT'S 6 AM AND I SPENT ALL THE LAST NIGHT AROUND THIS PROBLEM! – Leonel Apr 7 '14 at 9:08
  • 1
    Ok, so I have the opposite problem: I have a 1TB disk formatted using the enclosure. So it was formatted using 4096 bytes per sector addresses. I'm not comfortable editing the MBR by hand. And I need to use the HDD directly on SATA (512 byte per sector) Any suggestions? – Leonel Apr 7 '14 at 9:58
  • 1
    @Leonel You can use Linux fdisk to edit the MBR (I learned this later, no need for hex-editors :)) You can change each entry start point and size, and review the changes before applying. So: start fdisk, note down the current configuration (or better, backup the MBR with dd), multiply the start address and size values by 8 and alter them. Be sure to check everything with a calculator and to understand what the values mean. You'll see that Size = End - Start + 1, and that fdisk shows the size in 1000-sectors unit, so you may need to turn expert mode on to see the real value, etc. – NothingsImpossible Apr 7 '14 at 10:15
24

Sector-size issues are becoming quite complex. Until late 2009, the vast majority of hard disks used 512-byte sectors, and that was that. In late 2009, disk manufacturers began introducing so-called Advanced Format (AF) disks, which use 4096-byte sectors. These first AF disks (and, AFAIK, all AF disks today) present an interface to the computer that shows each 4096-byte physical sector as being split up into eight 512-byte logical sectors. This conversion enables older tools, including many BIOSes, that were built with 512-byte assumptions, to continue to work. I don't know if your disk uses AF or not, but in either case, it almost certainly uses a 512-byte logical sector size, meaning that the interface to the OS should use 512-byte sectors.

Complicating matters is certain USB disk enclosures. Some of these enclosures do the reverse of what AF does: They take eight disk sectors and bundle them into one new 4096-byte sector. I'm not sure what the reasoning is behind this move, but one practical advantage is that disks larger than 2TiB can be used with the old MBR partitioning system. One major disadvantage is that a disk partitioned in one of these enclosures can not be used directly or in an enclosure that doesn't do this type of translation. Likewise, a disk prepared without this translation can't be used when it's transferred into such an enclosure. Note that this problem goes well beyond the MBR itself; your disk might identify the first partition as beginning on (512-byte) sector 2048, but if your OS were to seek to (4096-byte) sector 2048, it would not find the start of that partition! You've run into this problem. As such, your initial thought that it's the USB enclosure's fault is closer to the mark than your more recent thought that your motherboard messed it up. I've never heard of a motherboard translating sector size in this way. (Some hardware RAID devices do so, though.)

I don't know of a way to force Linux to adjust its idea of the sector size, but if you have enough disk space, doing a low-level disk copy to another disk may help. For instance:

dd if=/dev/sdb of=~/image.img

This will copy your disk from /dev/sdb (the USB disk; adjust as necessary) to the file ~/image.img. You can then use the following script to mount the image's partitions:

#!/bin/bash
gdisk -l $1 > /tmp/mount_image.tmp
let StartSector=`egrep "^   $2|^  $2" /tmp/mount_image.tmp | fmt -u -s | sed -e 's/^[ \t]*//' | head -1 | cut -d " " -f 2`

let StartByte=($StartSector*512)

echo "Mounting partition $2, which begins at sector $StartSector"

mount -o loop,offset=$StartByte $1 $3

rm /tmp/mount_image.tmp

Save the script as, say, mount_image and use it like this:

./mount_image ~/image.img 2 /mnt

This will mount partition 2 of image.img to /mnt. Note that the script relies on GPT fdisk (gdisk), which most distributions include in a package called gptfdisk or gdisk.

In the long run, a better solution is to find a way to connect the disk that won't do the sector-size translation. A direct connection to a new motherboard should do the trick; or you can probably find an external enclosure that doesn't do the translation. In fact, some enclosures do the translation on USB ports but not on eSATA ports, so if your enclosure has an eSATA port, you could try using that. I realize that these solutions are all likely to cost money, which you say you don't have, but maybe you can trade your translating enclosure for one that doesn't do the translation.

Another option that occurs to me is to try using a virtual machine like VirtualBox. Such a tool might assume a 512-byte sector size when accessing the disk device, effectively undoing the translation; or you might be able to copy the disk's contents raw (as in dd if=/dev/sdc of=/dev/sdb) within the virtual machine, which might copy the contents with compression, thus enabling the image to fit on less disk space than the original consumes.

  • Very insightful answer, but not quite what I was looking for.. I have already tried the virtual machine method but it didn't undo the translation. I have just arrived home, and will try to align the first partition (a smaller, less important one) using dd, and let it run overnight. If successful, I will try to edit the MBR by hand if nobody puts an answer. – NothingsImpossible Nov 24 '13 at 1:13
  • 4
    DO NOT try to modify the disk's contents via dd! Unless you're very careful and understand things extremely well (or are extraordinarily lucky), you're more likely to trash the thing than fix it. It occurs to me that you might be able to adjust the partition table using fdisk: Back up the original and then divide every partition's start point by 8 (and set the end points to end just before the following partition's start point). This stands a chance only if the partition start point values are all multiples of 8. – Rod Smith Nov 24 '13 at 2:53
  • 1
    Holy cow! Thanks for the info. I've been trying to clone my Mac/Windows HDD to an SSD for a day now and I was finally able to identify the issue: the Rosewill SATA/IDE to USB adapter I was using to connect the SSD was performing this "reverse conversion" to 4096-byte sectors! So the GPT + Hybrid MBR on the SSD looked like nonsense after I did a dd clone to it while connected via USB, and I thought the clone had failed. But when I connected the SSD directly to my motherboard in place of my old HDD, everything worked fine! – Eliot Aug 29 '14 at 21:36
  • 1
    Can't edit my previous comment, but the Align tool is useless in this case, it's just for optimization purposes. However, note that you can use TestDisk and after a deeper scan, press P to list the files and recover the content of your disc (this is how I recovered my data, but I didn't find any way to fix the byte sector to this day...). – gaborous Feb 12 '15 at 16:36
  • 1
    An interesting read which confirms the problem and hint at the solution (emulating bridge's translation via Linux loopback device): goughlui.com/2013/10/02/… and this askubuntu.com/questions/337693/… . And just as an additional note, I also tried to forcefully edit the logical size to match the physical size, but the drive still wasn't recognized. But formatting it fix the mounting, but the files are of course lost, so better recover them before via loopback mounting or testdisk. – gaborous Feb 13 '15 at 22:43
4

This script generalized the Rod Smith proposal, when you have a raid or a crypto. No warranty. Feel free to improve it! (Updated with latest finding about mdadm)

#!/bin/sh
#
# This script solve the following problem:
#
# 1. create a GPT partition on a large disk while attached directly via SATA
#    when the device present itself with 512 bytes of block size:
#    sd 3:0:0:0: [sda] 5860533168 512-byte logical blocks: (3.00 TB/2.72 TiB)
#
# 2. try to use a SATA to USB adapter like ID 067b:2773 Prolific Technology, Inc.
#    this present the device with 4096 bytes of block size:
#    sd 19:0:0:0: [sdc] 732566646 4096-byte logical blocks: (3.00 TB/2.72 TiB)
#
# 3. The kernel is unable to read correctly the partition table with
#    the USB adaper.
#
#
# With the current tools (kernel and gdisk) in debian wheezy is
# possible to use losetup to remap the partitions to loop devices so
# you can use them as usual with any filesystem, raid or crypto
#
# I still do not know if this issue is originated by the adapter or by
# the disk and if there are any others workarounds.
#
# Known version of the software:
# $ apt-show-versions linux-image-3.2.0-4-amd64
# linux-image-3.2.0-4-amd64/wheezy uptodate 3.2.54-2
# $ apt-show-versions gdisk
# gdisk/wheezy uptodate 0.8.5-1


attach_device() {

    device="$1";

    MYTMPDIR=`mktemp -d`
    trap "rm -rf $MYTMPDIR" EXIT

    # gdisk on the device use the 4096 sector size
    # but we need to force it to 512
    # this is a knwon workaround from http://superuser.com/a/679800
    # basically we make a copy of the gpt partition table on a file
    dd if="/dev/$device" bs=16384 count=1 of="$MYTMPDIR/gpt" 2> /dev/null

    # we extract the offset and the size of each partition
    #
    # FIXME: the "+ 1" seems strange, but it is needed to get the same
    #        size value from:
    #
    #        blockdev --getsize64
    #
    #        without the "+ 1" some funny things happens, for example
    #        you will not be able to start a recognized md device:
    #
    #        md: loop1 does not have a valid v1.2 superblock, not importing!
    #        md: md_import_device returned -22
    #
    #        even if
    #
    #        mdadm --examine /dev/loop1
    #
    #        does not complaint

    gdisk -l \
     "$MYTMPDIR/gpt" 2> /dev/null | \
     awk '/^ *[0-9]/ {printf "%.0f %.0f\n", $2 * 512, ($3 - $2 + 1) * 512}' > $MYTMPDIR/offset-size

    # we create a loop device with the give offset and size
    while read line;
    do
        offset=$(printf "$line" | cut -d ' ' -f 1);
        size=$(printf "$line" | cut -d ' ' -f 2);
        losetup --verbose --offset "$offset" --sizelimit "$size" `losetup -f` /dev/$device;
    done < $MYTMPDIR/offset-size;
}

detach_device() {

    device="$1";

    for loopdevice in `losetup -a | grep "$device" | cut -d : -f 1`;
    do
        losetup --verbose --detach "$loopdevice";
    done;
}

usage() {
cat <<EOF
Usage:
- $0 -h to print this help
- $0 sda to attach the gpt partitions of sda
- $0 -d sda to detach the gpt partitions of sda
EOF
}


detach=0;

while getopts hd action
do
    case "$action" in
        d) detach=1;;
        h) usage;;
    esac
done
shift $(($OPTIND-1))

if [ $# -ne 1 ];
then
    usage;
fi

if [ "x$detach" = "x0" ]; then
    attach_device $1;
else
    detach_device $1;
fi
3

Another, fairly straight forward way to do this is using parted's rescue function. This requires you to create a new disk label though, so it involves risks. Parted acts directly on the disk so make backups as necessary before running parted. Then start:

parted /dev/sdb

parted will tell you something along these lines when trying to read a disk with different sector size than with which the partition table was created:

Error: /dev/sdb: unrecognised disk label                                  

Use mklabel to create a new MBR or GPT according to what you previously used

(parted) mklabel
New disk label type? mbr

Then run rescue to find your old partition

(parted) rescue
Start? 0
End? 4001GB
Information: A ext4 primary partition was found at 1049kB -> 2000GB.  Do you
want to add it to the partition table?
Yes/No/Cancel? y

Repeat the rescue process if you have more partitions. You are now done.

  • 1
    This worked perfectly for me to convert my partition table from mbr to gpt. Doing this so that I could expand a cloned 2TB disk to 4TB. A little nervous leaving my partition hanging there but this is so much faster than other methods. – OregonTrail Jun 27 '17 at 18:15
3

I had this problem when I removed a 4TB disk from a WD My Book external enclosure. The problem is:

  1. the MBR partition table is off by a factor of 8 and
  2. the MBR partition table cannot handle >2TB when the sector size is 512.

Solution: Rewrite the partition table into a GPT, converting the values to use 512 byte sectors.

In my case the partition started on a 1MB offset and ended (~856kB) before the end of the disk. This is good because then it allowed for the MBR+GPT (17408 bytes) before the partition and the backup GPT (16896 bytes) at the end of the disk.

I made images of both regions just in case (using dd).

I noted the output from fdisk -l /dev/sde.

I used gdisk to delete the first partition. If you want, you can do as I did and change the align value to 8 (4096) to use as much space as possible. Then, I created a new partition with the start at 2048, and the end at the end of the disk. I'll grow the file system later.

Thankfully, the change in sector size doesn't affect the file system, LVM, or LUKS.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.