0

This morning a user came up to me to let me know that email being sent from the outside world was not showing up in his inbox for his work email. We host the email server internally (not my idea). I tested by sending an email from my gmail account to my work email, and as expected the email never arrives.

I go to reboot the mail server (which is running Ubuntu 10.10) and right before the point where the GRUB boot loader is started, the server would reboot back to a POST/BIOS "Checking memory" message as if I had hit the reset button.

I went and burned a copy of 10.10 and tried to run the system rescue to see if I was able to fix the boot issue, but I was not able to fix it. I then burned a copy of Ubuntu 18.04 to see if there were any additional tools that might allow me to dig a little deeper to find out what was going on.

I checked /etc/fstab on the drive where Ubuntu is currently installed, and here is what I found

# /etc/fstab: static file system information.
#
# Use 'blkid -o value -s UUID' to print the universally unique identifier
# for a device; this may be used with UUID= as a more robust way to name
# devices that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
proc            /proc           proc    nodev,noexec,nosuid 0       0
/dev/mapper/mailadmin-root /               ext4    errors=remount-ro 0       1
# /boot was on /dev/sda2 during installation
UUID=e8c63c19-db49-4013-bec8-ef8479fb52f1 /boot           ext2    defaults        0       2
/dev/mapper/mailadmin-swap_1 none            swap    sw              0       0
10.6.9.5:/volume1/Backups/Zimbra/TarBackups /zmbackup nfs,size=8192,wsize=8192,timeo=14,intr

Notice the comment above the UUID for the /boot partition? I did not enter that comment, it was left by the previous IT manager a number of years ago. I modified the /etc/fstab file to point at /dev/sda2 (which I have confirmed is the actual location where the grub config is located) instead of the UUID, rebooted - but still the server goes into a reboot loop right as it is about to load grub.

After doing some googling, I found a script called bootinfoscript.sh that outputs a text file with a bunch of information related to the hard disks that are detected in my system. Here is the output of bootinfoscript.sh

                  Boot Info Script 0.61      [1 April 2012]


============================= Boot Info Summary: ===============================

 => Grub2 (v1.97-1.98) is installed in the MBR of /dev/sda and looks at sector 
    2048 of the same hard drive for core.img. core.img is at this location and 
    looks in partition 2 for /grub.
 => Syslinux MBR (4.04 and higher) is installed in the MBR of /dev/sdb.

sda1: __________________________________________________________________________

    File system:       BIOS Boot partition
    Boot sector type:  Grub2's core.img
    Boot sector info: 

sda2: __________________________________________________________________________

    File system:       ext2
    Boot sector type:  -
    Boot sector info: 
    Operating System:  
    Boot files:        /grub/grub.cfg /grub/core.img

sda3: __________________________________________________________________________

    File system:       LVM2_member
    Boot sector type:  -
    Boot sector info: 

sdb1: __________________________________________________________________________

    File system:       vfat
    Boot sector type:  SYSLINUX 6.03
    Boot sector info:  Syslinux looks at sector 32784 of /dev/sdb1 for its 
                       second stage. The integrity check of Syslinux failed. 
                       No errors found in the Boot Parameter Block.
    Operating System:  
    Boot files:        /boot/grub/grub.cfg /syslinux.cfg 
                       /efi/BOOT/grubx64.efi /ldlinux.sys

mailadmin-root': _______________________________________________________________

    File system:       
    Boot sector type:  Unknown
    Boot sector info: 
    Mounting failed:   mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-root': unknown filesystem type ''.

mailadmin-swap_1': _____________________________________________________________

    File system:       
    Boot sector type:  Unknown
    Boot sector info: 
    Mounting failed:   mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-root': unknown filesystem type ''.
mount: /tmp/BootInfo-TMXYqs5T/LVM/mailadmin-swap_1': unknown filesystem type ''.

============================ Drive/Partition Info: =============================

Drive: sda _____________________________________________________________________
Disk /dev/sda: 10.9 TiB, 11999064883200 bytes, 23435673600 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt

Partition  Boot  Start Sector    End Sector  # of Sectors  Id System

/dev/sda1                   1 4,294,967,295 4,294,967,295  ee GPT


GUID Partition Table detected.

Partition    Start Sector    End Sector  # of Sectors System
/dev/sda1           2,048         4,095         2,048 BIOS Boot partition
/dev/sda2           4,096       503,807       499,712 Data partition (Windows/Linux)
/dev/sda3         503,808 7,329,544,191 7,329,040,384 Logical Volume Manager (LVM) partition (Linux)

Drive: sdb _____________________________________________________________________
Disk /dev/sdb: 14.4 GiB, 15500574720 bytes, 30274560 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos

Partition  Boot  Start Sector    End Sector  # of Sectors  Id System

/dev/sdb1    *          2,048    30,274,559    30,272,512   c W95 FAT32 (LBA)


"blkid" output: ________________________________________________________________

Device           UUID                                   TYPE       LABEL

/dev/loop0                                              squashfs   
/dev/loop1                                              squashfs   
/dev/loop2                                              squashfs   
/dev/loop3                                              squashfs   
/dev/loop4                                              squashfs   
/dev/loop5                                              squashfs   
/dev/loop6                                              squashfs   
/dev/loop7                                              squashfs   
/dev/mapper/mailadmin-root 3e2b9e3a-7190-4240-804e-f1761fa96ade   ext4       
/dev/mapper/mailadmin-swap_1 2b7f558a-2abe-4981-879e-d79f7ed3fb01   swap       
/dev/sda1                                                          
/dev/sda2        e8c63c19-db49-4013-bec8-ef8479fb52f1   ext2       
/dev/sda3        WFe0Xa-LXJa-BzOr-Oyo9-kC5E-3W7p-qWA5Uy LVM2_member 
/dev/sdb1        6045-3252                              vfat       UBUNTU 18_0

========================= "ls -R /dev/mapper/" output: =========================

/dev/mapper:
control
mailadmin-root
mailadmin-swap_1

================================ Mount points: =================================

Device           Mount_Point              Type       Options

/dev/loop0       /rofs                    squashfs   (ro,noatime)
/dev/sdb1        /cdrom                   vfat       (ro,noatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)


============================= sda2/grub/grub.cfg: ==============================

--------------------------------------------------------------------------------
#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by /usr/sbin/grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
if [ -s $prefix/grubenv ]; then
  load_env
fi
set default="0"
if [ ${prev_saved_entry} ]; then
  set saved_entry=${prev_saved_entry}
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z ${boot_once} ]; then
    saved_entry=${chosen}
    save_env saved_entry
  fi
}

function recordfail {
  set recordfail=1
  if [ -n ${have_grubenv} ]; then if [ -z ${boot_once} ]; then save_env recordfail; fi; fi
}
insmod ext2
set root='(hd0,2)'
search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1
set locale_dir=($root)/grub/locale
set lang=en
insmod gettext
if [ ${recordfail} = 1 ]; then
  set timeout=-1
else
  set timeout=10
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/05_debian_theme ###
set menu_color_normal=white/black
set menu_color_highlight=black/light-gray
### END /etc/grub.d/05_debian_theme ###

### BEGIN /etc/grub.d/10_linux ###
### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_memtest86+ ###
menuentry "Memory test (memtest86+)" {
    insmod ext2
    set root='(hd0,2)'
    search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1
    linux16 /memtest86+.bin
}
menuentry "Memory test (memtest86+, serial console 115200)" {
    insmod ext2
    set root='(hd0,2)'
    search --no-floppy --fs-uuid --set e8c63c19-db49-4013-bec8-ef8479fb52f1
    linux16 /memtest86+.bin console=ttyS0,115200n8
}
### END /etc/grub.d/20_memtest86+ ###

### BEGIN /etc/grub.d/30_os-prober ###
if [ ${timeout} != -1 ]; then
  if keystatus; then
    if keystatus --shift; then
      set timeout=-1
    else
      set timeout=0
    fi
  else
    if sleep --interruptible 3 ; then
      set timeout=0
    fi
  fi
fi
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###
--------------------------------------------------------------------------------

=================== sda2: Location of files loaded by Grub: ====================

           GiB - GB             File                                 Fragment(s)


=========================== sdb1/boot/grub/grub.cfg: ===========================

--------------------------------------------------------------------------------

if loadfont /boot/grub/font.pf2 ; then
    set gfxmode=auto
    insmod efi_gop
    insmod efi_uga
    insmod gfxterm
    terminal_output gfxterm
fi

set menu_color_normal=white/black
set menu_color_highlight=black/light-gray

set timeout=5
menuentry "Try Ubuntu without installing" {
    set gfxpayload=keep
    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper quiet splash ---
    initrd  /casper/initrd.lz
}
menuentry "Install Ubuntu" {
    set gfxpayload=keep
    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash ---
    initrd  /casper/initrd.lz
}
menuentry "OEM install (for manufacturers)" {
    set gfxpayload=keep
    linux   /casper/vmlinuz  file=/cdrom/preseed/ubuntu.seed boot=casper only-ubiquity quiet splash oem-config/enable=true ---
    initrd  /casper/initrd.lz
}
menuentry "Check disc for defects" {
    set gfxpayload=keep
    linux   /casper/vmlinuz  boot=casper integrity-check quiet splash ---
    initrd  /casper/initrd.lz
}
--------------------------------------------------------------------------------

============================== sdb1/syslinux.cfg: ==============================

--------------------------------------------------------------------------------
DEFAULT loadconfig

LABEL loadconfig
  CONFIG /isolinux/isolinux.cfg
  APPEND /isolinux/
--------------------------------------------------------------------------------

=================== sdb1: Location of files loaded by Grub: ====================

           GiB - GB             File                                 Fragment(s)


================= sdb1: Location of files loaded by Syslinux: ==================

           GiB - GB             File                                 Fragment(s)


======================== Unknown MBRs/Boot Sectors/etc: ========================

Unknown BootLoader on mailadmin-root'


Unknown BootLoader on mailadmin-swap_1'



=============================== StdErr Messages: ===============================

cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory
cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory
cat: /tmp/BootInfo-TMXYqs5T/Tmp_Log: No such file or directory
  Failed to find logical volume "mailadmin/root'"
  Failed to find logical volume "mailadmin/root'"
  Failed to find logical volume "mailadmin/root'"
hexdump: /dev/mapper/mailadmin-root': No such file or directory
hexdump: /dev/mapper/mailadmin-root': No such file or directory
  Failed to find logical volume "mailadmin/swap_1'"
  Failed to find logical volume "mailadmin/swap_1'"
  Failed to find logical volume "mailadmin/swap_1'"
hexdump: /dev/mapper/mailadmin-swap_1': No such file or directory
hexdump: /dev/mapper/mailadmin-swap_1': No such file or directory

Just for some background info, /dev/sda is a 12TB partition where the mail server software is installed. /dev/sdb is a 16GB USB thumbdrive that I installed Ubuntu onto.

I mounted the 12TB (which is actually a RAID array) /dev/sda1 drive and when I browse to the /boot folder on the mounted drive, it appears as if the /boot directory is completely empty!

enter image description here

Anyone have any idea what is going on here? Is there some way to fix the grub boot loader on this old server?

Please let me know if you need any additional details.

  • And what do you see if you mount /dev/sda2? The fact that boot is empty in the screenshot is no surprise, it would only be populated if you manually mounted it, too, at this stage. – tink Dec 29 '18 at 0:28
  • 1
    You desperately need to update your server to a supported version of Ubuntu. Ubuntu 10.10 reached end-of-life in April 2012, and has not received any updates (including security updates!!) since then. – duskwuff -inactive- Feb 8 '19 at 2:44
  • @duskwuff tell me about it! I took over as the IT manager at this company 2 years ago and I’ve replaced everything except for this last server. It had 10 years worth of archived email for 75 users, and was expected to be up 100% of the time by upper management. We are in the process of migrating everything over to G Suite as I write this comment :) – Richie086 Feb 8 '19 at 2:49
1

We were able to resolve this issue by installing the same version of Ubuntu on another server and copying over the contents of /boot from the newly installed server and modifying the /etc/fstab to point to the correct partitions in /dev.

| improve this answer | |
  • 1
    Thanks for coming back to close the loop on this. Glad you got it sorted and shared this with us. – I say Reinstate Monica Feb 8 '19 at 2:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.