2

Issue: I have an AWS t2.micro (virtual machine) instance (name:webserver) set up with Ubuntu 14.04 LTS. The instance is unusable due to a configuration file that I edited improperly and saved. I need to get access to the file system of the virtual machine so I can fix the broken configuration file in the instance (name:webserver), and (hopefully) recover the instance (VM) so it is again usable.

Configuration: AWS t2.micro instance (name:webserver) set up with Ubuntu 14.04 LTS. This instance is stopped and the EBS volume (name:broken) containing the instance (name:webserver) has been mounted at /mnt on a second AWS t2.micro instance (name:recovery) running Ubuntu 14.04 LTS (the same OS as the stopped instance). The mount command used was:

sudo mount -t sysfs xvdf /mnt

I also tried with other FS types (ext2, ext4, auto), none worked.

The fsblk command produced this output:

ubuntu@ip-172-xx-xx-xxx:~$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
└─xvda1 202:1    0   8G  0 part /
xvdf    202:80   0  24G  0 disk
└─xvdf1 202:81   0  24G  0 part

Mounting xvdf1 with the command:

ubuntu@ip-172-xx-xx-xxx:~$ sudo mount -t sysfs xvdf1 /data

produces the same content in /data as in the /mnt directory , as described below.

Question: How do I access the Ubuntu file system in the mounted EBS volume (name:broken)? The directories and files I'm looking for are per this list, taken from the server (name:recovery).

total 84
drwxr-xr-x  22 root root  4096 Jul 27 05:35 .
drwxr-xr-x  22 root root  4096 Jul 27 05:35 ..
drwxr-xr-x   2 root root  4096 Mar 25 11:52 bin
drwxr-xr-x   3 root root  4096 Mar 25 11:52 boot
drwxr-xr-x  13 root root  4020 Jul 25 16:53 dev
drwxr-xr-x  89 root root  4096 Jul 25 21:37 etc
drwxr-xr-x   3 root root  4096 Jul 25 15:46 home
lrwxrwxrwx   1 root root    33 Mar 25 11:51 initrd.img -> boot/initrd.img-3.13.0-48-generic
drwxr-xr-x  21 root root  4096 Mar 25 11:51 lib
drwxr-xr-x   2 root root  4096 Mar 25 11:50 lib64
drwx------   2 root root 16384 Mar 25 11:53 lost+found
drwxr-xr-x   2 root root  4096 Mar 25 11:50 media
drwxr-xr-x   2 root root  4096 Mar 25 11:50 opt
dr-xr-xr-x 109 root root     0 Jul 25 15:46 proc
drwx------   3 root root  4096 Jul 25 15:46 root
drwxr-xr-x  18 root root   660 Jul 26 06:27 run
drwxr-xr-x   2 root root  4096 Mar 25 11:52 sbin
drwxr-xr-x   2 root root  4096 Mar 25 11:50 srv
drwxrwx---  13 root root     0 Jul 26 14:17 sys

The same directories and files should be somewhere in the mounted volume (name:broken), but I've not been able to find them.

Comment / status: When I use ls -al /mnt -- I get a very nice listing of the virtual machine's directories and files, per the following.

total 4
drwxrwx--- 13 root root    0 Jul 26 14:17 .
drwxr-xr-x 22 root root 4096 Jul 27 05:35 ..
drwxr-xr-x  2 root root    0 Jul 25 15:46 block
drwxr-xr-x 28 root root    0 Jul 25 15:46 bus
drwxr-xr-x 46 root root    0 Jul 25 15:46 class
drwxr-xr-x  4 root root    0 Jul 25 15:46 dev
drwxr-xr-x 16 root root    0 Jul 25 15:46 devices
drwxr-xr-x  4 root root    0 Jul 25 15:46 firmware
drwxr-xr-x  7 root root    0 Jul 25 15:46 fs
drwxr-xr-x  5 root root    0 Jul 25 21:38 hypervisor
drwxr-xr-x  7 root root    0 Jul 25 15:46 kernel
drwxr-xr-x 92 root root    0 Jul 25 15:46 module
drwxr-xr-x  2 root root    0 Jul 25 21:38 power

These directories and files are all about the the virtual machine that runs Ubuntu 14.04. So far as I can tell, there is nothing in this file structure related to the Ubuntu file system itself. I've done a full tree listing of /mnt, and searched it for the Ubuntu system directories and files -- without success.

I've reviewed the AWS documentation and the Ubuntu man pages for directories and files, and for access to them. I've also searched in SuperUser, AskUbuntu and StackOverflow (and other forums). So far I've not been able to suss out the answer, so am turning to the experts at SuperUser for assistance.

My suspicion is that the root of the Ubuntu file system is a link, somewhere in the /mnt structure; and that a bind command of some form will get me access to it. So far, though, I don't see where / how to do this.

TIA for help with improving the question or a real answer.

  • What command did you use to mount this at /mnt? What is the relevant line of output from df -h? Also show lsblk output, please. What you are trying to do should be very straightforward... – Michael - sqlbot Jul 28 '15 at 9:29
  • @Michael, thank you for the questions; I also thought this would be straightforward, but so far, I must be missing something... I've edited the post to include the information you mention. This is a real learning experience. Thanks. – dvhirst Jul 29 '15 at 4:52
  • @Michael, thanks for your question -- it helped me significantly in finding the answer. – dvhirst Jul 29 '15 at 7:31
  • This would have been much easier if you hadn't done the erroneous -t sysfs in the first place. Simply mount /dev/xvdf1 /mnt would have worked. – qasdfdsaq Jul 29 '15 at 10:05
  • too true -- that's why I called it a learning experience. I searched for a recipe, didn't find it, had to discover how to use mount and then was ok. – dvhirst Jul 29 '15 at 10:09
-1

OK, thanks to several folks on this and other forums, I found the answer. The key is to know the file system type of the partition of interest. The lsblk command shows where the (name:broken) volume is mounted. The mount command with no parameters provides the required information about file system type, which is ext4 in this case. Note that several other file system types also have data in them, but it is not the data needed to fix the problem.

ubuntu@ip-172-31-19-121:~$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0   8G  0 disk
└─xvda1 202:1    0   8G  0 part /
xvdf    202:80   0  24G  0 disk
└─xvdf1 202:81   0  24G  0 part
ubuntu@ip-172-31-19-121:~$ mount
/dev/xvda1 on / type ext4 (rw,discard)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
ubuntu@ip-172-31-19-121:~$

Now that this is known, make the mount point directory:

ubuntu@ip-172-31-19-121:~$ mkdir ./mnt

... mount the volume:

ubuntu@ip-172-31-19-121:~$ sudo mount -t ext4 /dev/xvdf1 ./mnt

... check to see that the right directories and files are available:

ubuntu@ip-172-31-19-121:~$ ls -al ./mnt
total 100
drwxr-xr-x 22 root   root    4096 Jul 25 14:32 .
drwxr-xr-x  6 ubuntu ubuntu  4096 Jul 29 07:20 ..
drwxr-xr-x  2 root   root    4096 Mar 25 11:52 bin
drwxr-xr-x  3 root   root    4096 Mar 25 11:52 boot
drwxr-xr-x  5 root   root    4096 Mar 25 11:53 dev
drwxr-xr-x 89 root   root    4096 Jul 29 06:46 etc
drwxr-xr-x  4 root   root    4096 Jul 23 02:21 home
lrwxrwxrwx  1 root   root      33 Mar 25 11:51 initrd.img -> boot/initrd.img-3.13.0-48-generic
drwxr-xr-x 21 root   root    4096 Mar 25 11:51 lib
drwxr-xr-x  2 root   root    4096 Mar 25 11:50 lib64
drwx------  2 root   root   16384 Mar 25 11:53 lost+found
drwxr-xr-x  2 root   root    4096 Mar 25 11:50 media
drwxr-xr-x  2 root   root    4096 Apr 10  2014 mnt
drwxr-xr-x  2 root   root    4096 Mar 25 11:50 opt
drwxr-xr-x  2 root   root    4096 Apr 10  2014 proc
drwx------  3 root   root    4096 Jul 22 04:40 root
drwxr-xr-x  3 root   root    4096 Mar 25 11:53 run
drwxr-xr-x  2 root   root    4096 Mar 25 11:52 sbin
drwxr-xr-x  2 root   root    4096 Mar 25 11:50 srv
drwxr-xr-x  2 root   root    4096 Mar 13  2014 sys
drwxrwxrwt  2 root   root    4096 Jul 29 06:52 tmp
drwxr-xr-x 10 root   root    4096 Mar 25 11:50 usr
drwxr-xr-x 12 root   root    4096 Mar 25 11:52 var
lrwxrwxrwx  1 root   root      30 Mar 25 11:51 vmlinuz -> boot/vmlinuz-3.13.0-48-generic

... they are, so now it is time to edit the broken file:

ubuntu@ip-172-31-19-121:~$ sudo nano ./mnt/etc/sudoers

Yes, found the bad line, fixed it, and wrote out the corrected file.

Next steps:

  1. unmount the recovered volume,
  2. stop the recovery instance,
  3. detach the recovered volume from the recovery instance,
  4. attach the recovered volume to the original instance (be sure to specify the block device name as /dev/sda1 -- use anything else then IT DOES NOT WORK),
  5. restart the original instance, and confirm that it functions properly.

I mis-keyed the required entry, /dev/sda1 (tried /dev/sda, /dev/xvda, etc. -- no go) and didn't find it for a while. So, while the volume did re-attach, it wasn't recognized as the boot volume. Once I entered /dev/sda1 as the block device name, it was accepted, recognized as the boot volume, and everything just worked. SO GLAD THIS IS DONE! It has been a real learning experience. I hope this helps someone else who runs into these types of issues.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.