4

Please post your thoughts ideas or anything you may come up with. I would be very interested to see what anybody else is thinking.

The Overall Problem

When I install a simple Java application (that I wrote) to run on boot (in the background) through /etc/init.d/, it works on the liveUSB that I explicitly install it on. When I make a copy of that stick, it never boots successfully. When booting the liveUSB copy the Java application will always hang when the liveUSB boot process reaches my script. My script, which does do exactly what it is supposed to do, even every 5 minutes and will continue running forever until you power down the machine.

  1. My script is blocking everything else
  2. Nothing loads beyond my script
  3. You cannot cancel my script
  4. There is no GUI
  5. The only text you can see is the command line output from my script

Setup & Test - Everything goes well :)

I have a Linux liveUSB with 3 partitions. Simple standard Xubuntu image is loaded.

  • sda1 > 2gb stoage
  • sda2 > 2gb system
  • sda3 > remaining gb for casper

I have created a simple Java application that runs in the background on start up. To get this far, I followed these steps:

  1. Compiled java application into classes
  2. Placed class files in /home/user/folder/
  3. Copied my startup.sh script into /etc/init.d/
  4. While inside /etc/init.d/
    • Typed "update-rc.d startup.sh start 20 2 5 . stop 20 0 1 6 ."
    • This updated run levels successfully
  5. Now I can restart / reboot / shutdown whatever operation and everything runs perfectly!

The Copy - Here's where it gets tricky!

When creating a copy of this stick, I follow these steps:

  1. Mount sda2
    • copy everything from that folder to /home/user/Desktop/tmp-system/
  2. Mount sda3
    • copy everything from that folder to /home/user/Desktop/tmp-casper/
  3. Go into /home/user/Desktop/tmp-system/
    • Type "tar -cvf system.tar ."
  4. Go into /home/user/Desktop/tmp-casper/
    • Type "tar -cvf casper.tar ."
  5. Umount
    • sda2
    • sda3
  6. Plug in empty USB (sdb for example)
    • Set up partitions (Same as the stick you are copying from)
    • Untar into partitions
      • tar -xvf system.tar ... into sdb2
      • tar -xvf casper.tar ... into sdb3

Testing - Here's where everything goes wrong!

  1. Plug in newly created liveUSB into a computer
  2. Boot from USB
  3. Everything starts to boot fine
  4. Java application that I wrote gets triggered
    • Boot process hangs forever
    • No cmd prompt available
    • NO GUI available
    • It is as if the thread is running (and it is! The output can be viewed every 5 minutes - which is exactly the way it should be)

Solution Attempts & Gotchas

1

I can mount the copied liveUSB, edit the startup.sh to not start my Java application and it will still not boot (just as I suspected?).

2

If I use "dd if=sda of=sdb" the copy of the liveUSB will work perfectly fine. However this is not an acceptable solution. If I were to copy a 16gb stick with dd to a 64gb stick, that would turn the 64gb stick into a 16gb. It would also make it much more difficult to change the values that need to be changed in each partition.

3

Tested many variations of startup.sh and the Java application itself. All of which produce the same error.

4

The method I am using to copy works for every other form of application, file or anything else.

5

I would also like to try and avoid using any additional libraries or programs to help run the Java application.

6

I have also mounted sda2 & sdb2 used cp to copy everything directly from one to the other, then followed the same for sda3 & sdb3. This produces the same error.

ADDITIONAL POINTS

  1. The sda3 partition is encrypted with cryptsetup
  2. There are 2 files in the system.tar (which will be sdb2, came from sda2) that will have a value changed after it has been written to the USB.
    • These two values have not caused any problems in the past and have always been changed every time a new liveUSB is created
  3. There is 1 file in the casper.tar (which will be sdb3, came from sda3) that will have a value changed after it has been written to the USB.
    • This value has not caused any problems in the past and has always been changed with every new liveUSB created.

Checksum Testing Process

Orignial liveUSB image

Working liveUSB > sda empty usb > sdb

Steps:

  1. Mount, copy & checksum sda2
    • Typed "mount /dev/sda2 /media/sda2"
    • Typed "tar -C /media/sda2 -cvf ~/Desktop/system.tar ."
    • Typed "find . -type f -exec sha1sum {} ';' > /tmp/sda2_checksum.txt"
    • Typed "umount /media/sda2"
  2. Mount, copy & checksum sda3
    • Typed "mount /dev/sda3 /media/sda3"
    • Typed "tar -C /media/sda3 -cvf ~/Desktop/casper.tar ."
    • Typed "find . -type f -exec sha1sum {} ';' > /tmp/sda3_checksum.txt"
    • Typed "umount /media/sda3"
  3. Create partitions for sdb
  4. Mount, write & checksum sdb2
    • Typed "mount /dev/sdb2 /media/sdb2"
    • Typed "tar -C /media/sdb2 -xvf ~/Desktop/system.tar"
    • Typed "find . -type f -exec sha1sum {} ';' > /tmp/sdb2_checksum.txt"
    • Typed "umount /media/sdb2"
  5. Mount, write & checksum sdb3
    • Typed "mount /dev/sdb3 /media/sdb3"
    • Typed "tar -C /media/sdb3 -xvf ~/Desktop/casper.tar"
    • Typed "find . -type f -exec sha1sum {} ';' > /tmp/sdb3_checksum.txt"
    • Typed "umount /media/sdb3"
  6. Compare checksums
    • sort /tmp/sda2_checksum.txt -o /tmp/sda2_checksum.txt.sort
    • sort /tmp/sda3_checksum.txt -o /tmp/sda3_checksum.txt.sort
    • sort /tmp/sdb2_checksum.txt -o /tmp/sdb2_checksum.txt.sort
    • sort /tmp/sdb3_checksum.txt -o /tmp/sdb3_checksum.txt.sort
  7. Results

sda2 & sdb2

Typed "diff sda2_checksum.txt.sort sdb2_checksum.txt.sort"

45d44
< 2ddf9802c9c15ac6e4575cc9de32e3530eae6b7d  ./efi/boot/grub.cfg
82d80
< 59bb2775a8e7e499e0590b7b8c2492eb250fb7d8  ./syslinux/txt.cfg
154a153
> ae6c127713e01fc5fb4a2e4e28f6bbddc6bd6af5  ./efi/boot/grub.cfg
158a158
> b78090b66b4e3fa04ca9d466ee78c9060adf744e  ./syslinux/txt.cfg

These 2 files contain 1 value in each that gets changed. Everything else is the same. The results are exactly what they should be.

sda3 & sdb3

Typed "diff sda3_checksum.txt.sort sdb3_checksum.txt.sort"

Identical - Keep in mind this is the original liveUSB image.

I will post further comparison results as I work through the next section.

NEXT STEPS - aka Action Plan

Starting from liveUSB image without the scripts that need to be run.

Step 1 - Success / Fail ?

SUCCESS - checksums match as they should

  1. Update java on liveUSB from 6 to 7
  2. Recreate tar's
  3. Create new liveUSB from tar's
  4. Test liveUSB

Step 2 - Success / Fail ?

SUCCESS - checksums match as they should

  1. Create /home/user/folder/
  2. Copy class files for java application into /home/user/folder/
  3. Recreate tar's
  4. Create new liveUSB from tar's
  5. Test liveUSB

Step 3 - Success / Fail ?

SUCCESS - checksums match as they should

  1. Add startup.sh into /etc/init.d/
  2. Without calling update-rc.d
  3. Recreate tar's
  4. Create new liveUSB from tar's
  5. Test liveUSB

Step 4 - Success / Fail ?

SUCCESS - checksums match as they should

(I have never made it this far successfully before) - however the value that needs to written has not been inserted into the casper (sda3) partition yet.

  1. Type "update-rc.d startup.sh start 21 2 5 ."
  2. Recreate tar's
  3. Create new liveUSB from tar's
  4. Test liveUSB

Step 5 - Success / Fail ?

SUCCESS - checksums match as they should

I can't believe this worked! Which brings me to... (To word this in a nice way) Why the world was it not working before?

-coincidently it was Version-13 that worked.

  1. Boot liveUSB
  2. Insert value to be over-written in casper (sda3) before creating tar
    • While running from liveUSB
    • Edit config file in /home/user/folder/config.properties
  3. Shutdown liveUSB
  4. Recreate tar's
  5. Create new liveUSB from tar's
  6. Test liveUSB

IMAGE COMPLETE!!

I'm not finished with this just yet!

*The process of writing to the usb has never changed.

Why did it not work before?

  1. Tar method? - Only a slight change...
    • From "tar -cvf casper.tar ."
    • TO "tar -C /media/sda3/ -cvf ~/Desktop/casper.tar .
    • Are these lines not accomplishing the same thing?
    • I will be testing this out shortly down the road. - I suspect no difference.
  2. Cut up the procedure into individual steps?
    • Before:
      • Under the NEXT STEPS - aka Action Plan I would complete all of these steps before making a new image.
    • After:
      • The NEXT STEPS - aka Action Plan was followed exactly
    • Could this be the difference?
    • I will be testing this out shortly down the road.
  3. Can deleting large (or small) files from the /home/ directory in the casper (sda3) partition cause some sort of corruption?
    • I have no idea..?
    • I will be testing this out shortly down the road.

Further Testing - I want my answer!

Starting from the original liveUSB image.

  1. Update java on liveUSB from 6 to 7
  2. Create /home/user/folder/
  3. Copy class files for java application into /home/user/folder/
  4. Add startup.sh into /etc/init.d/
  5. Type "update-rc.d startup.sh start 21 2 5 ."
  6. Edit config file in /home/user/folder/config.properties

All at once step this time. - Will it work?

Test 1 - Success / Fail ?

FAIL!

  1. Old tar method

Test 2 - Success / Fail ?

  1. Old tar method
  2. Deleted generated file in /boot/
    • This file is created by my script when the casper (sda3) partition gets written, contains only an id for verification has no effect on the boot process.

Test 3 - Success / Fail ?

  1. New tar method

Test 4 - Success / Fail ?

  1. New tar method
  2. Deleted generated file in /boot/
    • This file is created by my script when the casper (sda3) partition gets written, contains only an id for verification has no effect on the boot process.

Results

I will be testing in this order:

Test 1 -> Test 3 -> Test 4 -> Test 2

If Test 1 works... I will go jump out of the window! - I will have no idea why it would be working now as I have tested this many times and every time has yielded unsuccessful boots. - Perhaps the cp or tar process was corrupted somehow.

When test 1 fails: If Test 3 works... - The tar method was causing the error. - I don't understand what would be wrong with the old tar method vs the new tar method.

TBC...

Improve this question

2 Answers 2

Reset to default
2

Given your description of the problem I suspect something other than you describe is happening. In any case, try this:

# sda2
mount /dev/sda2 /mnt/tmp
tar -C /mnt/tmp -cf ~/Desktop/sda2.tar .
umount /dev/sda2 
# ... repeat for sda3 ...
# do your create partition shenanigans
mount /dev/newsda2 /mnt/tmp
tar -C /mnt/tmp -xpf ~/Desktop/sda2.tar
umount /dev/newsda2
# repeat ...
# test ..

If that works then chances are your /home is mounted noexec or is some fubar'ed filesystem because the problem was the execute bit being removed.

If it doesn't work, edit your start up script to give you debug info such as the output of mount, content of syslog etc and look for help there.

You could also generate checksums for every file and compare copy vs original with:

find . -type f -exec sha1sum {} ';' > /tmp/sda2_checksums.txt

for each mounted partition & diff the /tmp/*_checksums.txt files

Improve this answer
2
  • This has been very useful for testing. Thank you very much! I will keep updating with progress that I am making and the output from checksum comparisons. Thank you again.
    – Mr. H. Dumpty
    Sep 25, 2013 at 19:51
  • The new tar method does seem to do the trick. I still wonder why my old method would produce this error. I have looked into the /home folder and I cannot find any difference. I will have to keep looking and keep everyone posted.
    – Mr. H. Dumpty
    Oct 1, 2013 at 13:49
0

In response to Solution Attempts & Gotchas #2 regarding the use of the dd command:

Using the dd command to clone the original Live USB does not permanently turn the 64gb stick into a 16gb stick. It will do that only if you choose not to repartition the 64gb stick after using the dd command. You can resize the parition to recover all of the remaining free (unallocated) space left on the 64gb stick after using the dd commmand.

You can revise the partitions by using one of either GParted or Parted Magic partition editors to recover any amount of free space from 16GB to 64GB on the cloned USB flash drive after using the dd command.

The net effect is that the dd command does not permanently turn the 64gb stick into a 16gb stick by using either GParted or Parted Magic after launching the dd command to reclaim the remainder of the 64gb stick.

Use the Linux Disk Utility program for your Linux distribution to verify the partition space as free or allocated.

You can learn GParted from the tutorial "GParted partitioning software - Full tutorial" at: http://www.dedoimedo.com/computers/gparted.html

Information about Parted Magic can be found at: https://en.wikipedia.org/wiki/PartedMagic but in my experience I would recommend GParted as Parted Magic was a full distribution of tools unless you prefer the later approach from what you find out about it.

In short, you have already discovered the easiest solution to your problem, which is just a resized partition away if none of the tests you planned at the end of your posts work.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .