I'm trying to create a Makefile that would automate remastering Knoppix distribution.

I'm trying to copy filesystem from the read-only compressed loop device to normal filesystem to be able to modify it and then compress it to replace the old one. That's what Knoppix remastering is all about. I have to preserve all the details about the filesystem recreate hard link structure from one filesystem on the other filesystem. The ls -li that I've pasted is the same file in two different filesystem to show that source had hardlinks and the destination have lost them.

$ ls -li */bin/dnsdomainname
   10419 -rwxr-xr-x 4 root root 12992 2010-01-19 10:40 KNOPPIX_V6.2.1CD-2010-01-31-EN.cloop/bin/dnsdomainname
10641687 -rwxr-xr-x 1 root root 12992 2010-01-19 10:40 KNOPPIX_V6.2.1CD-2010-01-31-EN.tree/bin/dnsdomainname

This is obviously possible -- if the rsync is able to preserve hardlinks between different hosts it should be able to do it on between filestems on the same host.

For some strange reason this is not working using neither of three methods that I'm aware that should do the job and I'm puzzled what is wrong. I've spent almost a workday to create the script, test it and this seems to be a last piece that stops it from fully working.

The methods are:

sudo nice tar -C $(NAME).cloop -cf - . | sudo nice tar -C $(NAME).tree -xvpf -
sudo cp -av --preserve=all $(NAME).cloop $(NAME).tree
sudo rsync -x -a -H --progress --delete $(NAME).cloop/ $(NAME).tree

I have a test suite integrated (make changes) and a part of it is counting md5sums from both filesystems and the diff is empty - meaning that content of every file between filesystem is exactly the same. However, the diff between permissions and hard link numbers shows that every file that was refering to shared (hard-linked) inode is now a separate inode which is not what I wanted. All the three methods are doing the same thing wrong which is really suspicious and I'm running out of ideas.

For your review, here comes the full Makefile:

PWD=$(shell pwd)

.PHONY: all clean test test-orig

all: $(NAME)-cherry88.iso

 wget -c -O $(NAME).iso.download $(SITE)/$(NAME).iso && mv $(NAME).iso.download $(NAME).iso

 if [ ! -z "`mount | grep "$(PWD)/$(NAME).cloop"`" ]; then sudo umount $(NAME).cloop && rm -f $(NAME).cloop.mount || test 1=1; fi
 if [ ! -z "`mount | grep "$(PWD)/$(NAME)"`" ]; then sudo umount $(NAME) && rm -f $(NAME).mount || test 1=1; fi

$(NAME).mount: $(NAME).iso
 mkdir -p $(NAME)
 if [ -z "`mount | grep "$(PWD)/$(NAME)"`" ]; then sudo mount -o loop $(NAME).iso $(NAME); fi
 touch $(NAME).mount

$(NAME).cloop.iso: $(NAME).mount
 nice extract_compressed_fs $(NAME)/KNOPPIX/KNOPPIX - > $(NAME).cloop.iso
 touch $(NAME).cloop.iso

$(NAME).cloop.mount: $(NAME).cloop.iso
 mkdir -p $(NAME).cloop
 if [ -z "`mount | grep "$(PWD)/$(NAME).cloop"`" ]; then sudo mount -o loop $(NAME).cloop.iso $(NAME).cloop; fi
 touch $(NAME).cloop.mount

$(NAME).cdtree.touch: $(NAME).mount
 sudo mkdir -p $(NAME).cdtree
 sudo rm -Rf $(NAME).cdtree
 #sudo nice tar -C $(NAME) --exclude=KNOPPIX/KNOPPIX -cf - . | sudo nice tar -C $(NAME).cdtree -xvpf -
 sudo nice cp -av --preserve=all $(NAME) $(NAME).cdtree
 #sudo rsync -x -a -H --progress --delete $(NAME)/ $(NAME).cdtree
 touch $(NAME).cdtree.touch

$(NAME).tree.touch: $(NAME).cloop.mount
 sudo mkdir -p $(NAME).tree
 sudo rm -Rf $(NAME).tree
 #sudo nice tar -C $(NAME).cloop -cf - . | sudo nice tar -C $(NAME).tree -xvpf -
 sudo cp -av --preserve=all $(NAME).cloop $(NAME).tree
 #sudo rsync -x -a -H --progress --delete $(NAME).cloop/ $(NAME).tree
 touch $(NAME).tree.touch

Use an archive format like cpio or tar to preserve hard links. See cpio(1) and tar(1).

Also, cp only preserves hard links if you use -l or -a.

[EDIT] All tools on Unix detect hard links by looking at the inode number (first number in output of ls -li). If the number is different, then no tool can recreate the hard links.

To fix the issue, I suggest to create MD5 checksums, sort them and then filter for duplicates (uniq -d). You won't have to create checksums everywhere because most of the hard links will be in bin directories.

Then, you can keep the first file and hard link the rest.

  • I do that. You haven't read full text, did you? :) – dpc.pw Aug 27 '10 at 9:31
  • Sorry, missed the --preserve=all in cp. I'm worried that rsync doesn't work. That usually only happens if the list of files doesn't include all the hard-links (i.e. if only one of the four copies is actually copied). What happens if you copy those four files manually with cp -avl? Are the inode numbers (first number of ls -li) of the source identical? If so, then it must work because that's how cp determines the hard links. – Aaron Digulla Aug 27 '10 at 10:23
  • Any east way to find all the files sharing same inode? – dpc.pw Aug 27 '10 at 10:39
  • 1
    find . -inum N Anyway. It seems that even though ls -l reports the file on iso-9660 filesystem has hardlinks all the inode numbers are different. Here lays the root of the problem. :/ If found hardlink tool that is finding and hardlinking the stuff using md5sum, but there are still some things that are making recompressed image different from the original. sigh – dpc.pw Aug 27 '10 at 11:40

Use rsync -a -H.

  • -a preserves pretty much everything except hardlinks.
  • -H tells rsync to preserve hardlinks. I've used this numerous times.

Check rsync's man page for further info.

You could also use tar:

tar --preserve-permissions --preserve-order --check-links -jc /path/to/your/folder /path/to/your/backup_file.tar.bz2

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.