zip being too nice (Mac OS X)

Question

I use zip to do a regular backup of a local directory onto a remote machine. They don't believe in things like rsync here, so it's the best I can do (?). Here's the script I use

echo $(date)>>~/backuplog.txt;
if [[ -e /Volumes/backup/ ]];
then 
    cd /Volumes/Non-RAID_Storage/;
    for file in projects/*; 
        do nice -n 10 zip -vru9 /Volumes/backup/nonRaidStorage.backup.zip "$file" 2>&1 | grep -v "zip info: local extra (21 bytes)">>~/backuplog.txt;
    done;
else 
    echo "backup volume not mounted">>~/backuplog.txt;
fi

This all works fine, except that zip never uses much CPU, so it seems to be taking longer than it should. It never seems to get above 5%. I tried making it nice -20 but that didn't make any difference. Is it just the network or disc speeds bottlenecking the process or am I doing something wrong?

Answer 1

You will probably find that zip is spending most of its time waiting for I/O (reading files and writing the compressed versions) which is why it is not using as much CPU as you expect. Giving the process extra priority via nice has no effect on this as the task can't use any more CPU time if it isn't being fed data at a rate that would require it to.

Under Linux you can see this situation as a high %age as "IO Wait" time in the output from top and similar utilities, the same may be true for OSX.

Reasons for the IO wait time could include:

1. processing many small files (lots of head movement reading the files and related directory structures)
2. file fragmentation
3. competing with other activity on the relevant drives (users copying/moving/accessing files, scheduled AV scans, ...) while the backup takes place
4. reading the files over a network (a modern CPU can zip data far faster than a 100Mbit/s link can ever feed it, and network latency will exacerbate the effect of many-small-files) or pushing the compressed data over the network (unless your data is unusually compressible, the same "zip is faster then your network on modern CPUs" condition applies as it'll read from your local drives and process the daat faster than it can then send the result over the network)
5. network contention (if the server your are talking to has a 100Mbit link and others are using it this may be an issue, less so if it has a faster link of course)
6. slow drives or slow interfaces (if any of the drives concerned are USB2 connected they'll tend to transfer no more than 25Mbyte/sec, sometimes slower depending on the USB adaptor used and USB bus contention with other fast devices, where a modern internal drive will push double that if not more for bulk transfers)

If you want to make use of the "spare" CPU cycles, and can't do so by reducing the IO delays, you could try using 7zip instead - this uses much more CPU time per block of data but achieves better compression than zip by quite some margin too in many cases, reducing the size of your backups. Whether this will be faster (because 7zip results in sending less data over the network) or slower (because the extra computational complexity means you CPU may become the bottleneck not the disks/filesystems/network) depends on your machines exact spec.

Edit:

One other thing, some tools report process use per-core and some per-CPU (and some either way depending on settings), and zip is usually an threaded process. So if you have a quad-core CPU that 5% is not unlikely to be "5% of the CPU", or approx 20% of one core (though it may be bouncing between cores, if it is single threaded it won't be running on more than one at any given instant).

Answer 2

"nice -n 10" makes the program in question be even "nicer" by using a lower priority. Perhaps you meant "nice -n -10" or "nice --10" which makes the program less "nice" and thus use more CPU.

Answer 3

here's my script as it stands, if anyone is interested. Obviously there are a few hard-coded paths in here, so you can't just run it as it stands. It keeps a log of its operations and uses growl to notify you of any errors. If you don't have growl / don't want to install it, just comment out / remove any of the lines with growlnotify in them.

One of the other changes i made was to copy the remote zip archive to a local drive before adding the files. What was happening was that zip was copying the archive into the local temp file, making the changes and then moving the temp file back to the remote drive, for each file/folder in the top level directory. So now it only does the heavy network lifting once. I'm kinda thinking I should be looking at tar for this kind of thing..

localBaseDirectory=/Volumes/Non-RAID_Storage/;
backedUpDirectory=projects;
backupVolume=/Volumes/video-prod/Backup;
backupFile=$backupVolume/Non-RAID_Storage_backup.zip;
zipTempfile=/Volumes/B_media
localBackupTempFile=/Volumes/A_Media/backuptemp.zip;
log=~/backuplog.txt;
temp=/var/tmp/backupError.txt;
##############################################
/usr/local/bin/growlnotify -m "backing up $backedUpDirectory on $localBaseDirectory" 2>&1 >/Dev/Null  #need this redirect because growl chucks errors when run by cron;

echo $(date) > $log
if [[ -e $backupVolume ]];
then 
   if [[ -e $backupFile ]];
        then cp $backupFile $localBackupTempFile;
        mv $backupFile "$backupFile-old";
        echo "copied old backup to $backupFile-old">>$log;
    fi;
    cd $localBaseDirectory 2>$temp;
    if [[ -s $temp ]];
        #notify if there was an error cding to this directory. -s is true if the file exists and is not zero sized
        then cat $temp | /usr/local/bin/growlnotify -s;
        cat $temp >> $log;
    fi;
    for file in $backedUpDirectory/*;
    do
        /usr/local/bin/growlnotify -m "backing up $file" 2>&1 >/Dev/Null;
        #zip using verbose, recursive, update (ie don't overwrite files unless they're older), highest level compression
        #zip creates a lot of garbage errors when being verbose eo send errors to a temp file, and stdout to the log file
         nice -n 10 zip -vru9 -b $zipTempfile $localBackupTempFile "$file" 2>$temp |grep "adding:">>$log;
         #add just the important errors to the log
         cat $temp 2>/dev/null|grep "error:">>$log;
    done;
    echo "done adding to local zip file - moving to $backupFile">>$log;
    #move the local version to the remote backup file
    mv $localBackupTempFile $backupFile 2>>$log;
    echo "$(date) - backed up Non-RAID_storage">>$log
    /usr/local/bin/growlnotify -m "backed up $backedUpDirectory on $localBaseDirectory" 2>&1 >/Dev/Null;
else 
    /usr/local/bin/growlnotify -s -m "Backup volume not mounted" 2>&1 >/Dev/Null;
    echo "backup volume not mounted" >> $log;
fi;