Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am moving some 20Gb within my local network (100 Mbit/sec). The files are from a typical Linux desktop system.

Would compressing them using Tar\Gzip and then sending them improve performance?

EDIT: I'm moving a Developer's workspace, meaning lots of source and PDFs, and not much multimedia.

share|improve this question
A worthwhile upgrade may be to upgrade your LAN to gigabit Ethernet. I upgraded my switch and some NICs a few years ago and it was definitely worth it. Now most motherboards even come with on-board 10/100/1000 ports, so you might only need to upgrade your switch. – Chris W. Rea Aug 18 '09 at 14:39
up vote 5 down vote accepted

It largely depends on the type of files you are moving.

  • If your files are like PDF, JPEG movies, Installation files, etc,
    they are likely to be already compressed and will not give you a great advantage.
  • If its source files compressing will be quite useful.
  • If its lots of small files, at least a tar archive will be useful.

Finally, if your source machine has lots of processing power and memory,
compression would be in useful time -- else just a tar (based on above points) would suffice.

Since your network is just 100 Mbps, you should lean towards compression if that helps.
But, if you are transferring files that cannot be compressed much,
you should start accounting for the transfer time

Alternatively, you could consider other mediums for transfer (like USB/DVD).

share|improve this answer

Aside of the type of the file, this is especially dependent of the amount of files. While transferring bulk data is theoretically possible at network speed, there's a lot of overhead associated with file system operations such as enumerating files and properties, creating them and deleting them.

If you have a large amount of small files, the overhead can even get larger than the data to be transmitted.

In such cases, archiving the data before transmission can be a huge benefit. If it's badly compressible data (encrypted and/or already compressed data), I recommend to not compress the archive to save a lot of time - just use tar.

If the files are compressible (uncompressed bitmaps, text), also compressing might make sense.

share|improve this answer
Agreed. The data size saving from the compression will be limited. It is almost always faster to xfer a 1 20gb file then 100,000 files that when added together equal 20gb. – Tony Aug 18 '09 at 12:24

Nik is correct, it depends on the data. Generally speaking:

  • JPEG photos, movies, and music are not likely to be substantially further compressed by tar or zip, as these are already effectively compressed.
  • Compressing text files and program binaries will yield you significant space (and therefore transmission time) savings.
share|improve this answer

Technically yes, although around a LAN the gains would be small.

Basically in network transfers you go through several stages of requesting if the destination is ready to receiving your information, then you send some and check if it received it ok. Every individual file generates a extra step of 'new file incoming' and 'finished that file'. So if you zip/tar them all together you would only ever incur one 'new file incoming' message and one 'finished that file' instead hundreds or thousands of theses messages for sending non-compressed files.

Over a LAN it may take you longer to zip your files then send them instead of simply sending them. change your sending medium to a WAN and zipping is the way to go.

share|improve this answer

Probably the fastest technique is taring the data up, running it through a pipe, and then untaring at the other end.

Something like this

$ tar -czf - root_dir | ssh -c blowfish remote_machine  (cd parent_dir ; tar -xzf -)

The -z flag tells tar to compress, which should be very similar to a separate gzip step, which you include separately if you want.

If you need to copy or synchronize data a subsequent time, you can use rsync (-z gives compression). In particular, if the above command is interrupted, rsync will confirm your data, and send anything you missed.

It will be much cleaner if ssh is not asking you for passwords, but I think it will work even with passwords.

share|improve this answer

Compression might take longer to actually compress/send/decompress than to move the original files... However, often, a few large files get sent better than a large number of very small files, so compacting them together in a single file then transferring might be a good option.

share|improve this answer

In addition to the previous answers you should also bear in mind the load that you will be putting on the network and how this will impact other users. For such an amount of data, particularly if the same data is going to multiple destinations I would seriously consider using an external drive as the transfer medium.

share|improve this answer
+1 for common sense: External drives work great for local copying. – Chris W. Rea Aug 18 '09 at 14:41

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .