0

I am trying to reduce bandwidth when accessing files on a shared folder over a slow network.

I tried to achieve (filename.tar.gz) the folder and mount it but it seem to transfer more data than the file I am trying to access.

Is there a way to compress the data over the network to optimize transfer speed. If I use zfs on the remote system when will the file be decompress, before or after the transfer.

  • 2
    The file you're sending is already compressed. Attempts to compress it again could increase its size. – AFH Sep 3 at 14:22
  • How big are said files? How bad is your speed 10mega bit, dial up, or etc – cybernard Sep 3 at 18:58
  • I want to map my home directory for mostly text files on a 10Mb network for now. I may move it to web hosting later on for access from anywhere. – Michel S Sep 4 at 2:23
0

You have a few options. Choose 1 and only 1. There are no benefits to compressing multiple times.

  • Compress the file using tar -czv folder_name or any compression application of your choice. Then copy it over network. This will probably get you the best compression as gzip or lmza based compression programs are fairly unbeatable in general purpose compression.
  • Compress the file using a media specific compression algorithm. If you are copying music, pictures, or videos, media specific compression algorithms can greatly reduce file size. With media, you also have the option for lossy versus lossless compression and down-sampling. Your needs may vary.
  • Use compression when running scp/rsync/etc. (rsync -z). This is on the fly compression, and you effectively trade CPU for more apparent network bandwidth. It can be fairly decent, especially for low entropy data like text files on very slow connection. Assuming an average server, with actual network bandwidths over 100Mbps you will see diminishing returns. Over 1Gbps, on the fly compression may hurt transfer speeds as data can be sent faster than the cpu can compress it. The cutoff number varies depending on nature of the data, the choice of compression algorithm/settings, and CPU speed.
  • Tks for your reply. I may have miss the mark explaining my needs. After my post and tks to your reply, I came across an article on ext2fs compress patch (e2compr) that does exactly what I want. The file system is ext2fs and the patch expose the compress flag that can be set on file by file or directories as wanted/needed. Can you help me edit my question to better explain my point and tell me how I should share my experience once implemented. Link: en.wikipedia.org/wiki/Ext2#Compression_extension – Michel S Sep 4 at 2:51
  • Filesystem compression will not keep the file compressed when you send them over network. The moment you access the files, they get uncompressed. Are you removing the network transfer part of your question? – Andy Sep 4 at 23:10
  • Andy, the e2compr,when does compression happen, documentation (e2compr.sourceforge.net/manual-0.4/e2compr-0.4_17.html#SEC17) states that compression happen when cluster boundary are reached, letting me to assume that cluster are fetched from the remote storage as needed and in compress form to be decompress by the local kernel. I also assume that zfs does the same but it isn't clear in the documentation and I can not understand the code well enough to ascertain those assumptions. – Michel S Sep 5 at 19:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.