5

I left the title intentionally vague, because I don't care about using specifically FTP or gz if any other tool will work better.

I have a lot of data that I access very rarely (movies I watched, backups, etc.) that I want to store on a home FTP server. I want to automatically compress all files I upload on the server and automatically decompress them when I download the files. I know I can just use cron with gzip -r on the server and gzip -rd everything I download, but I'd prefer if the server was configured to do this itself seamlessly, so I could use it as if the files were stored decompressed.

Does FTPD or another file server have a functionality like this? A bonus point if the server keeps the decompressed file in a temporary storage until fully transferred, so things like streaming over FTP (using VLC) work without issues.

Improve this question
5
  • 7
    Binary files such as movies do not compress well. Encoding them with a more efficient codec is more effective.
    – harrymc
    Feb 10 at 12:06
  • @harrymc thank you, i'll look into that later. For now I'm just looking for a quick and easy solution to clean up my PC.
    – Reverent Lapwing
    Feb 10 at 12:21
  • 9
    @harrymc it's not because they are binary, it's because they are already compressed. There is no problem with compression and binary files.
    – user253751
    Feb 10 at 20:29
  • 1
    Sounds like your use case is perfect for a NAS (multiple systems accessing the same file store). I recommend two, with one syncing to the other as a backup. Indeed, you won't likely see much space gained on hard-to-compress files, but there's no downside (at least on a 1gbps LAN), as any CPU can stream the compression faster than it can be transferred, and gain the benefit for the files that can compress well. You can easily build your own, generally with better features/performance than a premade one.
    – Corrodias
    Feb 11 at 5:05
  • Compressed file formats like movies and songs (mp3, mp4, aac etc.) generally cannot be compressed further because they're at their limits of information entropy (they have as much randomness as possible while still containing information). Attempting to gzip or zip an mp4 movie will usually increase its size. The way to go is to re-encode the movie with lower quality encoding (higher compression and less data)
    – slebetman
    Feb 12 at 11:45

2 Answers 2

Reset to default
23

The easiest solution would be to create a compressed file system on BTRFS or ZFS. I think there are other Linux filesystems that support compression, but I am not sure. Then copy your data to it. Files will automatically be compressed and decompressed on access, by the server.

You mentioned storing movies and backups. Assuming these files are already compressed (media by format/codec), you won't be saving much space. The same goes for many other formats like images, audio, ebooks, etc.

On a side note, Windows NTFS has compression as an option, as well.

Improve this answer
10
  • 1
    This is exactly what I was looking for. The "backups" are basically an unorganized pile of files me and my fiance have accumulated over multiple PCs and smartphones and I want to have free access to them when I look for a specific file or I'm in a mood for organizing or reminiscing. There is also plenty of large, lossless recordings from video game streams I should probably convert to a more efficient codec when I find a time. Whatever the gains might be, I'd rather keep things compressed as a rule unless I use them often, even if the saved space is not much.
    – Reverent Lapwing
    Feb 10 at 13:08
  • 5
    @ReverentLapwing Are you sure? I tried compressing a 3GB video file I have here with gzip. Took 90 seconds, saved 6MB of space. That's about 0.2%. I also tried bzip2, that took 8 minutes and made the file larger.
    – marcelm
    Feb 11 at 0:13
  • 1
    +1 for ZFS. If you enable compression it will automatically detect blocks for which compression is ineffective and not compress those. I have used it for years on a NAS (as OP asked for)
    – Josh
    Feb 11 at 5:22
  • 5
    @ReverentLapwing "as far as I understand they become more efficient as you feed them more data" - They don't. Compression algorithms consider a limited amount of data when compressing, otherwise their memory/CPU usage would be unbounded. For many algorithms, this is simply "the last X bytes before the current position", called a sliding window. Sizes for this window ranges from KBs to MBs. gzip, for example, uses a fixed 32KiB window. I don't feel 2% is worth compressing video files, especially since compression amplifies the effects of data corruption. But that's your call, of course.
    – marcelm
    Feb 11 at 19:48
  • 2
    @ReverentLapwing Media formats almost universally use lossy compression, while general-purpose compression must be lossless. Lossy algorithms have much better compression ratios than lossless, but only work for that media type. If a lossless method was useful for media the codecs would already be using it. If a 2% gain is meaningful to you then spend your time re-compressing your videos with HEVC or AV1 instead of messing with GZip. You can probably get 50% savings. (very happy ZFS user here. As expected, I get basically zero compression on my media datasets.)
    – Adam
    Feb 12 at 6:09
0

I use Truenas for my Network Attatched Server setup. I built it with the parts from my old desktop. by default the (ZFS) filesystem, or "pool" has some level of compression (i think it's mostly useful for log files and the like).

It also allows for more complex "compression solutions" such as deduplication where you can make it look at a deduplication table which you can store on for example an SSD raid where it will look at hashes of blocks of data. i also believe this solution requiers your NAS to have a specific amount of free memory for this specific feature to work. this system is worth it only for very specific usecases and introduces more parts into the system so it's not neccesarily a good thing to start using unless you know that it will be good for your specific usecase.

For videofiles i use Tidarr. i run it on my gaming computer from where it looks at my videofiles and if i happen to create a non-h.265 videofile it will automatically encode without me having to do anything about it after i have everything set up.

Improve this answer
2
  • 1
    Warning: zfs deduplication requires massive amounts of RAM. You almost certainly do not want zfs dedup, no matter how much you think you do want it…
    – Josh
    Feb 12 at 15:05
  • Yeah, everywhere i read everyone/everything discourages it's usage. I think it introduces unnecesary risks with having to keep the deduplication stuff stored safely, and the ram thing, although it i think does depend on the size of the storage. I think truenas has a lot of other choises for compression which might take more or less cpu power, with the default being the "optimal" for fast and "ok" compression.
    – julkkis666
    Feb 14 at 20:30

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .