Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

short version first: I'm looking for Linux compatible software which is able to transparently cache HDD writes using an SSD. However, I only want to spin up the HDD once or twice a day (to write the cached data to the HDD). The rest of the time, the HDD should not be spinning due to noise concerns.

Now the longer version: I have built a completely silent computer running Xubuntu. It has a A10-6700T APU, huge fanless cooler, fanless PSU, SSD. The problem is: it also has (and needs) a noisy HDD and I want to forbid spinning it up during the night. All writes should be cached on the SSD, reads are not needed in the night.

Throughout every day, this computer will automatically download about 5 GB of data which will be retained for about a year, giving a total needed disk capacity of slightly less than 2 TB. This data is currently stored on a 3 TB noisy hard disk drive which is spinning day and night. Sometimes, I'll need to access some data from several months ago. However, most times I'll only need data from the last 14 days, which would fit on the SSD. Ideally, I'd like a transparent solution (all data on one filesystem) which caches all writes to the SSD, writing to the HDD only once a day. Reads would be served by the cache if they were still on the SDD, else the HDD would have to spin up.

I have tried bcache without much success (using cache_mode=writeback, writeback_running=0, writeback_delay=86400, sequential_cutoff=0, congested_write_threshold_us=0 - anything missing?) and I read about ZFS ZIL/L2ARC but I'm not sure I can achieve my goal with ZFS. Any pointers?

If all else fails, I will simply use some scripts to automatically copy files over to the big drive while deleting the oldest files from the SSD.

Edit: Cache really is the wrong term for what I wanted. It seems the original intended solution is not possible for me at this time. Thanks to everybody who contributed!

share|improve this question
1  
Why don't you just keep 14 days (or however much fits) worth of data on the SSD and write a script that runs once a day and moves anything older than that to the HDD? This way if you are using anything less than 14 days old you will be using the SSD and the HDD will only write once a day. If you needed something older you would have to access the HDD. –  Richard Powell Oct 31 '13 at 9:53
    
That's exactly what I will do, as I suggested in the last sentence of the question :) One of the reasons I asked the question was to avoid the (admittedly small) hassle of needing to spin up and mount the drive manually as opposed to just having a transparent file system. –  sirprize Nov 1 '13 at 19:24

5 Answers 5

up vote 4 down vote accepted
+50

Thats not really how or why caching works - caching is meant to take advantage of the faster speed and better random access of the SSD or other memory, rather than minimising writes to the hdd. Caching is designed to maximise speed, not reduce write, by having a smaller, faster buffer backing a large, slow storage device.

In fact, bcache, which is part of the mainline linux kernel, will simply pass on sequential writes to the hdd, rather than passing it through the SSD since there's no performance advantage.

ZIL/L2Arc uses the SSD to store logs (in the case of ZIL) and commonly used file clusters (with L2Arc). ZIL speeds up synchronous writes by storing them till they're all ready to be written. L2Arc stores commonly accessed files on faster storage. Neither of these will let you do what you need.

Fusion drive also uses a SSD and a HDD transparently putting commonly used files on the SSD for faster reads, and HDDs for bulk storage of less often used files. It too doesn't let you use the SSD as scratch space, and back up files to the main drive every so often

What you're looking at isn't caching at all, but rather periodic backups of the system to the HDD. I'm guessing ou might be able to fudge something together with Aufs -the first branch on the HDD with larger files (so it takes precidence), and a branch on the SSD, then run a script that moves files from the SSD to the HDD periodically, while having it accessable from the same location. I've not tested this out yet, but unlike caching, periodically moving files and using aufs will likely be exactly what you want.

The simplest way to sort files by age is ls -tr - t sorts by time (newest to oldest) and r reverses the order. (If you have a specific range in mind find . -mtime n works great n as a specific number gives you files modified n days ago, -n gives you files modified in the last n days and + n gives you files modified n days ago). You can find some ideas on how to use it here

Since quietness is your real goal, you can check if your drive supports silent mode with hdparm -M /dev/sda - this should output something like acoustic = 254 (128=quiet ... 254=fast), in which case you can make your hard drive quieter with the command hdparm -M 128 /dev/sda. Run the opposite command hdparm -M 1254 /dev/sda when you need more speed.

share|improve this answer
    
Thanks for the input, this is my favorite answer yet. I did not know about aufs and will look into it. –  sirprize Oct 31 '13 at 9:34
    
Transparency isn't really possible for a batched up process like yours - I can think of lots of ways to trigger a backup type job, but you're really looking at scripting, some sed/awk to select the files you need, and moving the files, maybe with cron. I half suspect hdparm alone may solve the 'loudness' issue with your hard drive significantly, and it'll let you power down the drive when not in use, which might be useful too. –  Journeyman Geek Oct 31 '13 at 9:49
    
Thanks again, I'll use some bash-fu, cron, hdparm, mount and rsync as suggested in the last sentence of the question. hdparm -M did help making the drive a little quieter, but I prefer the disk not to spin at all during the night. –  sirprize Nov 1 '13 at 19:28

Several variations of Puppy Linux are based on Ubuntu but using Aufs.

Puppy runs entirely from RAM, there are no HDD accesses in normal operation.

If you tell Puppy that your HDD is a flash drive, it will dump the RAM filesystem to disk periodically instead of continuously. This was designed to reduce the wear of frequent writes to flashdrives but works fine on HDDs too.

As with just about everything in Puppy, you can use a GUI to set how often you want the RAM filesystem saved. Typically, this will be every 10mins or so but there's nothing to prevent setting it to dump every 720mins (12 hours) for example.

Puppy Linux Event Manager

This excerpt from my Grub4DOS bootloader menu.lst file shows the setting for the pmedia kernel option that says it is using flashdrive instead of an HDD...

# menu.lst produced by grub4dosconfig-v1.7.1
#
# 'kernel ... pmedia=ideflash'
#     Treat the HDD as a flashdrive: infrequent writes of filesystem to savefile

title Puppy Lupu 5.2.8 (sda3/puppy528lu) Cached\nUse SWAP; changes to RAM; flush to disk every 10min (fastest)
  find --set-root --ignore-floppies --ignore-cd /puppy528lu/initrd.gz
  kernel /puppy528lu/vmlinuz   psubdir=puppy528lu pmedia=ideflash pfix=copy,fsck
  initrd /puppy528lu/initrd.gz

A note about Puppy Linux "variations"...
I'm using Puppy Linux 5.2.8 "Lupu" which is based on (built by paring down) Ubuntu 10.04 Lucid Lynx. This version is also often called "Lucid" instead of "Lupu" due to its origins. There are also versions based on Slackware and some other distributions. Since you are already using Xubuntu, Puppy Linux Lupu would be the most familiar. It would also be much faster. I used Xubuntu for about a year before finding Puppy.

There is also a much newer version based on "Ubuntu Precise Pangolin" called "Puppy Linux Precise." There are major changes to the kernel used in the newer version that may not work well with older hardware. If your machine is less than 5yo, you might be happier with Precise. I believe the technique described above will still work on Precise though the exact options, etc. may be different but am unable to test it.

share|improve this answer
    
Thanks! Have an upvote. –  sirprize Nov 1 '13 at 19:08

Since the other answers have treated the problem from the software point of view, I thought to include some information about hardware solutions.

The article Sound Proof your Hard Drive lists the following options:

  • Defragment your hard drive
  • Wrap hard drive in rubber bands to dampen vibrations
  • Go for a new hard drive: Most hard drives of present day are equipped with fluid dynamic bearings, which allow the platters to remain almost silent even when spinning at a very high rate.

I can testify from my own experience that I can never hear my new 4 TB hard disk, even when copying very large files.

With some research, one can also find acoustic dampening materials for hard disk such as the SilentDrive HD Enclosure & Acoustic Materials or the NoiseMagic NoVibes III Silent Hard Drive Mounting System. I don't know where these are sold, and the rubber bands above seem to me almost as efficient.

One can also sound-proof the entire computer. Some example websites that sell sound proofing materials are Quiet PC USA, AcousticPC or Dynamat, but there are many such.

Some computer cases are already designed for silent operation. See for example
Six Low-Noise, Performance-Oriented Cases, Tested.

As a last remark, a noisy hard drive that clunks, clicks, or screeches may be a sign that your hard drive is beginning to fail, so take care.

share|improve this answer
    
Thank you for your feedback, have an upvote. I am already using specialized hardware -- in this case, completely silent hardware except for the drive. By "noise" I'm referring to normal sound of a healthy hard drive. I will simply have to use cron in order to spin the drive up, mount it, make a backup, unmount it and then spin it down again. –  sirprize Nov 1 '13 at 19:20

You wont like my answer but your 're actually abusing the term cache, you are actually describing a backup solution. Have a look at rsync, or search further for other backup solutions if you would like something more complete.

share|improve this answer
    
This is absolutely true, and is what I covered in depth in my answer :) –  Journeyman Geek Oct 31 '13 at 23:22

short solution and quickly implementable (but not answer to exactly what you want) is: transfer the storage disk elsewhere. it may be a netowrk file server chassi (like the WD world edition) or USB3 disk and you can have it in the same room as the quiet pc, or in a different room so as to avoid listening to it completely.

This is better in my opinion as you'll surely only be using the drive when you want to transfer data over the network - the OS may be doing some other tidy-up or filesystem tree rebalancing that you can't avoid and would disturb you. So different room is a big plus.

Caching should be easy enough - just backup the day's data onto the external disk, which you can accomplish using solutions like rsync or unison.

The longer solution is again not ideal (ie no single application) but you'll get the drive in the same box with the ssd and achieve higher speeds (but do you really need higher speed than network for just 5GB / day?):

you can use smartctl to set the spin down time of your drive and retain this drive in an unmounted state - to avoid having the OS do stuff on the drive. Then just write a script to mount and backup (as above) the data you need.

share|improve this answer
    
Unfortunately, those are all things I wanted to avoid. Thanks for your time though! –  sirprize Oct 31 '13 at 9:36

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.