8

I have a laptop with a LAMP setup. The HDD is slow, which causes my unit tests to run slowly.

I was wondering whether I could mount the web root the mysql database on some kind of ramdisk.

From what I have read of ramdisks, they are non-persistent.

Is there anyway to create a ramdisk that writes changes to an area of the hdd when shutting down and re-mounts the ramdisk on bootup?

3
  • I am not great at scripts so are there any programs or scripts available that will do what I want?
    – Linus
    Nov 22, 2009 at 22:07
  • I appreciate all the suggestions of a SSD but I am looking for something free and preferably something I can setup now. Thanks
    – Linus
    Nov 22, 2009 at 22:13
  • 1
    This looks like a superuser.com question Nov 22, 2009 at 22:34

11 Answers 11

3

What you are looking for is something that stores the fs data in memory, persists it if it can, but doesn't worry too much about losing your data (as in if someone turns off the power).

You could look into cachefs, and see if you can configure it to be really lazy about writes.

I'm also suspicious that you are solving the right problem. With a reasonable amount of memory, you shouldn't be blocking on those disk writes.

you can also set the writeback mount option for your filesystem. See, for example, this.

2
  • This sounds promising, what do you mean "with a reasonable amount of memory you shouldn't be blocking on those writes"? I have 3gb RAM but the unit tests that involve a lot reads and writes to the database seem to take a lot longer on this laptop than other machines so I thought if I mount the database to a ramdisk the reads and writes would be a lot quicker.
    – Linus
    Nov 22, 2009 at 22:27
  • Oh, the DATABASE. You need to tune mysql, I suspect.
    – bmargulies
    Nov 22, 2009 at 22:56
2

A ramdisk is probably the way to go. However if you have some money to spend, and want a really persistent solution, you may also want to look at SSD Hard disks. The technology is quite young and not as robust as the classic hard disk yet, but it is maturing and becoming kind of affordable.

1

You could write a simple rc script that does that. I remember doing a similar thing back in DOS days (with autoexec.bat) for fast access to some files.

You might consider buying an SSD or RAM drives with HDD interface instead.

1
  • 1
    Are there any readily available scripts that you know of?
    – Linus
    Nov 22, 2009 at 22:10
1

Dealing with ramdisks in Linux is quite trivial. You can create and use them in several different ways depending on your needs, then simply copy their content (with cp or dd) to hdd before shutting down your box or at a scheduled interval to avoid data loss for unexpected shutdown. Take a look at these how-to, I think they're straightforward enough:

http://www.vanemery.com/Linux/Ramdisk/ramdisk.html

http://wiki.archlinux.org/index.php/Ramdisk

M

1

Persistent Ramdisk on Debain/Ubuntu

This is a software package that does exactly that. It is a ramdisk (standard tmpfs or ramfs type) that is automatically backed up (and restored on startup) all the time.

The package is for Debian, but tweaking it for other Linux distribution should not be too hard.

0

If your app is small enough, you could grab a 30gb solid state drive for your machine. Benchmarks show them doing 3x or 4x faster reads than traditional drives (and something like 2.4x as quickly as 10k rpm drives). The bigger drives cost a bundle, but the 30 gig ones are just under $200.

0

Though this is not directly related to ram disks, this may help you get around your problem ..

What you can do is run two mysql servers with one of them being a primary that exists on the ramdisk and the other being a slave that saves all data to an HDD.

This way, if you need to start up the ramdisk-based mysql server after it has been turned off or whatever, you can just ..

cat mysqldump -uroot -hslaveServer dbName | mysql -uroot -hramdiskServer dbName

2
  • Can you provide any more details. For instance how do I set up one to run from a ramdisk? Would I have to change all the table types to MEMORY?
    – Linus
    Nov 22, 2009 at 22:36
  • Well, I was mostly thinking out loud. But I was imagining that you actually install a separate mysql server onto your ramdisk mount (/media/whatever) then run it from there.
    – Matt
    Nov 22, 2009 at 23:13
0

The OS already does that for regular file system accesses (buffers the writes to RAM and then flushes them later). But to be more robust, many databases then do a fsync() to force the OS to flush the buffers to disk which slows things down.

Controlling whether MySQL does a sync operation is done at the level of the underlying DB. I looked at InnoDB and it does not allow you to disable fsync.

But searching around I found libeatmydata which can disable fsync for a process which should give you the behavior you are looking for.

1
  • I did not know the OS did this for you. Most my tables are InnoDB I think I will have a go at using libeatmydata for development only. Thanks.
    – Linus
    Nov 22, 2009 at 22:40
0

You say you're running this on a notebook computer. Does the "hibernate" and resume functionality work consistently on it? If so... you could CONSISTENTLY use that, and also do one or both of these two TOTALLY SEPARATE things to help improve access times for both web pages and for MySQL queries:

  • (a) set up and use a ramfs for your DOCROOT; google for "Linux Ramdisk mini-HOWTO" for details. This may or may not pay off. You might be better off saving that RAM for use by MySQL (i.e. (b) below) especially if you have a lot of data in those MEMORY tables.

  • (b) migrate your most often accessed MySQL database tables over to the MEMORY storage engine Make sure your table structure doesn't require BLOB or TEXT columns; if it does, this won't work. Read up on this here: http://dev.mysql.com/doc/refman/5.0/en/memory-storage-engine.html

For each of the above you would have to schedule something to periodically copy the in-memory images off to disk. The advantage is that you'll only be writing to that slow disk every so often - kind of a slow-motion "sync" of sorts - rather than writing with every change to a MySQL table and every change to a file in DOCROOT. So for both cases you could use CRON to schedule a "backup":

  • In the ramfs case you could create a tar archive of all the content in the filesystem.

  • For the MySQL tables, you could create a series of statements similar to the following:

    insert into someInnoDBdatabase.tablename select * from someMemoryDatabase.tablename

    Assuming your table in both databases has the exact same structure that'll copy everything from the table in the someMemoryDatabase into the someInnoDBdatabase table of the same name - effectively backing up the MEMORY table on-disk.

On creating those tables in the memory database:
Assuming you have tables managed by an on-disk engine now, you could use 'show create table tablename' to get MySQL to give you exactly the SQL you'll need to re-create that table... then change the engine part to specify to use the MEMORY engine, and execute that.

Hope this helps!
-pbr

0

What you want to achieve is indeed very much possible, but I'm not aware of any ready-made solutions. You'd have to write the according scripts more or less on your own. I have a feeling this fact alone is a deal-breaker for you, but if you want to take on the task I'm confident you'll get all the help you need.

0

"The OS already does that for regular file system accesses (buffers the writes to RAM and then flushes them later). But to be more robust, many databases then do a fsync()..." -- R Samuel Klatchko

So to speed up your unit tests, so they don't sit around waiting for each and every write to be completely fsync'ed, there are several things you can try:

  • run the tests in a virtual machine that lets you turn on "unsafe" write-back caching, which effectively turns fsync() into an extremely fast no-op. (But only do this for unit testing, not on a production database).
  • run the tests in a virtual machine with a -snapshot option such as QEMU -- this redirects all writes to a log file. During the test, it redirects some reads to that log file to maintain the illusion. Relatively linear writes to the log file are likely to be faster than writes that are scattered all over the disk platter.
  • somehow minimize the number of writes in your unit test -- does it really need to be writing to the test database?
  • If you can trim the test database used by your unit tests to fit entirely in RAM, perhaps getting the OS to pre-cache the entire database in one go (rather than read and cache a little bit at a time, as needed in the test) might make tests faster; with something like

    cat the_test_database.db > /dev/null

  • run the tests on a machine with RAID-0 striping to speed up reads and writes. (I've been looking for laptops that support RAID -- alas, laptops that support RAID are very rare).

You must log in to answer this question.