0

I have a VM and in it a process that consumes a lot of memory (~200GB). Some sort of in-memory DB. I need to run it on a standard laptop and I cannot recompile it or see the code.

I've added 256GB of swap space with pri=32767 in /etc/fstab and vm.swappiness=100 in /etc/sysctl.conf but it just won't load the DB quick enough (stuck on 4GB after 24h; doesn't seem to grow anymore).

(EDIT: I cloned the machine, changed the RAM from 256GB to 4GB, added hard disk, formated it as ext4 and created a swap file of size 256GB (dd -> mkswap -> swapon...))

Why did it stop growing?

I suspect the slowness is caused by it being a swap space, therefore the OS is busy "swapping" (load -> not enough room -> deciding what to swap out...).

I'm looking for a way to "add more memory" but make the OS treat it as normal memory. Or maybe my swap configuration is wrong?

I know it'll hurt performance, but it is acceptable for me.

The VM is CentOS 6.

2
  • 3
    “I need to run it on a standard laptop” Well, you won’t. Even if you could get it to start, an in-memory database isn’t built to cope with the slowness of disk access. You might as well perform whatever the program does by hand with pen and paper and it’d probably be faster.
    – Daniel B
    Aug 28, 2017 at 10:20
  • 2
    If there were no difference between swap and real RAM all our systems would have a very small RAM. If you need the speed of real RAM, you need real RAM.
    – xenoid
    Aug 28, 2017 at 12:12

2 Answers 2

2

Have you tried multiple swap files? 4 gigs sounds like you're hitting the file size limit for a 32 bit system.

I found this tutorial at centos.org which pretty cleanly laid out how to make swap files manually.

To add a swap file:

  1. Determine the size of the new swap file in megabytes and multiply by 1024 to determine the number of blocks. For example, the block size of a 64 MB swap file is 65536.
  2. At a shell prompt as root, type the following command with count being equal to the desired block size:

dd if=/dev/zero of=/swapfile bs=1024 count=65536

  1. Setup the swap file with the command:

mkswap /swapfile

  1. To enable the swap file immediately but not automatically at boot time:

swapon /swapfile

  1. To enable it at boot time, edit /etc/fstab to include the following entry:

/swapfile swap swap defaults 0 0

The next time the system boots, it enables the new swap file.

After adding the new swap file and enabling it, verify it is enabled by viewing the output of the command cat /proc/swaps or free.

8
  • You might want to add details on how this is done for everyone that doesn't know how to do it.
    – Ramhound
    Aug 28, 2017 at 13:38
  • 1
    this was a clean resource on how to basically do it. centos.org/docs/5/html/5.2/Deployment_Guide/… Aug 28, 2017 at 16:46
  • 1
    I like this tutorial a little more though because they include how to manage your permissions digitalocean.com/community/tutorials/… Aug 28, 2017 at 16:46
  • 1
    @assafmo Even if the processor is 64bit, that doesn't mean the filesystem is. For example, FAT32 is limited to 4Gb files. FAT16 and FAT12 are even worse.
    – jpaugh
    Aug 31, 2017 at 18:31
  • 1
    Who runs CentOS on a FAT partition?
    – Ken Sharp
    Feb 12, 2019 at 13:33
0

I eventually manged to convert the data into an SQLite database, and wrote https://github.com/assafmo/SQLiteQueryServer to be able to access it remotely. It worked flawlessly and was super fast.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .