How to keep Linux running after hard drive removal?

Question

I am working on an embedded Linux device with a USB hard drive. There is a swap file on this drive. The hard drive may be powered down at any moment before being powered back on few seconds later. This leads to various error messages, such as:

Write-error on swap-device
Kernel panic - not syncing: Attempted to kill init!

or exceptions in various applications, which is understandable since some page in/out requests failed. (Note that this kernel panic is also reproducible on a regular desktop Linux).

My goal is to somehow prevent this from happening and recover the system without rebooting the OS.

I can postpone the crash by calling scsi_block_requests, but I cannot keep them blocked forever.

I tried to kill some user mode tasks (using send_sig_info) in response to USB_DEVICE_REMOVE notification, but still keep getting exceptions.

I am also thinking whether it is possible to prevent SCSI device from being destroyed when USB device is removed, but keep it around and then somehow reconnect it to the new USB device, which will be created few seconds later when USB device is powered again. Since this is the same disk, then maybe I can resume all pending requests without re-mounting the disk.

What is your opinion? Can something like that be done? Or is it impossible to cleanly recover OS after hard drive removal?

Answer 1

Just killing the application that uses more RAM than you have won't guarantee that the swap file becomes unused. If that app had been running, its memory demands could have caused any other process to have had some of its pages swapped out to the swap device. Killing the memory hungry process won't bring those pages back until they are reloaded.

One possibility would be to do away with the kernel level swap device, and rewrite the problematic process so that rather than mallocing more memory than the system has access to and relying on the kernel to handle swap, just mmap the USB device (or a file on the device if the device is also used for other things) and manually handle your memory needs in userspace that go beyond the available memory (effectively implement an additional heap that lives on your USB Device). Then when the device needs to go away, simply killing or suspending that process will be enough to prevent the system from trying to access the device.

Answer 2

One option to consider would be compcache, which compresses memory pages, utilizing your RAM more efficiently so you don't need a swap file. Aside from that, the only way you are going to be able to solve your issue is by some hardcore hacking of the swapping code in the kernel.