First, a confession: no, I didn't do the backups I should have.

Second, the situation:

I have a Dell XPS 9550 with a solid state disk running Fedora 25.

I was working on a file and tried to save it when I was told I was trying to save to a read-only filesystem. Turns out my filesystem is read-only now and there are i/o errors all over the place.

I was able to save some of the files by emailing them to myself through an open web browser, but that crashed and I'm unable to relaunch it. But I still have files of interest open in an editor. I can't seem to save the files anywhere, but I can copy their contents. If only I could find a way to exfiltrate the file contents, I could save myself months of work.

But there are some horrible limitations. I attempted to insert a USB drive, but no device appears to represent it, and the mount command dies with a segfault. I can attempt to ssh to another computer, but I get "bus error" and it dies. ping, dmesg, ifconfig, none of these work. But I do have vim and less and ls and can spawn new bash instances.

No lynx, no firefox, no google-chrome. There's no DVD drive.

Basically it seems my SSD has died. Or maybe the whole motherboard. I have documents of great value still in memory, I have an IP address and network connection, I can run a few random commands and have 3500 more on the path that I could try.

cat and gcc seem to work. I can write to files in /tmp. I have a running ipython instance that still seems to work.

So... what I've tried so far has failed. But I feel like there are still a thousand possibilities. What am I not considering? How could I possibly get these files off of my dying computer?

There must be a way.

UPDATE: New stuff:

  • I lost my network connection due to my own stupidity.
  • I wrote a Python script to replace cp and cp -r
  • Unless I find some way to create a /dev entry for the SD card, or for USB drives, then my best bets for getting data out seem to be the screen and possibly the speakers/audio cable.
  • I'm writing a script to try reading files and output which ones are readable.

Suggestions still very welcome!

UPDATE 2: Newer stuff:

  • On the dying computer I wrote a Python script that will read a file bit by bit and try to convey those bits by flashing the screen one color or another. Right now it's trying to do a two-bit code where red, green, blue, and white all represent a two-bit pair. This isn't working that well, though, so I might just switch to two colors and do one bit at a time.
  • On my other laptop (the trusty old Thinkpad that I gave up for this hot new XPS) I wrote a script that reads from the webcam using the OpenCV Python library. The idea is to have it decode the codes sent by the other computer. The trouble is that the framerate from the camera is something like 15 frames per second, which means if I had a perfect, errorless transfer my maximum data rate would be 30 bits per second, i.e. 225 bytes per second. That's 324k per day.
  • On the dying XPS I can use tar to pack the desired files into a single archive, which is 1.7 MB. Unfortunately, gzip, bzip2, xz, lzop and whatever compression utilities are unavailable. BUT using Python's zlib module I can compress this file down to 820KB. Given that size, I could probably get this thing sent over in a couple of days.
  • Because this transfer method will likely be very error prone, I'm going to implement Hamming codes on the XPS to add some error correction as I transmit the data.
  • Likely there will be complications because that's what happens, but at least it seems somehow feasible to get this data out!
  • Since this is still a pretty sucky way of sending data, I looked more into USB serial drivers. The modules I've tried to load (usb-serial-simple, usb-debug, safe-serial) give i/o errors. I don't think it's built in to the kernel, either, because there are no /dev/ttyUSB* devices present.

Thanks for everyone's suggestions thus far---I know this isn't even a well-defined question since you guys don't know in advance which programs/files can be read or not. Still open to better suggestions than this video approach!

UPDATE 3: Newest stuff

  • I got a PS3 Eye webcam and, after disabling its automatic gain and exposure, am successfully reading data off of the XPS, albeit at an errorful 1 byte per second. This is a great success---the first data exfiltrated! But the rate is too slow to get my 820KB out in any sort of reasonable time, and the error rate is too high.

One bit transmission with clock

  • The problem is that writing to the terminal is too slow. The screen updates aren't anything like instantaneous, thanks (I think) to the slowness of the urxvt terminal emulator that I have access to.
  • I discovered that I have access to a Rust compiler on the XPS. I rewrote the transmission script using Rust to see if that would improve the terminal refresh speed, but it didn't help.
  • Because I'm unlikely to be able to increase the framerate, I'll have to try to increase how much data I get per frame. My current approach looks something like this:

grid transmission

The right half is still a clock signal, flashing on and off to mark the arrival of new frames. But the left is now a grid where each cell is marked by a red square in the corner, and then the green cell to the right and down from the red square is flashed on and off to indicate a bit. The red squares should let the receiving computer calibrate where the cells are located. I haven't got any data across this way yet, but it's what I'm working on.

  • Someone suggested that I look into writing QR codes instead of these ad hoc color patterns. I'm going to look into that, too, and perhaps implement that instead of this grid approach. The error correction would be a nice win, as well as being able to use standard libraries to decode.
  • I learned that I have access to libasound (the ALSA sound library), but not to the header files associated with it (alsa/asoundlib.h or whatever). If anyone knows how to make use of a shared library without the headers, or can help me write just the right header to let me produce audio output, then I could have an audio-based way of getting the files out.
  • Alternately, if someone could help me manipulate the USB devices without access to libusb then maybe I could do something with that?

Moving forward!

UPDATE 4: audio output produced!

User Francesco Noferi has done some great work helping me utilize the ALSA library mentioned in the previous update. The C compiler had a problem, but using the Rust compiler I was able to use the FFI to call directly into libasound. I've now played a bunch of my data over audio and it sounds like music to my ears! Still need to get a real communication channel established, but I'm feeling very hopeful. At this point my job is basically to implement a modem, so if anybody has any guidance on good ways to do that I'm all ears. Ideally modulation that's easy to implement by hand and demodulation for which there's an existing library I can use. Since this can go directly over an audio cable and not through the phone network, theoretically we can do much better than 56kbps or whatever the standard was back in the day, but in practice who knows what we'll get.

Thanks to everybody following along here and at /r/techsupportmacgyver and at /r/rust contributing so many excellent suggestions. Going to get this "modem" implemented soon and then I'll finish this up with an epilogue. I think I might put my code up somewhere for other desperate folks to make use of in the future---maybe even a repository of weird exfiltration tools that are easy to type into a dying machine by hand? We'll see what happens.

UPDATE 5: It took me a long time wrestling with ALSA and my cheap StarTech USB audio capture device (no builtin line in on the receiving laptop), and many false starts trying to roll my own transmission protocol, but finally under the advice of some Ham radio enthusiast friends of mine I implemented the RTTY line protocol running at 150 baud, which in practice gives me maybe about 10 bytes per second. It's not super fast but it's fairly reliable. And I'm very nearly done transferring my 820KB file, verified using CRC32 checksums (using the crc32 functionality from Python's zlib module, which I have access to). So I'm declaring victory, and want to give my thanks once again! I'll spend some more time finding further files that are readable and which I can transfer, but the foundation is in place. It's been fun working with you all!

FINAL UPDATE:

On the dying machine:

$ tar cf ./files
$ ./checksum.py ./files.tar 9999999
Part 1 checksum: -1459633665
$ ./zlib_compress.py ./files.tar
$ ./checksum.py ./files.tar.z 9999999
Part 1 checksum: -378365928
$ ./transmit_rust/target/debug/transmit ./files.tar.z
Transmitting files.tar.gz over audio using RTTY
Period size: 2048
Sample rate: 44100
Samples per bit: 294
Sending start signal.
Transmitting data.
nread: 2048
nread: 2048
...
nread: 2048
nread: 208
Transmission complete. Sending hold signal.

On the rescue machine:

$ minimodem --rx -8 --rx-one -R 44100 -S 915 -M 1085 --startbits 3
            --stopbits 2 --alsa=1 150 -q > ./files.tar.z
$ ./checksum.py ./files.tar.z
Part 1 checksum: -378365928
$ ./zlib_decompress.py ./files.tar.z
$ ./checksum.py ./files.tar
Part 1 checksum: -1459633665

:-)

  • 4
    Related:  Unix Recovery Legend. – G-Man Nov 16 '17 at 8:23
  • 2
    Go to the directory where you have the files and issue the command python -m SimpleHTTPServer. Now you are sharing the files through an http server in port 8000. Open a browser in other device in the same network and type the following : http://<IP address>:8000 and start to download everything you can. – jcbermu Nov 16 '17 at 8:36
  • 2
    Great piece of geek stuff you've got here. I wish I could upvote twice. – Kamil Maciorowski Nov 29 '17 at 18:06
  • 1
    How about cooling the crap out of the RAM to reduce volatility and moving it to an alternate XPS? en.wikipedia.org/wiki/Cold_boot_attack – root Nov 29 '17 at 23:47
  • 1
    Somewhat off-topic, but it would be great if you could link the related reddit threads so we can see what's been suggested there. – Bob Dec 2 '17 at 8:53

10 Answers 10

up vote 15 down vote accepted

here's an example libasound program with just enough definitions to get basic 2-channel 44.1k wav output going without the headers.

EDIT: I'm actually not sure if straight up dumping the data as wav would work, as noise when recording could easily damage it, but you can probably do something like a sine wave of bits at high frequency which is more reliable

EDIT2: if aplay is present and works you can also use that and just write a program that output raw audio and pipe it into aplay or anything that can play audio

EDIT3: modified it to not use any headers at all

if -lasound doesn't compile, add -L/path/where/libasound/is/located

/*
    gcc alsa_noheader.c -lasound
    cat stuff.wav | ./a.out
*/

typedef unsigned int uint;
typedef unsigned long ulon;

int printf(char*, ...);
void* malloc(long);
long read(int fd, void* buf, ulon count);

int snd_pcm_open(void**, char*, int, int);
ulon snd_pcm_hw_params_sizeof();
int snd_pcm_hw_params_any(void*, void*);
int snd_pcm_hw_params_set_access(void*, void*, int);
int snd_pcm_hw_params_set_format(void*, void*, int);
int snd_pcm_hw_params_set_channels(void*, void*, uint);
int snd_pcm_hw_params_set_rate_near(void*, void*, uint*, int*);
int snd_pcm_hw_params(void*, void*);
int snd_pcm_hw_params_get_period_size(void*, ulon*, int*);
long snd_pcm_writei(void*, void*, uint);
int snd_pcm_prepare(void*);
int snd_pcm_drain(void*);
int snd_pcm_close(void*);

int main(int argc, char* argv[])
{
    void* pcm;
    void* params;

    int rate;
    int nchannels;
    ulon frames;
    void* buf;
    int bufsize;
    long nread;

    snd_pcm_open(&pcm, "default", 0, 0);
    params = malloc(snd_pcm_hw_params_sizeof());
    snd_pcm_hw_params_any(pcm, params);

    /* 3 = rw_interleaved */
    snd_pcm_hw_params_set_access(pcm, params, 3);

    /* 2 = 16-bit signed little endian */
    snd_pcm_hw_params_set_format(pcm, params, 2);

    /* 2 channels */
    nchannels = 2;
    snd_pcm_hw_params_set_channels(pcm, params, nchannels);

    /* sample rate */
    rate = 44100;
    snd_pcm_hw_params_set_rate_near(pcm, params, &rate, 0);

    snd_pcm_hw_params(pcm, params);
    snd_pcm_hw_params_get_period_size(params, &frames, 0);

    bufsize = frames * nchannels * 2;
    buf = malloc(bufsize);

    /* read file from stdin */
    while (nread = read(0, buf, bufsize) > 0)
    {
        if (snd_pcm_writei(pcm, buf, frames) == -29)
        {
            printf("W: underrun\n");
            snd_pcm_prepare(pcm);
        }
    }

    snd_pcm_drain(pcm);
    snd_pcm_close(pcm);

    return 0;
}
  • I upvoted you and think you deserve tons of credit for this awesome approach! Here's the trouble: I get IO errors on all four of the include files you want to use. So I need a way around that. I can to the preprocessing on the other laptop, but that results in a 63K file, 11K gzipped. If I can find a means of getting data in to the XPS then that's feasible, but I'm not excited about typing it in by hand. The compiled file is 10.4K, or 2.4K gzipped. That's getting to where I could type it in by hand, but errors would be hard to detect. – Josh Hansen Nov 30 '17 at 6:49
  • 1
    I have edited the program to not use any headers at all, try that – Francesco Noferi Nov 30 '17 at 13:14
  • gcc gave me "Bus error" (which get with some programs but don't really understand.) Fortunately, I have a Rust compiler which is working, so using Rust's FFI capability I implemented your code above using Rust. It's compiling and running, but I still don't hear anything. But I'm not really sure if the volume is up---it might be muted. But I can't run alsamixer/xfce4-mixer to check. I'm looking into using alsalib directly to make sure the volume is up. Thank you so much for your effort on this! – Josh Hansen Dec 1 '17 at 4:04
  • 1
    nice to hear you got it running! for volume, here's a program that enumerates the default device's mixer controls and maxes them all: gist.github.com/008f0c3acdbcae886a19868a0554987b . if you still have issues you might need to write something to enumerate audio outputs and their mixer entries to make really sure it's really broken and not just outputting to the wrong/muted device. could also try adding error checking to the audio output program to see if it's erroring. bus error means "tried to access memory that couldn't possibly be there" so maybe corrupted memory mapped libs – Francesco Noferi Dec 1 '17 at 13:25
  • Turns out the volume was up, but I wasn't actually sending any bytes out to the speakers---somehow I don't understand how read into a buffer in Rust, it always reads 0 bytes. But using some bytes I constructed I was able to produce audio output! This is great, it seems like a good path to getting the data off in a nice clean way. Now it's time to learn about modulation/demodulation strategies so I can do this intelligently. Thanks so much for your work on this---much appreciated! – Josh Hansen Dec 1 '17 at 18:29

Is your HDMI or any other display out port working? If so you can use a screen capture device to record it as video and process later. Thus not being limited by your webcam's framerate.

How about you hex-encode your data and output it page per page to the terminal?

You can add a prefix with the the offset in the binary so that you can easily regenerate a page (for manual correction?)

Then on a different computer use some OCR software to scan the pages.

80x25 terminal would yield 1000 bytes per page (minus some space for the prefix). So in roughly 1000 pages you could get out your data. Even at one page per second, that's less than 20 minutes.

Hex encoding is easy to write and also provides a raw form of error correction (there are only 16 valid symbols).

can you set up a network connection? ssh might be a bit too much but if you can use netcat on 2 machines you might be able to transfer data. one in sending mode, one in listening mode. If all you need to do is transfer text then this 'could' be a solution..

edit: nevermind, just read that you lost your network connection as well..

  • 1
    I think that your answer deserves to be up there. nc has no dependencies other than a functioning network stack, which OP had until he blew it up. That drastically reduces the chances that the executable won't start. If someone else runs into a similar problem, that's definitely a useful solution. – zneak Dec 1 '17 at 17:31

You may be able to email yourself from the commandline, including sending files.

Something like:

$ mail -s "Hello World" user@yourmaildomain.com < /tmp/urgentFileToSave.txt

should work.

Some further examples: http://www.binarytides.com/linux-mail-command-examples/

  • Sadly, mail is not among the (essentially random) collection of programs that I can run. Plus I lost my network connection :-( – Josh Hansen Nov 16 '17 at 9:27
  • Ouch. OK, all I've got left is cat the files and take photos with your mobile. – mcalex Nov 16 '17 at 9:32
  • It may come to that! – Josh Hansen Nov 16 '17 at 9:35

Can Python access your audio-out port? You may try transferring data using that - see this or that.

  • I don't think I have Python's audio libraries installed, so probably not. But there are ALSA devices in /dev/snd. alsa-lib is installed, but I don't think I can read it. Is there some way to write to the audio devices directly? – Josh Hansen Nov 17 '17 at 1:18
  • @user17219 Are you sure there is no ossaudiodev library available in your Python installation? It should be a standard module in Linux and FreeBSD, if I read the documentation correctly. import ossaudiodev worked in my Ubuntu out of the box. – lukeg Nov 17 '17 at 8:50
  • actually you're right that it's present, but unfortunately it wasn't in memory so I can't read it---it tries to read the module from disk which results in an i/o error :-( That was an excellent suggestion, though – Josh Hansen Nov 17 '17 at 16:32
  • @user17219 that's a pity. tldp.org/HOWTO/Alsa-sound-6.html - this link shows that you may try just cat - ting the file to /dev/snd/pcm... device. – lukeg Nov 17 '17 at 23:19
  • I've attempted it but get some weird errors. I think the pcm devices have changed since that doc was written since now there are *p and *c variants. I've tried cating /dev/random into all of them but to no avail. – Josh Hansen Nov 21 '17 at 6:50

You could remove the SSD from your machine and dd/photorec/etc it on another working machine.

Here's the Dell service manual for the 9550 - see the section for "Removing the solid-state drive". Once you remove the drive, you can get an external M.2 USB enclosure and attach the disk to another machine to recover your data.

  • 3
    -1. In this case your advice may turn things so much worse. First of all, we cannot be sure which files were committed to the SSD, they may exist in the cache. Some files were not even saved at all: "But I still have files of interest open in an editor. I can't seem to save the files anywhere, but I can copy their contents." Next there's a risk the SSD won't initialize anymore; it works now because it initialized before the malfunction. Your solution means losing the RAM contents for sure and SSD content maybe. I'm afraid it would be coup de grâce. – Kamil Maciorowski Dec 1 '17 at 7:00

Long shot, but some distros have ham radio protocols built in. One common one is fldigi. Try which fldigi.

If you have this or another, they convert text into audio, typically using some variation of phase-shift keying. The output goes to your speakers/headphones, and is received by a stable computer in the same room, listing on its microphone. (This eliminates the ham radios which transmit and received the audio over the airwaves).

The QR codes approach could be good. You could fit as many as you can on the screen, and refresh them as soon as your terminal allows you. Then record the XPS' screen with a camera (as opposed to using a webcam), and just work on deciphering the video you've got saved. Might be helpful to use slow motion capture even.

Great show :) I have 1 suggestion: you can also read data from screen using a photo diode connected to the sound card of the other computer

  • Or wire the audio output to the input of some external sound card (to eliminate noise). Get inspired here: chdk.wikia.com/wiki/Obtaining_a_firmware_dump – Milo O'h Dec 2 '17 at 19:31
  • The question itself, and a couple of earlier answers, discuss capturing the screen images with a camera.   Would your approach be better?   Or even anywhere near as good?   If so, please explain.   Do not respond in comments; edit your answer to make it clearer and more complete. – G-Man Dec 3 '17 at 0:12

protected by Community Dec 2 '17 at 23:10

Thank you for your interest in this question. Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.