$ cat important_file > /dev/null &
[1] 9711
$ rm important_file 
$ killall -STOP cat

[1]+  Stopped                 cat important_file > /tmp/p
$ ls -l /proc/`pidof cat`/fd/
total 0
lrwx------ 1 vi vi 64 May 13 20:32 0 -> /dev/pts/29
l-wx------ 1 vi vi 64 May 13 20:32 1 -> /tmp/p
lrwx------ 1 vi vi 64 May 13 20:32 2 -> /dev/pts/29
lr-x------ 1 vi vi 64 May 13 20:32 3 -> /home/vi/important_file (deleted)

How to recover this important_file?

I tried something like

injcode -m dup2 -ofd=3 -ofilename=/tmp/recovered_file -oflags=O_CREAT $PID_OF_CAT

but it does nothing.


If /home is NFS, there will be a .nfsNNNNNNNNNN file in /home/vi that you can access/copy. If home is a local filesystem, you should be able to do the same thing via the /proc/PID/fd/3 link:

cp /proc/PID/fd/3 /tmp/recovered_file

If you want to actually undelete the file, here's a blog post on the subject.

  • 2
    OK, I was confused by that readlink /proc/13381/fd/3 -> "/home/vi/important_file (deleted)" and /home/vi/important_file\ \(deleted\) obviously does not exist.
    – Vi.
    May 14 '11 at 1:50
  • lmao! I love you! I was examining a particular vmlinux in gdb for a very particular stack trace for a mysterious race condition, forgot about that and rebuilt the kernel with different .config options. Sep 16 at 7:08

... better than copying at a given time (and gathering only that time's snapshot of the file's content) is to "tail -f" that file into a new file:

tail -c +0 -f /proc/PIDofProgram>/fd/# > /new/path/to/file

(thanks to tail's cautious programmers, that will even work with binary output.)

During its runtime, the tail -f itself keeps the file open, safely preventing it from being purged off disk when the original program ends. Thus, don't stop the tail -f immediately after your original program ends - check the tail'ed /new/path/to/file first whether it is what you want. If it isn't (or is unsatisfying for any other reason), you can copy the original file again, but this time after all writing to it has finished by "Program" and from the still-running tail -f's /proc/PIDoftail/fd/ directory.

  • 4
    What about creating a hardlink to /proc/PIDofProgram>/fd/# ?
    – becko
    Nov 14 '15 at 22:26
  • 3
    @becko Invalid cross-device link. Apr 27 '17 at 14:18

Use lsof to find the inode number, and debugfs to recreate a hard link to it. For example:

# lsof -p 12345 | grep /var/log/messages
syslogd 12345 root    3w   REG                8,3    3000    987654 /var/log/messages (deleted)
# mount | grep var
/dev/sda2 on /var type ext3 (rw)
# debugfs -w /dev/sda2
debugfs: cd log
debugfs: ln <987654> tmp
debugfs: mi tmp
                      Mode    [0100600] 
                   User ID    [0] 
                  Group ID    [0] 
                      Size    [3181271] 
             Creation time    [1375916400] 
         Modification time    [1375916322] 
               Access time    [1375939901]
             Deletion time    [9601027] 0
                Link count    [0] 1
               Block count    [6232] 
                File flags    [0x0] 
debugfs:  q
# mv /var/log/tmp /var/log/messages
# ls -al /var/log/messages
-rw------- 0 root root 3301 Aug  8 10:10 /var/log/messages

Before you complain, I faked the above transcript as I don't have a deleted file to hand right now ;-)

I use mi to reset the delete time and link count to sensible values (0 and 1 respectively), but it doesn't work properly - you can see the link count remains at zero in ls. I think the kernel might be caching the inode data. You should probably fsck at the earliest opportunity after using debugfs, to be on the safe side.

In my experience, you should create the link using a temporary file name and then rename to the proper name. Linking it directly to the original file name seems to cause directory corruption. YMMV!

  • 1
    Why exactly are you suggesting this if it doesn't really work properly and causes system damage? I think you should have a more vivid disclaimer within the answer that this is simply a WiP, and should not really be tried in production.
    – cnst
    Sep 4 '14 at 16:25

You can just cp the file, i.e:

cp /proc/<pid>/fd/<fdno> /new/path/to/file

Of course, if the file is still being modified, you'll run into trouble with this approach.

  • On my system (Xubuntu 18.04.2), /proc/<pid>/fd/<fdno> is just an orphaned symbolic link: cp: cannot open '/proc/<pid>/fd/<fdno>' for reading: No such file or directory
    – db-inf
    Jan 31 '20 at 8:33

If your file is still being written to (so you cant use cp), and you're not using ext (so you cant use debugfs) here's an agnostic way of doing it.

Once you find the process using ps aux | grep or however, and you've double checked the removed filename via lsof -p $pid -a -d $fd you can use the following spawn a new process whose sole job is to revive it:

(tail -f -n +1 --pid $pid /proc/$pid/fd/$fd) >$file </dev/null 2>&1 &

The weird wrapping is just to safeguard against HUP, so feel free to not use it or swap it out for nohup

  • I remember sometimes /proc/.../fd/... being just unopenable for deleted files. I don't remember details like FS or original file open mode.
    – Vi.
    Aug 27 '20 at 14:42
  • I had a long-running nohupped script piping into gzip and I stupidly renamed a file to where it was going yesterday. Doing this it successfully pulled the file back, but maybe it was because it's fd1 specifically and gzip wasnt killed
    – Hashbrown
    Aug 27 '20 at 23:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.