17

I'm building a new system with Linux installed for a specific purpose that involving a third party to interact with it. Apart from applying necessary firewall rules to isolate the machine from the intranet while still allowing access from/to the internet, I also need all the files on this system to persist (i.e I am able to retrieve them later) even when issuing rm, possibly as root, or other means (a program makes a systemcall to delete it). I don't want just the log of filesystem operations (which can be retrieved via tools like auditd and Tripwire I believe), I also need the files' contents. I have thought about several ways to achieve this:

  • aliasing rm to mv some-where-else, or
  • modifying the rm binary. While these two methods can work to some degree, it would miss if a program is trying to delete the files, because now the rm binary does not get called.
  • hooking the write and remove function in the kernel so that calls to write will write simultaneously to two locations and remove will only remove the "visible" copy. I think this is another feasible approach and not sure if it would cause any havoc. This involve writing a small kernel module. While I don't have much expertise in this, I can and will build one if there's source.
  • giving the users fakeroot. This likely won't work because they insist on having root access to install programs and modifying the systemwide configuration (e.g those in /etc and swap). They will know it the second they try to touch these files.
  • keep a mirror of the directory somewhere else i.e mirroring on the logical level. To the best of my knowledge, it can be done with tools like rclone. However, this could only work when the original process that's creating the file is done with it. I would assume that if a very large zip file is copied somewhere else into the system, only to extract a tiny text file out of it and then the zip file gets deleted, then it would be missed.
  • choose a filesystem that supports this level of operation or modify the filesystem configuration. Because I haven't yet created the system, I can make the decision of what filesystem to use. I always go with ext4 for normal installations, but I don't have any experience or know any configuration to modify regarding this scenario. I have spent sometime with squashfs and overlayfs before, and it does have a feature that I desire. When the original file is deleted, overlayfs only hides the file. If the overlayfs partition is deleted, the original file reaapears. However, I also want the content of new files before they are deleted.
  • actively monitor and creating references (softlinking?) to files so that rm the file will not make it dangled and subject to reclaim. In theory, I think this is pretty good, but I don't know if such a tool exists.
  • create a software RAID 1, but somehow disable delete function on one volume. I don't think this is possible given that RAID 1 operates on lower level than logical filesystems. However, I can afford to create more volumes if required.
  • Close off the system in question to a VM, and make snapshots of its disk. Giving the third party a VM is acceptable. But then, the VM is still lower level to the logical filesystem stored inside. I don't think there's a way for the host to know what exact filesystem operation is being carried out inside. Snapshotting by interval would be a bad idea, because it would take up a lot of storage and also miss files created and deleted in between the snapshot window.

Please correct me if there's something wrong in my understanding. Any insights would be much appreciated!

EDIT: I want to clarify that from the user's perspective:

  • I need rm to succeed, and binaries that might make systemcalls to delete the file like remove function to return 0.
  • ls does not show the file anymore after it has been "deleted".

However, I still want to be able to access the file after that. It could be stored somewhere else.

  • The original directory and filename should be kept if possible, the filename can be changed in arbitrary fashion in the event that an old file was deleted, and a new file of same name is written.
  • I could add more volumes to the system to accomplish this goal if needed.
8
  • 1
    Are you able to tolerate holes for files that are created and deleted very quickly (i.e: < 15 mins?) A filesystem that supports snapshots might be something to look into, like ZFS...
    – Attie
    Apr 21, 2023 at 12:08
  • I honestly have no idea about that, but everything is gonna be in excess so that might have some hole-tolerant nature.
    – tvc
    Apr 21, 2023 at 15:38
  • 3
    If the user truncates a file to 0 bytes with the truncate command, do you want to keep a copy of the original file?
    – pts
    Apr 22, 2023 at 15:18
  • 1
    Here is how to delete file b with the rename(2) system call instead of unlink(2) if file a exists: mv a b followed by mv b a. Do you want to keep a copy of b in this case?
    – pts
    Apr 22, 2023 at 15:22
  • 1
    Ideally yes -- but I guess that circumvention would rarely happen on purpose. I might not want to preserve b in order to make the solution a bit easier to implement. And zero byte files (afaik those like pid and lock files), I think they might not be that important to keep.
    – tvc
    Apr 23, 2023 at 3:11

7 Answers 7

17

For your requirements (continuous snapshotting, user with root access, hiding backup-action from user) the right choise of the filesystem should be the easiest way to realize your aim.

One keyword you have to keep in mind is "CoW" = "Copy on Write". This feature will duplicate data only at the moment of change. Copying a file creates a new pointer to the data only, so no new space is occupied (except by the meta data).

At OS-level a hardlink ln Path1/OrgFile Path2/OrgfileCopy would achieves a similar effect by linking to the same inode (to view inodes use ls -i). But if a software writes to the same file (the inode persists) also the copy will change content — That's no CoW and will not satisfy your expectations.

On a CoW aware filesystem a copy creates a new inode and the pointers of the new inode refer to the data-blocks of the original file. At the time when the file changes, the CoW-feature will copy the old data-blocks to a new location before the new version of the block is written. Like this only the changed blocks reserve new space. – CoW is the core idea of any snapshot, so real snapshots do not duplicate data (occupy new space) at creation time.

At this point using btrfs might be an option, especially as the snapshots can be transferred to other locations, even via ssh (see btrfs send and btrfs receive).

As you preference is a continuously but not time scheduled type of backup, btrfs might not be the choise No 1, as this is time-scheduled and you might loose some short living files.

Similar to btrfs is ZFS, when it comes to snapshots. ZFS' snapshots also represent a state in time but not a continuous description of the changing state. The advantages of ZFS play on other features, i.e. extremely large volumes or native encryption support, making it a favorite for NAS Systems. (Although encrypted on storage, you've to use the option -w to send a snapshot encrypted to its destination: zfs send -w ...)

The other solution is NILFS (better NILFS2).

"NILFS" = "New Implementation of a Log-structured File System" — The whole filesystem is organized like a logfile!

So no data will be overwritten until the end of the filesystem is reached. As its organization follows the logic of a ringbuffer a garbage collector will locate the oldest deletions to free space for new data.

As long as you are not filling the whole disksize with new data or newly created & deteted data by continiously writing/deleting garbage-data (or using a "wipe-disk"-tool), you can reconstruct deletions and changes. This also is true for data someone wrote several times with different data to the identical inode of interest just a few seconds ago.

It is possible to list the "checkpoints" by lscp then create a snapshot by mkcp -s (protecting this state from destruction by garbage collector for some time) and mount a snapshot, even while the current volume is active. — The snapshot can be synced i.e. by rsync to archive a state of interest on an other volume.

See also:
https://www.kernel.org/doc/Documentation/filesystems/nilfs2.txt


Annotation:

The NILFS development has been taken over by the Linux Kernel Team and is currently maintained (despite a possibly first impression when searching the internet).

Original repository of "Ryusuke Konishi" (founder of NILFS / almost inactive)
NILFS2 filesystem branch for Linux kernel (Fork from Linus Torvalds / maintained)
Real Homepage of NILFS: https://nilfs.sourceforge.io(beware of nilfs.org – That's a fake!)

6
  • Do you know if it's possible to have NILFS send its garbaged data out to a separate disk? I couldn't find anything but it would be a nice way to keep things really permanent.
    – pipe
    Apr 20, 2023 at 21:17
  • Thank you for this detailed answer. I guess I overlooked the prospects of content changes, but nevertheless NILFS2 looks promising. I'm just a bit worry about its current state, because some reported that it has problems with async IO. It also seems like it has been unmaintained for quite some time. I'm still open for other methods, though.
    – tvc
    Apr 21, 2023 at 3:47
  • 1
    Add ZFS to this answer (alongside btrfs) and you’ll get my upvote!
    – Josh
    Apr 21, 2023 at 12:24
  • 1
    Updated the answer // @Josh: Hopefully you're happy now. But you're right, ZFS should have its place in this list.
    – dodrg
    Apr 22, 2023 at 0:09
  • 1
    @pipe About such usage of NILFS' garbage collector I don't know. But it might be difficult to realize as the garbage to delete is not a valid state of the disk. - in contrast to a snapshot. I added a solution path to preserve old states to my answer.
    – dodrg
    Apr 22, 2023 at 0:18
10

Have you looked into GitFS? It looks like you need full auditing capabilities in your filesystem, which is a characteristic of a version control system. GitFS is a FUSE filesystem that will commit every change and push it to the remote.

Now versioning the entire filesystem may be overkill but you may mount only the directories of interest for that third party.

On a side note: if you have a trust issue with said third party, a technical solution may just not be appropriate for this particular situation.

4
  • Thanks for GitFS! It does indeed make sense, but I anticipated there would be some large files (potentially larger than common Git providers' limit which is 50MB I believe). Not sure how it would react, and the codebase feels unmaintained though
    – tvc
    Apr 21, 2023 at 3:13
  • 2
    Upvoted for that last sentence all by itself. There is a point where trust-issues go past the capabilities of a (reasonable) technical solution. At that point you need to ask yourself whether the time/effort invested in a purely technical solution (for initial setup AND future maintainability) is worth it.
    – Tonny
    Apr 21, 2023 at 12:04
  • 1
    I fully agree. In fact, the third party was hired by the upper management for their document management system. Me and my immediate supervisor can't have a say in that, but we have observed their behavior in other branches and felt that they lacked transparency. It worries me more when they can handle our internal documents, insists on an unreasonably huge server (16 cores) for a userbase not exceeding 100, and maintaining their presence as "customer service". We can do nothing but to launch the investigation, collect the evidence and report back.
    – tvc
    Apr 21, 2023 at 15:36
  • 2
    @tvc Large files shouldn’t be a problem if you yourself manage the remote on another machine. No need to have the underlying repository on an online service.
    – Seb
    Apr 21, 2023 at 20:40
5

My initial thought is to try Syncthing to provide a one-way (read only) replica of the target folder/s somewhere else, and enable versioning.

As a solution it avoids the need for low-level cleverness, but without trying it I am not sure how it would handle your zip file scenario.

https://syncthing.net/

4

Your rm aliasing won't work. It is still possible to type /bin/rm bypassing the rm alias.

Changing rm won't work too. touch dummy; mv dummy file_to_erase will replace your file with an empty file. perl -e 'unlink file will be handy too.

If you really want to go back if something is wrong, use backup/clone, snapshots, or overlay fs.

4
  • I agree -- aliasing or modifying rm still leaves a lot of other ways to delete a file. However, I'm not looking to rollback some changes. I want to investigate the actions and inspect the files the third party have transferred to our system.
    – tvc
    Apr 20, 2023 at 10:19
  • With a snapshot or clone, you can launch a diff command and seek modified/added/removed files. With an overlay fs, it is easy to look directly at the top layer. Apr 20, 2023 at 10:53
  • I understand. I do know some hypervisors offering incremental snapshots of disks so basically it's like diff'ed or an overlayfs already baked in. However, as I mentioned in the last point, there could be files which lifetime is shorter than the snapshot interval. Their contents will be missed. Even then, making snapshots frequently enough is expensive both in terms of performance and storage. I'm trying to achieve 100% persistence -- absolute zero missed files.
    – tvc
    Apr 20, 2023 at 11:10
  • 1
    Capturing temporary file creation and deletion will be tricky. But since the net result is null, it shouldn’t matter. There are two ways to trace system calls. A strace command with carefully chosen syscalls. Or should on a library level, patching fakeroot to log chosen syscalls. These won’t prevent to modify your system (then a snapshot will be handy), but you will have plenty of log to analyse. Apr 20, 2023 at 11:46
2

If you want to actually have all that the other answers suggest done automatically and transparently, by far the easiest way to implement it and have guarantees that it will enforce your rules, no matter what, would be to create your own very simple FUSE file system implementation.

Because all you would have to do, is:

  • Use your favorite language to write a trivial “empty” pass-through file system, where each function merely calls the same function for the original place in the file system. For many languages, an implementation of that already comes with the FUSE library implementation for the language, and so takes zero work. Here’s the original example from libfuse for C, which you could just use as-is.
  • Alter the necessary function(s), like unlink() or write(), to create a new version instead, using the solutions the other answers offered.
  • Mount the original file system in an unusual location, away from user access.
  • Mount your altered pass-through FUSE file system as the root file system.

Ideally, the last two steps are done in an Initramfs, just like when using other more unconventional file systems as the root file system.
It’s feasible that even implementing this in a simple language like Python will still be fast, given the simplicity of it. So even if you don’t know much about programming, this is absolutely doable. (And if you are a business, any programmer can implement this in C for you in minutes and for peanuts.)

The only trouble you will run into with this, is that your storage space will fill up. So you need a garbage collector. Ideally a daemon to push running it to idle times, and, very importantly, a hook in your FUSE file system, to trigger the garbage collector when an operation gets into trouble because there’s no space left. This would be the non-trivial part.

But when you truly want to to be transparent to your system, without making a mess of custom hacks that will break, sooner or later, there’s no way around this.
Anyway, the garbage collector could just be a simple shell script, and it would be fine. (For a first version, you can even implement it right in the FUSE file system code, and avoid creating a separate daemon to get it up and running.)


(Also, the last time I’ve tried using btrfs, and I ran into the fundamental design problem that it’s unable to know the actual size the different snapshots on the same volume take up. Because it’s implemented as multiple root directories. So you need to traverse an entire tree, to be able to know how much space it takes. Since knowing how much space it takes is crucial for this solution (and for multi-user servers in general), I can only recommend avoiding it as your base file system.)

3
  • If I'm not mistaken FUSE is about to fall into the third bullet point I listed above. I forgot filesystems could also be implemented in userspace, as is the case with tools like rclone mount or sshfs. I'm still in the process of trying out different options, but FUSE does worth trying out nevertheless.
    – tvc
    Apr 23, 2023 at 12:40
  • @tvc: Regarding the third bullet point: Yes and no. No, there is no need to mess with the kernel. But yes, it is a user space interface offered by the kernel (and then used by the library that you’d use. Just like anything else.) … I know my answer is kinda more hassle than you (and I) wanted. But it’s clean and correct. I guess the problem is that nobody wrote this file system yet. Otherwise I could just recommend that. :)
    – anon
    Apr 24, 2023 at 12:45
  • Fun fact: Android also uses such a solution to treat the user data storage in a special way.
    – anon
    Apr 24, 2023 at 12:47
0

A very simple solution is to cron an rsync -a command without the --delete option, thus deleted files will remain at their last version in the copy. Just add mount/umount around the rsync for security.

1
  • This is asynchronous and cannot catch quick changes though. Like, given a file betty.c, the change to evil in the command echo "$evilCode" > betty.c; gcc betty.c -o betty; echo "$goodCode" > betty.c; ./betty would not be caught.
    – anon
    Apr 24, 2023 at 12:52
0

It seems from the way you worded your question that you want to persist deleted files that you already own yourself rather than tracking or auditing filesystem activity from third-parties in your VM/physical machine.

NILFS could work but it would also track every file event for files that you don't own as well rather than only protecting your VM/physical machine from unwanted modifications.

In the Windows world we call this kind of software solutions 'freezer'. On Windows we have tools such as Deep Freeze Enterprise or Shadow Defender for this.

Freezing is extensively used in school environments for example and seems to match exactly what you want to do but for Linux instead.


I did some quick lookups and searching for 'freezer' brings results that could be of interest for you:

https://github.com/TheArqsz/deep-freezer

https://alternativeto.net/software/deep-freeze/?platform=linux

Such solutions would solve all the problems you mentioned:

  • file & directory deletes will succeed without using trap programs or syscall hooks.
  • deleted files & directory will no longer exist nor be usable anymore ntil the computer reboots.

And perhaps if you actually just want to share remote access to a native linux application but only from your 'work' machine (no personal machines) have you considered using something such as a 'Thin client' / 'Citrix server' kind of access control?

That's basically some kind of RDP but it only allows using a particular containerized application (think of it as a RDP/VNC server for 'VMWare ThinApp' applications for example).

Your third-parties will only be able to persist files that are related to the particular application that you're sharing with them: you would be doing this using isolation types known as Full, WriteCopy & Merge.

Full: totally isolated, cannot even see the real version of a file/folder from the host OS.

WriteCopy: you can read host OS files/folders but you can't modify them, it instead gets modified only in your sandbox (just like Sandboxie for Windows).

Merge: full access, no isolations, allows modifying host OS files.


If you want system protection & tracking/auditing, you could combine both:

  • first do the linux freezing part.

  • then try finding a fuse driver for SFTP/FTP.

  • at every boot, mount their home directory, and any directory they're allowed to modify from your SFTP/FTP server.

  • it will look like a real folder for them, but every file rename, delete, addition & modification has to pass through your SFTP/FTP server.

  • from there you can silently do your audit logging however you want, and they won't know about any of it other than having their personal folders mounted from a SFTP server.

  • this will be fully compatible with the linux freezing & rollbacks at reboot since they will get their files back from a separate SFTP/FTP server.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .