0

I have a file that has a listing of files/folders with the full paths. I'm looking for a way to be able to run thru this as if it was a filesystem.

I attached a small part of this file (the original is 33MB with over 460,000 lines)

./
./termux/
./termux/apt.tar
./termux/usr/
./termux/usr/lib/
./termux/usr/lib/libthread_db.so.1
./termux/usr/lib/libpython3.11.so.1.0
./termux/usr/lib/libgssapi_krb5.so.2.2
./termux/usr/lib/libTECkit.so
./termux/usr/lib/gnome-settings-daemon-3.0/
./termux/usr/lib/gnome-settings-daemon-3.0/gtk-modules/
./termux/usr/lib/gnome-settings-daemon-3.0/gtk-modules/at-spi2-atk.desktop
./termux/usr/lib/qt/
./termux/usr/lib/qt/mkspecs/
./termux/usr/lib/qt/mkspecs/integrity-armv7/
./termux/usr/lib/qt/mkspecs/integrity-armv7/qplatformdefs.h
./termux/usr/lib/qt/mkspecs/integrity-armv7/qmake.conf
./termux/usr/lib/qt/mkspecs/dummy/
./termux/usr/lib/gnome-settings-daemon-3.0/gtk-modules/canberra-gtk-module.desktop
...
./ubuntu-fs/
./ubuntu-fs/home/
...
./Other/
./Other/temp/
./Other/temp/Month.db
./Other/2021/
./Other/2021/January.txt
./Other/2021/February.txt
...

What I would like to do is something as follows

start with the root folder /

ls should give the listing of root

termux/
ubuntu-fs/
Other/
...

cd termux should take me into the termux folder ls now would list everything in the termux folder

apt.tar
usr/
...

And so on ...

I't doesn't need to work specifically with ls & cd. This is just to explain the idea I'm looking for.

10
  • 1
    What about prepending each line ending with a slash with mkdir and the rest with touch. Then run it as a script in a temp directory and you can use the shell itself to navigate the files.
    – choroba
    Commented Mar 10, 2023 at 14:13
  • Can you explain what the difference between your question and the output of find / would need to be? Commented Mar 10, 2023 at 14:23
  • @LjmDullaart find / would be looking at my current filesystem I;m lookin for something that would use the information from the file as if it is a filesystem
    – Sruly
    Commented Mar 10, 2023 at 16:23
  • @choroba This would mean creating a whole filesystem I don't want to create it just traces it from the file. The info in the file is from a backed up filesystem and I need from time to time to traves it and see some of the files and folder structure.
    – Sruly
    Commented Mar 10, 2023 at 16:26
  • I second @choroba suggestion with an optonal addtional step: create the the folders and empty files in a temp directory. Then, optionally, zip it into a compressed folder. Then you have a single file you can quickly extract when you want your "dummy" file system. Commented Mar 15, 2023 at 11:31

2 Answers 2

1

This answer will allow you to do the following:

  1. According to the listing, create a hierarchy of directories and regular files in a directory of your choice. This is itself a solution because you will be able to browse the hierarchy with cd+ls, mc or whatever.

  2. (Optionally) Using the hierarchy, create a SquashFS image for convenient mounting (even as a regular user) in the future.


1. Creating a hierarchy

Take this script:

#!/bin/sh -
file="$1"
dir="$2"

mkdir -p -- "$dir" || { echo >&2 'Cannot create directory.'; exit 1; }

<"$file" grep '^\./.*/$'    | tr '\n' '\0' | (cd "$dir" && xargs -0 mkdir -p --)
<"$file" grep '^\./.*[^/]$' | tr '\n' '\0' | (cd "$dir" && xargs -0 touch    --)

Save it as txt2dir, make it executable (chmod +x txt2dir) and use like this:

./txt2dir /path/to/listing /path/to/directory

/path/to/directory may or may not exist yet. It will be created if needed. Specifying a non-empty directory is possible, but probably you would want to use an empty or not-yet-existing directory.

Notes:

  • /path/to/directory may be /dev/shm/something for performance.

  • The listing is processed twice. Directories (from lines ending with /) are created first, regular files (from lines not ending with /) are created later. If in the listing some entry for a regular file precedes the entry for its directory, this will still work.

  • I assume that for every line not ending with / there exists a line ending with / that describes all directories in the path. In other words, when touch tries to create a regular file (e.g. ./foo/bar/baz), all directories it needs are already there because mkdir has created them (./foo/bar/ was in the listing).

    If this is not necessarily the case (i.e. if ./foo/bar/ may be missing), replace grep '…' in the line with mkdir with sed -n '\|^\./| s|/[^/]*$||p'. This variant will use all entries to create necessary directories (e.g. ./foo/bar/baz alone will be enough to create ./foo/bar/), so when touch starts, all directories will be there for sure. This will be at the cost of performance.

  • Your listing uses newline characters as delimiters, but I don't let them enter xargs. With tr I convert them to null characters, then I use xargs -0. xargs -0 is the best way to process entries verbatim, without treating backslashes and quotes specially; it's not portable though. Any portable solution has its quirks and/or downsides, I won't elaborate.

  • The regular expressions for grep (or sed) deliberately omit lines not starting with ./. This is in case some original pathname contained a newline character and generated more than one line in the listing. Example:

    ./foo/bar/dir with
    newline character/baz
    

    From the above pair only the first line will be processed. Unfortunate filenames will be truncated, unfortunate directories will appear as regular files; but at least there will be no misleading garbage in the top-level directory (newline character/baz, if created, would be such garbage).

    Since ultimately we use xargs -0, a better approach (but I'm not going to implement it) would be to detect such cases and deal with them, keeping newlines that belong to pathnames and converting others to null characters. Still pathnames with newline-dot-slash substring would deceive even this.

    Hopefully the hierarchy you created your listing from did not contain pathnames with newline characters.

And here you go. A browsable hierarchy is in /path/to/directory. When no longer needed, you can remove it with rm -r /path/to/directory. Keep the script and the listing and restore the hierarchy anytime; or proceed with SquashFS (keep reading).


2. Creating and using a SquashFS image

Having a hierarchy created with the above script, you can now create a SquashFS image. If the hierarchy is in /path/to/directory, the command to create an image is:

mksquashfs /path/to/directory image.sqfs

Now you can mount the filesystem:

  • as root with

     mount -t squashfs image.sqfs /path/to/mountpoint
    
  • or as a regular user with

     squashfuse image.sqfs /path/to/mountpoint
    

    (this requires FUSE to be supported and enabled in your OS).

And now you can browse /path/to/mountpoint. The hierarchy in /path/to/directory is no longer needed. Keep the image and mount/unmount it at will.

Notes:


Addendum

This is the command I used to create a listing that resembles your listing:

( cd /path/to/hierarchy && find . ! -type d -print -o -exec printf '%s/\n' {} \; ) > listing

I also used it for creating listings from hierarchies created from the original listing. This and diff <(sort listing) <(sort listing2) in Bash allowed me to test if my script works well.

7
  • Your first option worked great. I haven't had a chance to try squashfuse. Would you be able to explain the last 2 lines of the script what each part does, I would like to learn from it.
    – Sruly
    Commented Mar 20, 2023 at 0:34
  • @Sruly I don't know what you know, I don't know what you don't know. To fully explain, I would need to elaborate on <, shell variables, quoting in a shell, grep with regular expressions (including at least wildcards, some anchors, escaping, bracket expressions), pipelines, tr with \X notation, subshells, &&, xargs, why xargs -0 is good, mkdir -p, touch, double-dash. This is not how this site is supposed to work; each topic should be explained under its own specific question. Are there parts you understand? Are there parts you totally don't understand? Commented Mar 20, 2023 at 13:47
  • I appreciate any help you can give. The main part I need help with is the first part <"$file" How does the < work. Also how does it know where to create the file (or Folder) in this part (cd "$dir" && xargs -0 mkdir -p --)
    – Sruly
    Commented Mar 20, 2023 at 16:09
  • 1
    @Sruly For < see this, this, maybe this. Paths in your file are relative (they start with ./) and my greps deliberately filter out other paths. What mkdir -p or touch get as arguments are these relative paths, but they run after the subshell I deliberately created successfully (&&) performs cd to $dir. This is how I set working directory for mkdir or touch to $dir. Commented Mar 20, 2023 at 17:01
  • Thanks for the help. I just want to clarify one thing. You mention subshell I deliberately created, is the subshell created by the parentheses ( )?
    – Sruly
    Commented Mar 21, 2023 at 12:31
0

Here is a proof of concept of a simple Perl program that does what you need (but I didn't handle all the possible edge cases). Give it the input file name as an argument.

#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };

sub dir :lvalue {
    my ($fs, @dirs) = @_;
    return '.' unless @dirs;
    return $fs->{ $dirs[0] } if 1 == @dirs;
    return dir($fs->{ $dirs[0] }, @dirs[1 .. $#dirs])
}

my %fs;
while (<>) {
    chomp;
    my $is_dir = m{/$};
    my @dirs = split m{/};
    my $last = pop @dirs;
    if ($is_dir) {
        dir(\%fs, @dirs, $last) = {};
    } else {
        dir(\%fs, @dirs)->{$last} = undef;
    }
}

my $current = '.';

sub ls {
    my @args = @_;
    @args = ("") unless @args;
    for my $arg (@args) {
        if ($arg !~ m{^/}) {
            substr $arg, 0, 0, "$current/";
        }
        my @dirs = updirs(split m{/}, $arg);
        for my $member (sort keys %{ dir(\%fs, @dirs) }) {
            print $member;
            say dir(\%fs, @dirs)->{$member} ? '/' : "";
        }
    }
}

sub cd {
    my @args = @_;
    warn "One argument expected.\n" unless 1 == @args;
    my @dirs = updirs(split m{/},
                      ($args[0] =~ m{^/}) ? $args[0] : "$current/$args[0]");
    if (ref dir(\%fs, @dirs)) {
        $current = join '/', @dirs;
    } else {
        warn "Non-existent path.\n";
    }
}

my %dispatch = (ls   => \&ls,
                cd   => \&cd,
                exit => sub { exit });
while (print({*STDERR} "$current: "), $_ = <STDIN>) {
    chomp;
    my ($command, @args) = split;
    warn "Unknown command '$command'.\n"
        unless exists $dispatch{$command};
    $dispatch{$command}->(@args);
}

sub updirs {
    my @dirs = @_;
    shift @dirs if "" eq $dirs[0];
    for (my $i = 0; $i <= $#dirs; ++$i) {
        if ($dirs[$i] eq '..') {
            if ($i > 0) {
                splice @dirs, $i - 1, 2;
                $i -= 2;
            } else {
                splice @dirs, 0, 1;
                $i = -1;
            }
        }
    }
    return @dirs
}
8
  • I'm not familiar at all with perl so I can't do much with this example. There is no indication which files are files and which are folders. (In my post you can see the folders end with the folders mark /)
    – Sruly
    Commented Mar 16, 2023 at 23:28
  • Updated to mark directories.
    – choroba
    Commented Mar 17, 2023 at 21:38
  • I'm getting this error Can't return undef from lvalue subroutine at ./test-fs line 10, <> line 1.
    – Sruly
    Commented Mar 27, 2023 at 20:28
  • When do you get the error? At the start of the script or after entering some commands - in such a case, which ones? Have you specified the input file as an argument?
    – choroba
    Commented Mar 28, 2023 at 7:56
  • At the start of the script
    – Sruly
    Commented Mar 29, 2023 at 19:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .