11

I kind of screwed up the backup of my personal directory with rsync (maybe because I'm backuping on an NTFS filesystem): all files are here but all files and directories access rights are 777. I was wondering if there was a magic utility that would recursively change:

  • the directories from 777 to 755.
  • the regular files from 777 to 644. I don't have many executables in my home, so I can manage that by hand later.
  • leave other files (links, any thing else ?) unchanged.

Doing it in shell is easy, but it will take hours...

Subsidiary question: any advice to backup properly a linux directory hierarchy on NTFS (with rsyncor other).

4
  • How big is that directory? Why would it take hours from the shell? The magic utilities you are asking for are exactly in the shell..
    – Fanatique
    Jul 20, 2019 at 10:13
  • What is a Linux hierarchy? Are you talking about WSL? If so, note that there's a lot more metadata than just the file permissions that you might need to worry about, especially if the directory is on LXFS rather than DrvFS. Including stuff like per-directory case sensitivity.
    – user541686
    Jul 20, 2019 at 22:50
  • "Subsidiary question: any advice to backup properly a linux directory hierarchy on NTFS (with rsyncor other)." – You should ask your question as a question, but the short answer on how to backup properly a linux directory hierarchy on NTFS is to not backup a linux directory hierarchy on NTFS. NTFS is designed for Windows, and while it technically does support (most if not all of) the metadata required to back up Linux files, a simple copy or rsync will not use the special APIs required to maintain those. Use pax, that's what it's for. Jul 21, 2019 at 12:22
  • For archival purposes use "tar" to pack up a directory. This will not give you rsync update speeds but will maintain all unix attributes. Jul 22, 2019 at 15:22

4 Answers 4

16

The standard recommended solution is straight-forward:

find . -type d -exec chmod 0755 "{}" \+
find . -type f -exec chmod 0644 "{}" \+

This will append as many filenames as possible as arguments to a single command, up to the system's maximum command line length. If the line exceeds this length, the command will be called multiple times.

If you want to call the command once per file, you can instead do:

find . -type d -exec chmod 0755 "{}" \;
find . -type f -exec chmod 0644 "{}" \;
7
  • 1
    Actually, I see that @Fanatique's way is faster because it involves fewer calls to chmod thanks to argument appending. Your answer can be improved by changing \; to \+, which will also enable argument appending.
    – Deltik
    Jul 20, 2019 at 10:48
  • @Deltik: I saw your edit, but was in the process of doing mine.
    – harrymc
    Jul 20, 2019 at 11:10
  • 5
    The \; way is so slow that my benchmark of it has been running for over 5 minutes compared to about 14 seconds with \+. I would recommend removing \; from your answer.
    – Deltik
    Jul 20, 2019 at 11:19
  • 3
    @Deltik: I still want the answer to be comprehensive, but I made it clear that this is the preferred method for many files.
    – harrymc
    Jul 20, 2019 at 12:32
  • 5
    + is not a shell special; there's no need to escape it. Jul 21, 2019 at 3:03
16

chmod -R a=,u+rwX,go+rX $DIR seems to work fine, and is very likely to be the fastest, however you look at it.

(I checked with strace, and it makes only one fchmodat() syscall per file/directory -- for files that is with 644 and for directories with 755).

The trick is the X permmission, documented in man chmod, which acts like x for directories only -- the very distinction you wanted.

What is not documented is my guess that they would be applied in the same sequence as they are specified, and not just in some random order, but repeated tests with several variants have convinced me that they do indeed run in the order given, so I am pretty sure this is always going to work like this.

I should mention this is on Linux, though a cursory reading of the BSD manpage for chmod suggest that it should work there also.

4
  • This way is indeed the fastest, as shown in my answer with updated benchmarks.
    – Deltik
    Jul 20, 2019 at 14:15
  • Update: I found a ≈40% faster method that makes use of task parallelism!
    – Deltik
    Jul 21, 2019 at 19:08
  • about the parallelism: is this on an SSD or some very fast disk? Does the benchmark tool run a sync afterward to ensure everything is written out to disk?
    – user1003916
    Jul 26, 2019 at 13:16
  • I ran the tests on a NVMe SSD in my answer, but I was able to get similar results on two machines (1, 2) with HDDs. sync didn't seem to make a difference. I did not run the tests on just one plain hard drive, though, because I don't have any machines with just a hard drive at the moment. (More info)
    – Deltik
    Jul 26, 2019 at 13:32
10

I benchmarked sitaram's answer, Peter Cordes's comment, Fanatique's answer, and harrymc's answer, but this answer has the fastest way.

Averages:

  • Deltik's answer* – 7.480 seconds
    * Credit to Peter Cordes for suggesting parallelism
  • sitaram's answer – 12.962 seconds (73.275% slower than best)
  • Peter Cordes's comment – 14.414 seconds (92.685% slower than best)
  • Fanatique's answer – 14.570 seconds (94.772% slower than best)
  • harrymc's updated answer – 14.791 seconds (97.730% slower than best)
  • harrymc's original answer – 1061.926 seconds (14096.113% slower than best)

Full statistical summary:

Author              N      min     q1      median  q3      max     mean    stddev
------------------  --     ------- ------- ------- ------- ------- ------- --------
Deltik              10     7.121   7.3585  7.4615  7.558   8.005   7.4804  0.248965
sitaram             10     12.651  12.803  12.943  13.0685 13.586  12.9617 0.276589
Peter Cordes        10     14.096  14.2875 14.375  14.4495 15.101  14.4136 0.269732
Fanatique           10     14.219  14.512  14.5615 14.6525 14.892  14.5697 0.211788
harrymc (updated)   10     14.38   14.677  14.8595 14.9025 15.119  14.791  0.21817
harrymc (original)  1      1061.93 1061.93 1061.93 1061.93 1061.93 1061.93 N/A

Deltik's command, in benchmark format:

find "$(pwd)" -type d -print0 | xargs -0 -P4 chmod 755 & \
find "$(pwd)" -type f -print0 | xargs -0 -P4 chmod 644 & wait

sitaram's command, in benchmark format:

chmod -R a=,u+rwX,go+rX "$(pwd)"

Peter Cordes's command, in benchmark format:

find "$(pwd)" \( -type d -exec chmod 755 {} + \) \
           -o \( -type f -exec chmod 644 {} + \)

Fanatique's command, in benchmark format:

find "$(pwd)" -type d -print0 | xargs -0 chmod 755 ; \
find "$(pwd)" -type f -print0 | xargs -0 chmod 644

harrymc's updated command, in benchmark format:

find "$(pwd)" -type d -exec chmod 755 {} + ; \
find "$(pwd)" -type f -exec chmod 644 {} +

harrymc's original command, in benchmark format:

find "$(pwd)" -type d -exec chmod 755 {} \; ; \
find "$(pwd)" -type f -exec chmod 644 {} \;

My command was the fastest thanks to the four parallel chmod processes per file type. This allowed multiple CPU cores to run chmod, which moves the bottleneck towards kernel I/O threads or the disk.

sitaram's command was the runner-up because everything is done within the chmod command. This substantially reduces overhead compared to the other answers because:

  • The files only need to be scanned once (similar to doing one find instead of two), and
  • No child processes need to be created.

This command is the least flexible, however, because it relies on a trick involving the differing meaning of the executable bit between regular files and directories.

Peter Cordes's comment, which uses one find command, prevents double lookups of directory entries. The more files there are, the more substantial this improvement is. It still has the overhead of creating child chmod processes, which is why it is quite a bit slower than the chmod-only solution.

Between Fanatique's command and harrymc's updated command, find piped into xargs (find | xargs) was faster because the stream of results is processed asynchronously. Instead of find pausing its find behavior for -exec, the found results are sent to xargs for concurrent processing.
(The null byte delimiter (find -print0 | xargs -0) did not seem to affect running time.)

harrymc's original command was too slow because of the overhead of a new chmod command for every single file and folder, each executed in sequence.


In the test setup, there were 1000002 regular files contained within 1001 directories:

root@demo:~# echo {0..999} | xargs mkdir -p
root@demo:~# find -type d -exec bash -c "cd {}; echo {0..999} | xargs touch" \;
root@demo:~# find | wc -l
1001003
root@demo:~# find -type d | wc -l
1001
root@demo:~# find -type f | wc -l
1000002

I set all of the files and folders to have 777 permissions, like the initial conditions of the question.

Then, I benchmarked the commands ten times, each time restoring the permissions to 777 with chmod -R 0777 "$(pwd)" before running the test.

With OUTPUT representing a file that contains the output of each benchmark command, I calculated the average time using:

bc <<< "scale=3; ($(grep real OUTPUT | grep -Po '(?<=m).*(?=s)' | xargs | sed 's/ /+/g'))/10"

Results of the benchmark of Deltik's answer

root@demo:~# for i in {0..9} ; do chmod -R 0777 "$(pwd)" ; time { find "$(pwd)" -type d -print0 | xargs -0 -P4 chmod 755 & find "$(pwd)" -type f -print0 | xargs -0 -P4 chmod 644 & wait ; } ; done
[1] 9791
[2] 9793
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.634s
user    0m2.536s
sys     0m23.384s
[1] 9906
[2] 9908
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.443s
user    0m2.636s
sys     0m23.106s
[1] 10021
[2] 10023
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m8.005s
user    0m2.672s
sys     0m24.557s
[1] 10136
[2] 10138
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.480s
user    0m2.541s
sys     0m23.699s
[1] 10251
[2] 10253
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.397s
user    0m2.558s
sys     0m23.583s
[1] 10366
[2] 10368
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.482s
user    0m2.601s
sys     0m23.728s
[1] 10481
[2] 10483
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.679s
user    0m2.749s
sys     0m23.395s
[1] 10596
[2] 10598
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.243s
user    0m2.583s
sys     0m23.400s
[1] 10729
[2] 10731
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.320s
user    0m2.640s
sys     0m23.403s
[1] 10844
[2] 10847
[1]-  Done                    find "$(pwd)" -type d | xargs -P4 chmod 755
[2]+  Done                    find "$(pwd)" -type f | xargs -P4 chmod 644

real    0m7.121s
user    0m2.490s
sys     0m22.943s

Average time: 7.480 seconds

Results of the benchmark of sitaram's answer

root@demo:~# for i in {0..9} ; do chmod -R 0777 "$(pwd)" ; time chmod -R a=,u+rwX,go+rX "$(pwd)" ; done

real    0m12.860s
user    0m0.940s
sys     0m11.725s

real    0m13.059s
user    0m0.896s
sys     0m11.937s

real    0m12.819s
user    0m0.945s
sys     0m11.706s

real    0m13.078s
user    0m0.855s
sys     0m12.000s

real    0m12.653s
user    0m0.856s
sys     0m11.667s

real    0m12.787s
user    0m0.820s
sys     0m11.834s

real    0m12.651s
user    0m0.916s
sys     0m11.578s

real    0m13.098s
user    0m0.939s
sys     0m12.004s

real    0m13.586s
user    0m1.024s
sys     0m12.372s

real    0m13.026s
user    0m0.976s
sys     0m11.910s

Average time: 12.962 seconds

Results of the benchmark of Peter Cordes's comment

root@demo:~# for i in {0..9} ; do chmod -R 0777 "$(pwd)" ; time find "$(pwd)" \( -type d -exec chmod 755 {} + \) -o \( -type f -exec chmod 644 {} + \) ; done

real    0m14.096s
user    0m1.455s
sys     0m12.456s

real    0m14.492s
user    0m1.398s
sys     0m12.897s

real    0m14.309s
user    0m1.518s
sys     0m12.576s

real    0m14.451s
user    0m1.477s
sys     0m12.776s

real    0m15.101s
user    0m1.554s
sys     0m13.378s

real    0m14.223s
user    0m1.470s
sys     0m12.560s

real    0m14.266s
user    0m1.459s
sys     0m12.609s

real    0m14.357s
user    0m1.415s
sys     0m12.733s

real    0m14.393s
user    0m1.404s
sys     0m12.830s

real    0m14.448s
user    0m1.492s
sys     0m12.717s

Average time: 14.414 seconds

Results of the benchmark of Fanatique's answer

root@demo:~# for i in {0..9} ; do chmod -R 0777 "$(pwd)" ; time { find "$(pwd)" -type d -print0 | xargs -0 chmod 755 ; find "$(pwd)" -type f -print0 | xargs -0 chmod 644 ; } ; done

real    0m14.561s
user    0m1.991s
sys     0m13.343s

real    0m14.521s
user    0m1.958s
sys     0m13.352s

real    0m14.696s
user    0m1.967s
sys     0m13.463s

real    0m14.562s
user    0m1.875s
sys     0m13.400s

real    0m14.609s
user    0m1.841s
sys     0m13.533s

real    0m14.892s
user    0m2.050s
sys     0m13.630s

real    0m14.291s
user    0m1.885s
sys     0m13.182s

real    0m14.843s
user    0m2.066s
sys     0m13.578s

real    0m14.219s
user    0m1.837s
sys     0m13.145s

real    0m14.503s
user    0m1.803s
sys     0m13.419s

Average time: 14.570 seconds

Results of the benchmark of harrymc's updated answer

root@demo:~# for i in {0..9} ; do chmod -R 0777 "$(pwd)" ; time { find "$(pwd)" -type d -exec chmod 755 {} + ; find "$(pwd)" -type f -exec chmod 644 {} + ; } ; done

real    0m14.975s
user    0m1.728s
sys     0m13.050s

real    0m14.710s
user    0m1.586s
sys     0m12.979s

real    0m14.644s
user    0m1.641s
sys     0m12.872s

real    0m14.927s
user    0m1.706s
sys     0m13.036s

real    0m14.867s
user    0m1.597s
sys     0m13.086s

real    0m15.119s
user    0m1.666s
sys     0m13.259s

real    0m14.878s
user    0m1.590s
sys     0m13.098s

real    0m14.852s
user    0m1.681s
sys     0m13.045s

real    0m14.380s
user    0m1.603s
sys     0m12.663s

real    0m14.558s
user    0m1.514s
sys     0m12.899s

Average time: 14.791 seconds

Results of the benchmark of harrymc's original answer

Due to how slow this command was, I only ran the benchmark once.

root@demo:~# for i in {0..0} ; do chmod -R 0777 "$(pwd)" ; time { find "$(pwd)" -type d -exec chmod 755 {} \; ; find "$(pwd)" -type f -exec chmod 644 {} \; ; } ; done
 
real    17m41.926s
user    12m26.896s
sys     4m58.332s

Time taken: 1061.926 seconds

19
  • 2
    +1 : Good and comprehensive analysis.
    – harrymc
    Jul 20, 2019 at 18:34
  • 1
    @JoL: I used "$(pwd)" to simulate longer paths in the argument list. Performance would be a bit better with . because more paths could fit in before exceeding the maximum argument length.
    – Deltik
    Jul 20, 2019 at 18:55
  • 1
    @PeterCordes: Both find -exec and find | xargs split the same number of times. I think find | xargs is a bit faster because the two commands work in parallel. With find -exec, the file search pauses while the -exec command runs. (That's what I understood from the strace outputs, which were too big to paste.)
    – Deltik
    Jul 21, 2019 at 15:57
  • 1
    Oh yes, parallelism would do it. I even thought of that at one point but forgot to comment about it >.<. You can run both find|xargs commands in parallel, too. Use & instead ; to separate them. They probably don't spend much of their time holding locks on directory entries the other process needs to traverse so they should overlap nicely. Jul 21, 2019 at 16:46
  • 1
    @PeterCordes: Brilliant idea! Using Bash jobs (&) and xargs -P4 sped things up by about 42.667%. Revision 9 of my answer adds the benchmarks for the resulting parallel chmod command.
    – Deltik
    Jul 21, 2019 at 19:12
3

If the directories are way too large and contain way too many files, the original way that @harrymc has shown you will fail.

If you have too many files, you'll need to pipe find to xargs with chmod:

find /base/dir -type d -print0 | xargs -0 chmod 755 
find /base/dir -type f -print0 | xargs -0 chmod 644
4
  • 2
    Your way is actually faster than @harrymc's, but the reason is not correct. The reason is that the invocations of chmod are reduced because xargs automatically stacks arguments without exceeding the shell limit. @harrymc's way runs chmod on every single file and directory.
    – Deltik
    Jul 20, 2019 at 10:46
  • I was thinking about a recursive shell doing test -dand test -f and then chmodaccordingly. I just didn't think about find. Thanks @Fanatique and @harrymc
    – mszmurlo
    Jul 20, 2019 at 11:06
  • @mszmurlo: I benchmarked the two answers and found that @Fanatique's answer is faster.
    – Deltik
    Jul 20, 2019 at 11:08
  • 2
    Why do you claim harrymc's answer would fail with too many files? I think that's an incorrect claim. Jul 21, 2019 at 3:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .