0

I have a folder with many sub-folders. I want to remove all smaller files from each subfolder, only leaving the biggest file.

For example:

Subfolder1
---------- File 1 ---- 300k
---------- File 2 ---- 299k
---------- File 3 ---- 800k

Only file 3 should remain the with 800k. If the folder only has one file, it remains.

This code works, but i can't put it in a for loop (for directory recursive):

find . -type f -maxdepth 1 | sort -n -r | tail -n +2 | xargs -I{} rm -v {}

How can I do this?

  • This command works, but i can't put it in for loop, doesnt works: find . -type f -maxdepth 1 | sort -n -r | tail -n +2 | xargs -I{} rm -v {} – Lordjohnny May 27 '17 at 16:13
  • What should the algorithm do when you got a "folder" containing <800k files only? – SΛLVΘ May 27 '17 at 16:18
  • 800k is a example, the script should delete all files except the largest for folder... – Lordjohnny May 27 '17 at 16:20
0
~$ tree -fQFi --sort=size pluto
"pluto"
"pluto/pluto1"/
"pluto/pluto1/pluto3"/
"pluto/pluto1/pluto3/nozero.txt"
"pluto/pluto1/pluto3/zero ed.txt"
"pluto/pluto1/nozero.txt"
"pluto/pluto2"/
"pluto/pluto2/nozero.txt"
"pluto/pluto2/nozer.txt"
"pluto/pluto2/zero.txt"
"pluto/pluto4"/
"pluto/pluto4/zeroed.txt"
"pluto/zeroed.txt"

4 directories, 8 files

~$ tree -fQFic --noreport --sort=size pluto | \
> awk -F"/" 'NR==1||/\/$/{next}; \
>     {path=""; for(i=1;i<NF;i++) path=path$i; if(a[path]++) print}'
"pluto/pluto1/pluto3/zero ed.txt"
"pluto/pluto2/nozer.txt"
"pluto/pluto2/zero.txt"

~$ tree -fQFic --noreport --sort=size pluto | \
> awk -F"/" 'NR==1||/\/$/{next}; \
>     {path=""; for(i=1;i<NF;i++) path=path$i; if(a[path]++) print}' | \
> xargs rm -v
'pluto/pluto1/pluto3/zero ed.txt' rimosso
'pluto/pluto2/nozer.txt' rimosso
'pluto/pluto2/zero.txt' rimosso

~$ tree -fQFi --sort=size pluto
"pluto"
"pluto/pluto1"/
"pluto/pluto1/pluto3"/
"pluto/pluto1/pluto3/nozero.txt"
"pluto/pluto1/nozero.txt"
"pluto/pluto2"/
"pluto/pluto2/nozero.txt"
"pluto/pluto4"/
"pluto/pluto4/zeroed.txt"
"pluto/zeroed.txt"

4 directories, 5 files

tree lists by directory, and then by descending size.

  • awk's 1st line of code skips tree's output 1st line or lines with trailing slashes (i.e. directories)
  • awk's 2st line of code builds a dirname from full path (for loop), then prints full path names iff dirname was encountered once on previous lines (i.e. it prints, for each directory, from 2nd listed file on)
  • 1
    Why not parse ls?. – Kamil Maciorowski May 28 '17 at 5:52
  • Could you please be more verbose? I quickly read that link, is your idea that ls outputs shouldn't be fed to other commands? – SΛLVΘ May 30 '17 at 3:26
  • It shouldn't. There's a link in the very first sentence of the question I linked to; its target is better than my verbose comment, whatever it would be. – Kamil Maciorowski May 30 '17 at 3:37
  • Edited, I am following your advice Kamil. Meta: pointing directly to the link you were referring to would have saved both your time and mine. – SΛLVΘ May 31 '17 at 11:19
  • That link is external, the site may disappear. Therefore I chose to point to SE question where the subject is covered. – Kamil Maciorowski May 31 '17 at 13:02
0

Justification

This is my attempt to build a command that will work with any directory and file name(s). In general paths in Linux (and names in filesystems) may contain any character but null (0x00) and /. Troublesome characters may be "" (space), any other white character, ', ", newline, other non-printable characters. Therefore it's important to:

  • abandon tools that replace some characters with others (e.g. many implementations of ls will print ? for non-printables);
  • pass all names as null-terminated strings (choose tools that can parse them);
  • quote properly.

I was inspired by the discussion under this other answer.


Actual commands

Testing version, it will only ls files that would be removed:

find -type d -exec sh -c 'find "$0" -maxdepth 1 -mindepth 1 -type f -exec stat --printf "%s %n\0" \{\} + | sort -znr | tail -zn +2' {} \; | cut -zf 2- -d " " | xargs -0r ls -l

Yes, I'm using ls here in spite of what I've just said. This is because ls output is not being parsed further. I'm using it only to display the result. If you happen to have directories or files with troublesome characters in their names then you will observe the behavior of ls which should convince you to never parse ls (unless you know you're absolutely safe with it). Still the troublesome names will pass all the way to ls and this is the point.

Understand the testing version (see down below for some explanation) and try it before you let the working version (just below) remove your files. Remember I'm just a random guy on the Internet.

Working version, it will remove your files:

find -type d -exec sh -c 'find "$0" -maxdepth 1 -mindepth 1 -type f -exec stat --printf "%s %n\0" \{\} + | sort -znr | tail -zn +2' {} \; | cut -zf 2- -d " " | xargs -0r rm

Explanation

Here's the testing version split into multiple lines (although it's still one line to bash; note I use this trick to inline comments):

find -type d -exec   `# Find all directories under (and including) the current one.` \
  sh -c '            `# In every directory separately...` \
    find "$0" -maxdepth 1 -mindepth 1 -type f -exec   `# ...find all files,...` \
      stat --printf "%s %n\0" \{\} + |   # ...get their sizes and names,...
    sort -znr |                          # ...sort by size...
    tail -zn +2'                        `# ...and discard the "biggest" entry.` \
    {} \
  \; |                                   # (All the directories have been processed).
cut -zf 2- -d " "  |                     # Then extract filenames...
xargs -0r ls -l                          # ...and ls them (rm in the working version).

Techniques used, obstacles overcome:

  • Tools that parse strings are told to work with null-terminated strings:
    • stat --printf "…\0";
    • sort -z, tail -z, cut -z;
    • xargs -0 …;
    • find -print0 (not needed in this example but very common in general, therefore I mention it anyway).
  • sh -c '…' is the way to use pipes inside find -exec.
  • find -type d -exec sh -c 'find "{}" … will break for directory name containing "; find -type d -exec sh -c 'find "$0" … ' {} \; works fine.
  • {} in the inner find statement are escaped (\{\}) to prevent the outer find substituting them.
  • cut could immediately follow tail, it would run one cut per directory. Placing it outside the outer find makes a single cut do all the cutting at once.
  • The -r option to xargs prevents ls (rm in the working version) from running when there's no input to xargs.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.