1

I am doing a bash script to backup my database daily(on Mac OS X) and I want to save the newest 7 backup directories. Here is the problematic fragment of my script(for listing everything but the newest directories):

# Assigning a value to a variable named NUMBEROFFOLDERS. 
NUMBEROFFOLDERS=7

# Command that leaves the newest NUMBEROFFOLDERS and removes everything else. 
TOBEDELETED=`(ls -t|head -n $(($NUMBEROFFOLDERS));ls)|sort|uniq -u|sed -e 's,.*,"&",g'|xargs rm -rfv`

# Executing the TOBEDELETED command. 
eval "$TOBEDELETED"

P.S. The command assigned in TOBEDELETED works when I enter it in a Terminal window, but when I run the script nothing happens.


EDIT: i) The script should list all directories in a specific destination twice differently; sort should parse them and only the newest directories will be taken off the list; sed should handle all spaces correctly and rm -rfv will delete all directories from the lists (directories to be deleted) with all their files.

ii) The script should leave the newest $NUMBEROFFOLDERS folders.

iii) I haven't received an error. I believe the issue is when I refer to the variable of the assigned command.

3
  • Please edit your question and explain exactly how your script fails. You need to tell us i) what it should do ii) what it is actually doing iii) what error messages, if any, are printed. Also, please explain what the ls -t... is supposed to be doing. You're running ls twice, and you're also attempting to parse the output of ls which is never a good idea. If you want to find the 5 newest folders, there are better ways. – terdon Mar 21 '15 at 15:48
  • It's not safe to parse ls command, especially when you're using rm -rfv. Good luck with your script, as I'm not going to test that:) It seems it won't handle the spaces correctly, so at least you should add -0 to your xargs and use find instead of ls. – kenorb Mar 21 '15 at 16:12
  • i) it should list all directories in a specific destination twice differently; sort should parse them and only the newest directories will be taken off the list; sed should handle all spaces correctly and rm -rfv will delete all directories from the lists (directories to be deleted) with all their files. ii) it should leave the newest $NUMBEROFFOLDERS folders. iii) I haven't received an error. I believe the issue is when I refer to the variable of the assigned command. @kenorb - thanks for spamming anyway. – Anton Todorov Mar 23 '15 at 12:08
1

A not so elegant working solution (check on your environment:
Let's start with a dividi and impera approach because it's more clear:

ListToSave=$(ls -dt */ |head -n $NUMBEROFFOLDERS| cut -f1 -d'/';)

This line store the list of folders to be saved in the variable ListToSave, maybe recyclable later in the script...

echo "$ListToSave"| awk 'BEGIN{printf("find . -maxdepth 1 -not -path \".\" ")}\
 {printf ("-not -path \"./"$0"\" ") } \
 END{print " -exec /bin/rm -rvf \"{}\" \\; "}' | /bin/sh

With this 3 line spanned command you construct a find command to delete all but the directories to be saved and you execute it piping it to a shell (in this case /bin/sh).
You can check the command construction wiping out the pipe to the shell | /bin/sh.
For security reasons it is always safer to run command with their full path. (better /bin/rm then rm, just to avoid some trojan )

Comment: It's not safe to parse the output of ls when you plan to do a script. It happens often that you start with a limited and under control situation; after some time, when you already forgot about it, you should try to use in a more complex case and less under control. When finally you are convinced that all works fine... it happens something and you realise it too late.

Notes:

  1. It's not safe to parse the output of ls... check here i.e.
  2. ls -dt */ | cut -f1 -d'/' should list only the directories and not even the files.
  3. xargs execute immediately the command line as you can read from its man page

    xargs - build and execute command lines from standard input


I strongly suggest to avoid to do what it follows.
If you want however to parse ls you can write as minimal modification from your previous attempt something similar to this

TOBEDELETED=`(ls -dt */ |head -n $NUMBEROFFOLDERS | cut -f1 -d'/'; ls)| sort | uniq -u | xargs echo "rm -rfv"`

in this way you will have the output stored in TOBEDELETED that you can execute later in the script as you write.

Note: it doesn't work with not kind file-names as spaces, \t,\n... and it can create unpredicted side effects. Imagine you have to delete the directory "1 bis" but not the directory "1" that it exists too. You will delete the directory "1" and you will have and error when it will try to delete the directory "bis" if this doesn't exist.

7
  • If any of the filenames have spaces, this is unsafe. – glenn jackman Mar 21 '15 at 17:26
  • 1
    I agree and I warn it twice. To parse ls is not safe not only for the presence of spaces but even for other characters as \t, \n, expansion glob characters... it's possible to give a look to any reference about why is not safe to parse the output of ls. – Hastur Mar 21 '15 at 19:11
  • Thank you, but the problem is that it does NOT run. However, I understand your suggestion and I agree with it. Now if I can only make it work when the script is being ran... – Anton Todorov Mar 23 '15 at 11:00
  • On my system It runs. Save the 3 parts (NUMBEROFFOLDERS=7, and the 2 other above) in a file; call it myscript.sh then try with mkdir tmp; cd tmp; mkdir -p 1 2 3 4 5 6 7 8 9 "1 bis" ; touch 1 2 3 4 5 6 8 ; /bin/bash ../myscript.sh . It should work removing 7 9 and "1 bis". If not try to wipe out ` | /bin/sh` from the lines that start with echo "$ListToSave" and see what it prints on the screen. Last check if the paths of the commands are different in your system. If you have an error post it in the answer... ps> you're welcome. – Hastur Mar 23 '15 at 11:32
  • Unfortunately even after all the times I tried it still does NOT work. Is there a way so I can contact you personally? – Anton Todorov Mar 24 '15 at 14:16
4

You're not doing what you think you're doing

# Command that leaves the newest NUMBEROFFOLDERS and removes everything else. 
TOBEDELETED=`(ls -t|head -n $(($NUMBEROFFOLDERS));ls)|sort|uniq -u|sed -e 's,.*,"&",g'|xargs rm -rfv`

The backticks mean the pipeline is actually executed, and the output of the rm command(s) are captured in the variable. This output does not contain commands, it's just the rm -v output.

I would simplify that commanda bit:

stat -f "%m %N" * | sort -nr | sed -e "1,${numfolders}d" | cut -d " " -f 2 | xargs rm -rfv

I don't have a BSD system to play with, so I'm not sure if the stat format string is correct.

2
  • Your command works, but my issue is with making it work in the script. – Anton Todorov Mar 23 '15 at 11:01
  • P.S. Is there a way so I can contact you in personally and share the entire source-code with you and hopefully get it fixed? – Anton Todorov Mar 23 '15 at 13:02
0

I add another answer with a test environment constructor. I created a shared executable project here.
Follow the link write in the shell bash main.sh. It works. After copy the script on you system and check if it works. If not post the error you receive.

#!/bin/bash
printf "\n\n\n# Let's we create your test directory\n" 
TESTDIR="tmptmp"
    rm -rf $TESTDIR
    mkdir $TESTDIR
    cd $TESTDIR
      printf "\n### Now we are in \n"
        pwd 
        mkdir -p 1 2 3 4 5 6 7 8 9 "1 bis";
      printf "\n### Here the list in the creation order\n"
         ls -dt */
         sleep 1.5s             # Take your time ...
         touch 1 2 3 4 5 6 8;   # Make those the last modified
      printf "\n### Here the list with modified order\n"
         ls -dt */

printf "\n\n# Let's go with your script \n\n"

#
# It follows your script. You can put it where you prefer but not in the 
# directory that you want to delete else you can delete it too...
# If you put in the upper directory you can run it with: 
# /bin/bash ../myscript.sh
#
# If you do one time `chmod u+x myscript.sh` you can execute it with
# ../myscript 
# Remember to put as a 1st line "#!/bin/bash"
# Note: check the path of all the executable... 
#

#!/bin/bash
NUMBEROFFOLDERS=7
ListToSave=$(ls -dt */ |head -n $NUMBEROFFOLDERS| cut -f1 -d'/';)
printf "This is the list of directories to be saved:\n%s\n" "$ListToSave"

printf "\n# That's the command that you'll execute piping it into a shell\n"
echo "$ListToSave" \
     | awk 'BEGIN{printf("find . -maxdepth 1 -not -path \".\" ")}\
     {printf ("-not -path \"./"$0"\" ") } \
     END{print " -exec /bin/rm -rvf \"{}\" \\; "}' 
  sleep 1 ; 

printf "\n# NOW we execute those commands with \"the power of piping\" \n\n"
 echo "$ListToSave" \
      | awk 'BEGIN{printf("find . -maxdepth 1 -not -path \".\" ")}\
 {printf ("-not -path \"./"$0"\" ") } \
 END{print " -exec /bin/rm -rvf \"{}\" \\; "}' | /bin/sh


printf "\n# That's the list of the survived as 'ls -d *' says to us\n\n"
ls -d *
printf "\n\n ### It worked like a charm ###\n\n"

exit 0

Note: if you execute it with bash -f main.sh it will not work because you will ask to bash to avoid the Pathname Expansion. Search about it in man bash.

5
  • Actually I would like to have all of the code in a single file, because it will be difficult to maintain after all. Even after I tried many times to implement edited version of your code here, I was still unable to make it work. – Anton Todorov Apr 2 '15 at 9:50
  • I get this error when I try to enter ListToSave to Terminal: head: option requires an argument -- n usage: head [-n lines | -c bytes] [file ...] ls: */: No such file or directory – Anton Todorov Apr 3 '15 at 10:59
  • The error from head means that the variable $NUMBEROFFILDERS is empty. In a Terminal from the same path from which you run the script, try to write NUMBEROFFOLDERS=7 and after ls -dt */ |head -n $NUMBEROFFOLDERS and see if there is an error (write if there is). We suppose that there are subdirectory in that point of your file system. BTW have you tried to follow this link and write bash main.sh in that ?shell. In a normal bash it seems to work fine. I'm sorry Anton a lot of work this week, I cannot be prompt in answers, but post however. – Hastur Apr 3 '15 at 11:22
  • I tried to directly replate the "$NUMBEROFFOLDERS" with 7 in the ls -dt */ |head -n $NUMBEROFFOLDERS , but the result was still the same. P.S. If I run the command in terminal (with admin account) all goes good and it gives the expected/desired results. Is it possible to be a matter of access rules? Also even if I run the script with "sudo sh databasebackup.sh", I am still receiving this error. – Anton Todorov May 8 '15 at 17:19
  • I am also able to print-out the value assigned in NUMBEROFFILDERS after the "ls -dt......" command and it is correct, so I guess this is not a problem. – Anton Todorov May 8 '15 at 17:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.