I would like to use AWK to remove the first 8 columns from the following output:

ls -l

lrwxrwxrwx 1 user user   23 jul 27 00:04 file1.pdf
-rw-rw-r-- 1 user user  107 may  8 13:59 file 2 with spaces.mp3
lrwxrwxrwx 1 user user   11 jul 24 19:43 file3-with-hyphens.txt
lrwxrwxrwx 1 user user   11 jul 24 19:43 and_another_file4_with_underscores.md
-rw-rw-r-- 1 user user  107 may  8 13:59 file 5 with way more spaces than the rest.mp3

and send the result to a text file. I can do that manually in vim with visual block select, however I would prefer to have a script to do it automatically.

Looking around I was able to find this page where, by changing the relevant parts of the 9th example and piping the output to itself as many times as required, I was able to get the desired result, but I feel that there must be a better (more elegant and/or compact) way to do it but haven't been able to find it or come up with my own.

My final code is the following:

awk '{for(i=1;i<=NF;i++)if(i!=x)f=f?f FS $i:$i;print f;f=""}' x=1 ~/file_folder_content.txt | awk '{for(i=1;i<=NF;i++)if(i!=x)f=f?f FS $i:$i;print f;f=""}' x=1 | ... | awk '{for(i=1;i<=NF;i++)if(i!=x)f=f?f FS $i:$i;print f;f=""}' x=1 >> ~/file_folder_content.txt

Note 1: I expect the actual output from the ls command to be much larger than this.

Note 2: I tried to print the 9th field, but since some file names contain spaces, this only prints the first word in the file name.

  • 1
    Remove the first 8 columns? That output only has 8 columns. Are you trying to just print the filenames? ls -1 does that. – confetti Aug 31 '18 at 5:55
  • Trial and error left me with this ridiculous command: ls -l | tail -n +2 | column -tdN test -o "|" | cut -d "|" -f 9 - It prints only the 9th column. There must be a better way but hey, it works, lol. – confetti Aug 31 '18 at 6:03

This should do what you want in awk:

ls -l | awk '$9~/./{for (i=1;i<9;i++){$i=""}; gsub(/^ +/,""); print}'

Here's a slightly different approach in perl:

ls -l | perl -lane 'if ($F[7]=~/./) {s/.*$F[7]//; print}'

Note: I cannot but recommend you think thoroghly about what you actually want to achieve, since there might be better ways. As mentioned by @confetti, just printing the filenames is more straighforwardly accomplished with ls -1.

Note: As correctly pointed out in the comments, this solution is not robust. I would add that no parsing of ls can be considered safe and sound.

Learning AWK: Since the OP explained in the comments that this question is part of a broader quest to become proficient in AWK, I have a few recommendations.

  • 1
    Create a file with multiple consecutive spaces in its name and see what your command prints for it. – Kamil Maciorowski Aug 31 '18 at 7:45
  • 1
    (for the awk) or a file named 0000 or -00.00 or 0e42 – dave_thompson_085 Aug 31 '18 at 10:02
  • @dave_thompson_085 rightful observation! This significant case can be taken into account substituting $9 with $9~/./ and $F[7] with $F[7]=~/./, but the whole idea of parsing ls output should be reexamined. – simlev Aug 31 '18 at 10:56
  • @simlev The broader problem that I'm trying to solve is to become proficient with AWK, so this was just a problem that I came up with to practice, but couldn't find a way to solve (except for the code that I provided). I was expecting this to be a fairly inefficient way of doing things, but was also expecting to learn a better way to do it in the process. – Jesus DA Sep 4 '18 at 17:12
  • @simlev I diffed the output from your answer with my manually edited file and they are equal, so this is the answer I was looking for. Plus, It gives a better way to do it and mentions the shortcomings of parsing the result from ls. – Jesus DA Sep 4 '18 at 17:16

This looks like an XY problem. ls -1 should print what you want. This option is required by POSIX so it's unlikely your ls doesn't support it.

However if you're about to parse the result, you shouldn't use ls at all. Use find, preferably with -print0 action.

So ls is not the right tool. If you insist on parsing the bare ls output anyway, then awk may not be the right tool either. Reasoning:

  • Your columns are separated by spaces; sometimes there are two or three consecutive spaces, this differs from line to line. So with this naive separator a filename in one line may begin at Nth column, while in another line it's at (N+1)th column.
  • You can tell awk to treat multiple spaces as one separator (awk -F ' +') but this will mangle filenames with many consecutive spaces.
  • So you need some logic to identify where a filename starts in every single line. This can be done (I believe awk is Turing complete), I can't give you any working solution though.

Even the output of ls -1 shouldn't be parsed, ls -l is worse.

  • Once you point out ls is not robust, it's no longer useful to speculate about how it could be parsed with any particular tool. To further prove how utterly useless this is, I for one have group names that contain spaces. – simlev Aug 31 '18 at 8:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.