Super User is a question and answer site for computer enthusiasts and power users. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a long log file where each entry begins with a line containg only hyphens.

share|improve this question

migrated from Jul 14 '10 at 2:01

This question came from our site for professional and enthusiast programmers.

You can do it with a shell script thus:

if [[ -z "$1" ]] ; then
    echo Usage: $0 '<inputFile>'
    exit 1
line=$(grep -n '^--*$' "$1" | tail -1 | sed 's/:.*//')
if [[ -z "${line}" ]] ; then
    cat "$1"
    sed "1,${line}d" "$1"

Given the input file:

this is line 1
this is line 3
this is line 5
this is line 6

it produces:

this is line 5
this is line 6

By way of explanation, the grep -n produces a series of lines like:


where the 2 and 4 are the line numbers. The tail -1 then just filters out all but the last and the sed strips out everything from the colon to the end of the line, leaving just the line number

Then, if there was no lines with the desired pattern, it just outputs the entire file. Otherwise it deletes all the lines between 1 and the last hyphen line.

As an aside, my original answer included this awk snippet which will process the file only once:

awk '/^--*$/{s=""}{s=s$0"\n";}END{print s}'

However, keep in mind that it works by accumulating lines into a string and clearing the string out whenever it finds a hyphen line. Then, at the end, it simply outputs the string (all the lines after the last hyphen line).

While at first glance, this may appear to be more efficient, it doesn't seem to be in reality. In (admittedly non-exhaustive) tests on my system, it actually ran quite a bit slower, I think to do with the many string appends going on. The fact is that the script solution seems to be faster despite the fact that it makes multiple passes of the data (possibly because each pass is very limited in what it does).

share|improve this answer
This finds the first instead of the last hyphen-only line. If I understand correctly, the OP wants only line 5 in this example. – Philipp Jul 13 '10 at 12:17
@Philipp, the second awk command and the bash script both deliver the final section of the input file. – paxdiablo Jul 13 '10 at 12:24
And, since the bash script is a lot more efficient, I've ditched the awk answer anyway. – paxdiablo Jul 13 '10 at 12:39
But your awk version was surely better, in that it only read the file once, and was only a single program invocation. In this shell version, grep has to read the entire file, then tail, sed and cat or sed have to be invoked (on top of the shell invocation). – Norman Gray Jul 13 '10 at 13:29
The awk script that read the file once and used no extra storage was the one that started at the first hyphen line. The one that started on the last hyphen line did so by storing every line in-process so potentially used a lot of memory. – paxdiablo Jul 13 '10 at 13:42
awk -vRS="-+" 'END{print}' ORS="" file
share|improve this answer
Minor improvement: BEGIN{ORS=""} – Dennis Williamson Jul 13 '10 at 13:56
+1: Damn! neat. – Norman Gray Jul 13 '10 at 14:07
That is fast (and sneaky, which I particularly like in solutions) but I think it will match every line that has one or more hyphens in them, not "a line containing only hyphens". In other words, the line "abc-xyz" will match as well. Is there any way to put start- and end-line markers in the record separator? – paxdiablo Jul 13 '10 at 14:30
awk's RS regex doesn't "recognise" start of line. A more appropriate RS regex to use would be RS="\n-+". – user31894 Jul 13 '10 at 14:35
Wouldn't it be RS="\n-+\n" to ensure it's the whole line containing nothing but hyphens? – paxdiablo Jul 14 '10 at 2:00

You can also do it with sed:

% cat t.txt
this is line 1
this is line 2
this is line 3
this is line 4
this is line 5
this is line 6
% sed -n -e '/^---*/{h;d;}' -e H -e '${g;p;}' t.txt
this is line 5
this is line 6

(with some seds, those semicolons would have to be newlines).

share|improve this answer
If you don't want to print the dashes, make the last -e section like this: '${g;s/^-\+\n//;p}'. Also, your pattern would be better if it was /^-\+$/ or /^--*$/. – Dennis Williamson Jul 13 '10 at 14:06

I think this can be easily done using sed. You want a command to find the final (i.e. last) line of only-hyphens, and you want to print from that point to the end of file.

Unfortunately, I'm not very good with sed. Hoping someone else can elaborate.


OK, sed is not ideal. Here's how to do it with ex, the text-only twin of vi:

ex filename
share|improve this answer
I think it's very hard. If you read a file line-by-line, how would you know whether a specific line is the last of its kind before reading the remaining lines? I think there is no other way than to read the whole file. – Philipp Jul 13 '10 at 12:10
@Philipp: You're right, this is not a good job for a sequential editor. It does seem to be easy for any "real" editor, though. See my update. – Carl Smotricz Jul 13 '10 at 12:17
tac file | grep -B 10000 -m 1 -- '------' | tac
share|improve this answer
Very ingenuous solution, but I think you can replace the grep by sed '/^-\+$/Q' – Philipp Jul 13 '10 at 12:25

This is probably not the most efficient solution:


declare -i count=0
declare -i index=0

while read -r line
    [[ $line =~ $pattern ]] && index=$count
done < "$file"

tail -n "$((count - index))" "$file"
share|improve this answer

Use tac and sed:

$ cat log-file 

$ tac log-file | sed -e '/^-\+$/,$d' | tac
share|improve this answer
echo "`sed -n '/^--*$/=' <file> | tail -1`,\$p" <file>  | xargs sed -n

But I like Norman Gray's solution much better. May like it even more if he explained it :-)

share|improve this answer
Thanks! It uses the 'hold space', which is the one bit of state that sed can use. At every '---' line, the 'h' replaces the hold space with the current line, thus discarding anything else that was there; every other line is appended to the hold space; then on the last line, the pattern space is replaced by the current hold space, and printed. – Norman Gray Jul 14 '10 at 23:50

You must log in to answer this question.