73

Examining the output from

perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";'

in a text editor (e.g., vi) shows the following:

^[[37mABC
^[[0m

How would one remove the ANSI color codes from the output file? I suppose the best way would be to pipe the output through a stream editor of sorts.

The following does not work

perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' | perl -pe 's/\^\[\[37m//g' | perl -pe 's/\^\[\[0m//g'
  • Not an answer to the question, but you can also pipe the output to more or less -R which can interpret the escape codes as color instead of a text editor. – terdon Jul 3 '13 at 13:50

10 Answers 10

96

The characters ^[[37m and ^[[0m are part of the ANSI escape sequences (CSI codes).  See also these specifications.

Using GNU sed

sed 's/\x1b\[[0-9;]*m//g'
  • \x1b (or \x1B) is the escape special character
    (sed does not support alternatives \e and \033)
  • \[ is the second character of the escape sequence
  • [0-9;]* is the color value(s) regex
  • m is the last character of the escape sequence

⚠ On macOS, the default sed command does not support special characters like \e as pointed out by slm and steamer25 in the comments. Use instead gsed that you can install using brew install gnu-sed.

Example with OP's command line:   (OP means Original Poster)

perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' | 
      sed 's/\x1b\[[0-9;]*m//g'

Tom Hale suggests to remove all other escape sequences using [a-zA-Z] instead of just the letter m specific to the graphics mode (color) escape sequence. But [a-zA-Z] may be too wide and could remove too much. Michał Faleński and Miguel Mota propose to remove only some escape sequences using [mGKH] and [mGKF] respectively. Britton Kerin indicates K must also be used in addition to m to remove the colors from gcc error/warning (do not forget to redirect gcc 2>&1 | sed...).

sed 's/\x1b\[[0-9;]*m//g'           # Remove color sequences only
sed 's/\x1b\[[0-9;]*[a-zA-Z]//g'    # Remove all escape sequences
sed 's/\x1b\[[0-9;]*[mGKH]//g'      # Remove color and move sequences
sed 's/\x1b\[[0-9;]*[mGKF]//g'      # Remove color and move sequences
Last escape
sequence
character   Purpose
---------   -------------------------------
m           Graphics Rendition Mode (including Color)
G           Horizontal cursor move
K           Horizontal deletion
H           New cursor position
F           Move cursor to previous n lines

Using perl

The version of sed installed on some operating systems may be limited (e.g. macOS). The command perl has the advantage of being generally easier to install/update on more operating systems. Adam Katz suggests to use \e (same as \x1b) in PCRE.

Choose your regex depending on how much commands you want to filter:

perl -pe 's/\e\[[0-9;]*m//g'          # Remove colors only
perl -pe 's/\e\[[0-9;]*[mG]//g'
perl -pe 's/\e\[[0-9;]*[mGKH]//g'
perl -pe 's/\e\[[0-9;]*[a-zA-Z]//g'
perl -pe 's/\e\[[0-9;]*m(?:\e\[K)?//g' # Adam Katz's trick

Example with OP's command line:

perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' \
      | perl -pe 's/\e\[[0-9;]*m//g'

Usage

As pointed out by Stuart Cardall's comment, this sed command line is used by the project Ultimate Nginx Bad Bot (1000 stars) to clean up the email report ;-)

  • 2
    Thanks for the sed command and the explanation. :) – Redsandro Feb 5 '13 at 14:15
  • 2
    Some color codes (e.g. Linux terminal) contain a prefix, e.g. 1;31m so better add ; to your regex: cat colored.log | sed -r 's/\x1b\[[0-9;]*m//g' or they won't be stripped. – Redsandro Mar 3 '14 at 13:11
  • 1
    this is great used it in github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker/blob/… to clean up the email report. – Stuart Cardall Jun 7 '17 at 18:59
  • 2
    Keep in mind that the OSX version of sed didn't work w/ the example shown, the gsed version however does. – slm Mar 1 at 21:50
  • 2
    More context for slm's comment about OSX sed: it doesn't support control characters like \x1b. E.g., stackoverflow.com/a/14881851/93345 . You can get the gsed command via brew install gnu-sed . – steamer25 May 7 at 15:48
21

I have found out a better escape sequence remover. Check this:

perl -pe 's/\x1b\[[0-9;]*[mG]//g'

10

What is displayed as ^[ is not ^ and [; it is the ASCII ESC character, produced by Esc or Ctrl[ (the ^ notation means the Ctrl key).

ESC is 0x1B hexadecimal or 033 octal, so you have to use \x1B or \033 in your regexes:

perl -pe 's/\033\[37m//g; s/\033[0m//g'

perl -pe 's/\033\[\d*(;\d*)*m//g'
6

If you prefer something simple you could use the strip-ansi module (Node.js required):

$ npm install --global strip-ansi-cli

Then use it like this:

$ strip-ansi < colors.o

Or just pass in a string:

$ strip-ansi '^[[37mABC^[[0m'
  • This a useless use of cat (UUOC) — it should be possible to do strip-ansi colors.o or at least strip-ansi < colors.o. – Scott Feb 11 '16 at 5:36
  • 1
    @Scott Sure, you can also do strip-ansi < colors.o, but from experience people are more familiar with piping. I've updated the answer. – Sindre Sorhus Feb 11 '16 at 9:09
  • good simple solution – Penghe Geng Jun 28 at 21:48
3

commandlinefu gives this answer which strips ANSI colours as well as movement commands:

sed "s,\x1B\[[0-9;]*[a-zA-Z],,g"

For just colours, you want:

 sed "s,\x1B\[[0-9;]*m,,g"
2

The "answered" question didn't work for me, so I created this regex instead to remove the escape sequences produced by the perl Term::ANSIColor module.

cat colors.o | perl -pe 's/\x1b\[[^m]+m//g;

Grawity's regex should work fine, but using +'s appears to work ok too.

  • 4
    (1) What do you mean by The "answered" question?  Do you mean the accepted answer?  (2) This command does not work — it does not even execute — because it has an unmatched (unbalanced) quote.  (3) This a useless use of cat (UUOC) — it should be possible to do perl -pe command colors.o.  (4) Who ever said anything about the codes being in a .o file? – Scott Feb 11 '16 at 5:35
2

I believe this is an authoritative removal of all ANSI escape sequences:

perl -pe '
  s/\e\[[\x30-\x3f]*[\x20-\x2f]*[\x40-\x7e]//g;
  s/\e[PX^_].*?\e\\//g;
  s/\e\][^\a]*(?:\a|\e\\)//g;
  s/\e[\[\]A-Z\\^_@]//g;'

(Please note that perl, like many other languages (but not sed), accepts \e as the escape character Esc, \x1b or \033 by code, shown in terminals as ^[. I'm using it here because it seems more intuitive.)

This perl command, which you can run all on one line if you prefer, has four replacements in it:

The first goes after CSI sequences (escape code sequences that begin with the "Control Sequence Introducer" of Esc[, which covers a lot more than the Select Graphic Rendition sequences that make up the color codes and other text decorations).

The second replacement removes the remaining sequences that involve trailing characters and terminate with ST (the String Terminator, Esc\). The third replacement is the same thing but also allows Operating System Command sequences to end with a BEL (\x07, \007, often \a).

The fourth replacement removes the remaining escapes.

Also consider removing other zero-width ASCII characters such as BEL and other more obscure C0 and C1 control characters. I've been using s/[\x00-\x1f\x7f-\x9f\xad]+//g, which also includes Delete and Soft Hyphen. This excludes Unicode's higher coded zero-width characters but I believe it's exhaustive for ASCII (Unicode \x00-\xff). If you do this, remove these last since they can be involved in longer sequences.

1

"tput sgr0" left this control character ^(B^[
Here is a modified version to take care of that.

perl -pe 's/\e[\[\(][0-9;]*[mGKFB]//g' logfile.log
  • Thanks for this... this worked for me to get rid of that tput sgr0 that the other solutions never seem to be able to get rid of. – TxAG98 Jul 26 at 22:28
0

I had similar problem with removing characters added from collecting interactive top output via putty and this helped:

cat putty1.log | perl -pe 's/\x1b.*?[mGKH]//g'
  • 3
    This a useless use of cat (UUOC) — it should be possible to do perl -pe command putty1.log. – Scott Feb 11 '16 at 5:36
0

This is what worked for me (tested on Mac OS X)

perl -pe 's/\[[0-9;]*[mGKF]//g'

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.