"sed" ("stream editor") is a Unix utility that parses and transforms text files.

learn more… | top users | synonyms

82
votes
4answers
102k views

How to match whitespace in sed?

How can I match whitespace in sed? In my data I want to match all of 3+ subsequent whitespace characters (tab space) and replace them by 2 spaces. How can this be done?
26
votes
11answers
16k views

How to Remove the Last 2 Lines of a Very Large File

I have a very large file ~400G, and I need to remove the last 2 lines from it. I tried to use sed, but it ran for hours before I gave up. Is there a quick way of doing this, or am I stuck with sed?
21
votes
5answers
23k views

use of alternation “|” in sed's regex

I am using sed, GNU sed version 4.2.1. I want to use the alternation "|" symbol in a subexpression. For example : echo "blia blib bou blf" | sed 's/bl\(ia|f\)//g' should return " blib bou " but ...
20
votes
5answers
5k views

Substitution in text file **without** regular expressions

I need to substitute some text inside a text file with a replacement. Usually I would do something like sed -i 's/text/replacement/g' path/to/the/file The problem is that both text and replacement ...
15
votes
5answers
7k views

How to mass prepend text to file names?

Lets say I have a directory full of .md files all named various things. Lets say I wanted to prepend "text" to the front of each file name. So for example: file a.md, b.md, and c.md would become test ...
15
votes
4answers
11k views

How to use sed to remove null bytes?

What is the sed incantation to remove null bytes from a file? I'm trying: s/\000//g but that is stripping out strings of zeroes. s/\x00//g seems to have no effect. I'm trying to do this in a ...
14
votes
6answers
7k views

Removing ANSI color codes from text stream

Examining the output from perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' in a text editor (e.g., vi) shows the following: ^[[37mABC ^[[0m How would one ...
13
votes
5answers
21k views

How to get the pid of a running process using a single command that parse the output of ps?

I am looking for a single line that does return the pid of a running process. Currently I have: ps -A -o pid,cmd|grep xxx|head -n 1 And this returns the fist pid, command. I need only the first ...
13
votes
4answers
10k views

Newlines in sed on Mac OS X

I find that \n doesn't work in sed under Mac OS X. Specifically, say I want to break the words separated by a single space into lines: # input foo bar I use, echo "foo bar" | sed 's/ /\n/' But ...
11
votes
3answers
4k views

sed: how to replace line if found or append to end of file if not found?

With a single input file that only contains comments (starting with #) and VARIABLE=value lines, is it possible to replace a value for a single variable if found and, otherwise, append the pair to the ...
10
votes
4answers
7k views

Better way to do “echo $x | sed …” and “echo $x | grep …”

I often find this in scripts (and, I have to admit, write it myself): a=`echo $x | sed "s/foo/bar/"` or if echo $x | grep foo then ... fi Consider "foo" to include some regex stuff. I feel ...
10
votes
3answers
2k views

Cygwin bash sed locks my files

When I change files in cygwin bash with the sed command, the file gets locked. Reproduce: Open cmd and cd to non-user directory (f.e. temp) echo aaa > test.txt Open in texteditor, add line, try ...
10
votes
2answers
10k views

How can I make a table of contents for a markdown document with Python/AWK/SED?

I have the following markdown document: Heading-a ========== ---text--- Heading-b ------------ --- text --- Heading-c ---------- --- text--- Heading-d ======= --- text---- Heading-e --- ... ...
9
votes
2answers
3k views

Edit first line of large text file

I have a huge text file, far too big for the whole thing to be paged into memory. All I need to do with this text file is edit the first line (its a CSV file and I need to alter the titles). Is there ...
9
votes
1answer
3k views

Can GNU sed (for Windows) handle Unicode? If so, is it a code-page/locale issue, or a switch?

I've been using GNU SED on and off for a couple of years now. It spins me out a bit sometimes, but it does a good job... for single-byte char sets! I now and then notice references to GNU SED being ...
8
votes
1answer
693 views

How to make sed omit not matching lines?

In the following example, sed matches lines starting with an a or a c and prints the first character of that line (a or c): $ echo "ag bh ci dj ek fl" | sed 's/\(a\|c\)./\1/' # Matches lines starting ...
8
votes
8answers
11k views

List only the device names of all available network interfaces

I want to get a list of all available Network-Device Names on my Linux server. I figured that netstat -a would do the job, however netstat produces quite much output: eth0 Link encap:Ethernet ...
7
votes
6answers
1k views

Joining text files with 600M+ lines

I have two files, huge.txt and small.txt. huge.txt has around 600M rows and it's 14 GB. Each line has four space separated words (tokens) and finally another space separated column with a number. ...
7
votes
3answers
4k views

Transforming CSV file using sed

In order to be able to import some data into a certain tool, I have to transform a CSV file from this format "data","data","data data","data","123" into this format data;data;data data;data;123 ...
7
votes
3answers
5k views

How can I delete U+200B (Zero-width space) using sed

I have a very large file that has zero-width spaces scattered throughout. It takes too long to open and edit using vi so I'd like to delete all instances of the character using sed. The problem is, ...
6
votes
2answers
9k views

How to match digits followed by a dot using sed?

I'm trying to use sed to substitute all the patterns with digits followed immediately by a dot (such as 3., 355.) by an empty string. So I try: sed 's/\d+\.//g' file.txt But it doesn't work. Why is ...
6
votes
2answers
5k views

Properly escaping forward slash in bash script for usage with sed

I'm trying to determine the size of the files that would be newly copied when syncing two folders by running rsync in dry mode and then summing up the sizes of the files listed in the output of rsync. ...
5
votes
3answers
1k views

Am I using sed properly to perform this conditional replace? Is there a better way?

I have a space-delimited file that's about 3200 lines long. Each line contains 7+ fields. What I would like to do is edit the file with sed so that every line containing a certain variable in field 5 ...
5
votes
2answers
8k views

Delete the first known character in a string with sed

How does one delete the first known character in a string with sed? For example, say I want to delete the first character (which is @) in the string "@ABCDEFG1234" I want sed to verify whether "@" ...
5
votes
3answers
686 views

Search for files with more than one term (grep, awk?)

I am using a command like this to find files with the word 'term' in them: grep -l term * But I now want to be able to find files which have two different words in them (let's call them termA and ...
5
votes
2answers
3k views

Multiple search and replace actions in one large text file

I have a big text file (about 2GB). I want to do five search and replace actions on the same file, and would like to do this in one command. Normally I use vim, open the file, do one replace action, ...
5
votes
1answer
296 views

Non-greedy regular expression to convert command tags

Background Consider the following text: There are three types of font families: serif, sans serif, and teletype. To switch between these families, use <cmd>rm</cmd> for serif, ...
5
votes
4answers
4k views

SQL like group by and sum for text files in command line?

I have huge text files with two fields, the first is a string the second is an integer. The files are sorted by the first field. What I'd like to get in the output is one line per unique string and ...
5
votes
4answers
5k views

Print back-reference in regular expression

I was hoping for a way to make sed replace the entire line with the replacement (rather than just the match) so I could do something like this: sed -e "/$some_complex_regex_with_a_backref/\1/" and ...
4
votes
3answers
5k views

How append a string at the end of all lines?

I'm trying to add a string at the end of all lines in a text file, but I have a mistake somewhere. Example: I've this in a text file: begin--fr.a2dfp.net begin--m.fr.a2dfp.net begin--mfr.a2dfp.net ...
4
votes
1answer
6k views

Using grep to remove lines from a file which contain a string from another file

I have a file containing words (one per line) such as Dog Fish Cat Shoes I have a secondary file in CSV format such as 1,shoes,red 2,apple,black 3,fog,blue I would like to use grep with ...
4
votes
2answers
7k views

use sed to replace nbsp, 160, Hex 00a0, Octal 240, non-breaking space

I am having some problems with config files which have the non-breaking space chars in them. How should I specify that character with sed so I can replace it with a space. sed -n 's/ / /g' ...
4
votes
4answers
12k views

Using sed get substring between two double quotes

I have a file xyz... rsync: "/home/path/to/file": Permission denied (13) rsync: "/home/path/to/file1": Permission denied (13) rsync: "/home/path/to/file2": Permission denied (13) rsync: ...
4
votes
1answer
2k views

Where does Macports install GNU sed when I install coreutils port?

Where does Macports install the GNU sed when I install coreutils port? I want to use GNU sed instead of BSD sed on Mac OS X.
4
votes
3answers
5k views

sed: delete text between a string until first occurrence of another string

Imagine I have something like the following text: The quick brown fox jumps in 2012 and 2013 And I would wish to delete the part from "fox" including the four numbers but only in the first ...
4
votes
2answers
6k views

replace nth occurence of string in each line of a text file

I have large text files with space delimited strings (2-5). The strings can contain "'" or "-". I'd like to replace say the second space with a pipe. What's the best way to go? Using sed I was ...
4
votes
2answers
1k views

shell script to replace a keyword in a file with the contents of another file

I'm to create a .sh to replace a specific keyword in a file using the contents of an other file. template.html contains a unique string "PLACEHOLDER" it must be replaced with the contents of file ...
4
votes
1answer
2k views

sed weirdness, unmatched {

I'm trying to use the r (read command to append the contents of data2 to data. I want to append the contents after the line containing 'two'. I want to delete the line containing 'two'. But I've hit ...
4
votes
2answers
2k views

Use sed command to replace , appearing between numbers

I have a CSV file where data are in the following format |001|,|abc,def|,123456,789,|aaa|,|bbb|,444,555,666 I want to replace only those "," that appears between numbers with some other character ...
4
votes
1answer
65 views

Can I create a “file” that, when opened, actually gives the output of a command?

I've found myself in a couple of situations now where I'd like to have such a "file" that essentially runs a script whenever it's "opened", allowing me to "read" it's output. For example, I might ...
4
votes
2answers
2k views

sed: replace first occurance of a string in every line

Imagine I have a file like this: INSERT INTO table VALUES('1','<p><em>The lazy fox jumps again</em></p>bunch of other html<p><em>Is the lazy fox ...
4
votes
3answers
675 views

sed: replace only the first range of numbers

Imagine I have an input file like this: INSERT INTO video_item_theme VALUES('9', '29'); INSERT INTO video_item_theme VALUES('19', '312'); INSERT INTO video_item_theme VALUES('414', '1'); And I wish ...
4
votes
1answer
1k views

How can I use sed in /etc/apache2/sites-available?

I'm attempting to write a script that uses sed to copy the default file for apache and writes a new file, replacing /var/www ($DOC_ROOT below) with my own directory, $NEW_SITES_DIR. However, sed isn't ...
4
votes
1answer
2k views

How do I parse file paths separated by a space in a string?

Background: I am working in Automator on a wrapper to a command line utility. I need a way to separate an arbitrary number of file paths delimited by a single space from a single string, so that I may ...
4
votes
3answers
2k views

remove words containing non-alpha characters

Given a text file with space separated string and a tab separated integer, I'd like to get rid of all words that have non-alpha characters but keep words consisting of alpha only characters and the ...
3
votes
5answers
4k views

How to print last character of a file

In Unix, using a simple command like sed, is there a way to print the last character of a file?
3
votes
5answers
3k views

Using sed to remove digits and white space from a string

I am trying to remove the first occurence of digit(s), the dot, the second occurence of digit(s) and the space before the word. I have come up with this regex: sed 's/^[0-9]\+.[0-9]\+\s//' input.txt ...
3
votes
4answers
8k views

grepping a substring from a grep result

Given a log file, I will usually do something like this: grep 'marker-1234' filter_log What is the difference in using '' or "" or nothing in the pattern? The above grep command will yield many ...
3
votes
3answers
5k views

sed: replace any number of occurrences of a certain pattern

Given this input: "hell -- 'this -- world --is'-- beautiful' --thanks-- we'-- are-- here" I want to replace every '--' in between single quotes with 'X-X-X' using sed. It should give the ...
3
votes
4answers
13k views

Print line X lines before/after found line

I want to search for a particular string in a certain file. If I found the string, I also want to print the line X lines before (or after) that line. Can this with grep or awk, or do I need a ...