"sed" ("stream editor") is a Unix utility that parses and transforms text files.

learn more… | top users | synonyms

65
votes
4answers
77k views

How to match whitespace in sed?

How can I match whitespace in sed? In my data I want to match all of 3+ subsequent whitespace characters (tab space) and replace them by 2 spaces. How can this be done?
26
votes
10answers
14k views

How to Remove the Last 2 Lines of a Very Large File

I have a very large file ~400G, and I need to remove the last 2 lines from it. I tried to use sed, but it ran for hours before I gave up. Is there a quick way of doing this, or am I stuck with sed?
16
votes
4answers
3k views

Substitution in text file **without** regular expressions

I need to substitute some text inside a text file with a replacement. Usually I would do something like sed -i 's/text/replacement/g' path/to/the/file The problem is that both text and replacement ...
14
votes
5answers
16k views

use of alternation “|” in sed's regex

I am using sed, GNU sed version 4.2.1. I want to use the alternation "|" symbol in a subexpression. For example : echo "blia blib bou blf" | sed 's/bl\(ia|f\)//g' should return " blib bou " but ...
13
votes
5answers
4k views

How to mass prepend text to file names?

Lets say I have a directory full of .md files all named various things. Lets say I wanted to prepend "text" to the front of each file name. So for example: file a.md, b.md, and c.md would become test ...
13
votes
4answers
7k views

Newlines in sed on Mac OS X

I find that \n doesn't work in sed under Mac OS X. Specifically, say I want to break the words separated by a single space into lines: # input foo bar I use, echo "foo bar" | sed 's/ /\n/' But ...
12
votes
6answers
4k views

Removing ANSI color codes from text stream

Examining the output from perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' in a text editor (e.g., vi) shows the following: ^[[37mABC ^[[0m How would one ...
12
votes
4answers
8k views

how to use sed to remove null bytes?

What is the sed incantation to remove null bytes from a file? I'm trying: s/\000//g but that is stripping out strings of zeroes. s/\x00//g seems to have no effect. I'm trying to do this in a ...
10
votes
4answers
6k views

Better way to do “echo $x | sed …” and “echo $x | grep …”

I often find this in scripts (and, I have to admit, write it myself): a=`echo $x | sed "s/foo/bar/"` or if echo $x | grep foo then ... fi Consider "foo" to include some regex stuff. I feel ...
9
votes
1answer
3k views

Can GNU sed (for Windows) handle Unicode? If so, is it a code-page/locale issue, or a switch?

I've been using GNU SED on and off for a couple of years now. It spins me out a bit sometimes, but it does a good job... for single-byte char sets! I now and then notice references to GNU SED being ...
8
votes
5answers
15k views

How to get the pid of a running process using a single command that parse the output of ps?

I am looking for a single line that does return the pid of a running process. Currently I have: ps -A -o pid,cmd|grep xxx|head -n 1 And this returns the fist pid, command. I need only the first ...
8
votes
1answer
521 views

How to make sed omit not matching lines?

In the following example, sed matches lines starting with an a or a c and prints the first character of that line (a or c): $ echo "ag bh ci dj ek fl" | sed 's/\(a\|c\)./\1/' # Matches lines starting ...
8
votes
3answers
2k views

Cygwin bash sed locks my files

When I change files in cygwin bash with the sed command, the file gets locked. Reproduce: Open cmd and cd to non-user directory (f.e. temp) echo aaa > test.txt Open in texteditor, add line, try ...
8
votes
3answers
3k views

sed: how to replace line if found or append to end of file if not found?

With a single input file that only contains comments (starting with #) and VARIABLE=value lines, is it possible to replace a value for a single variable if found and, otherwise, append the pair to the ...
8
votes
2answers
9k views

How can I make a table of contents for a markdown document with Python/AWK/SED?

I have the following markdown document: Heading-a ========== ---text--- Heading-b ------------ --- text --- Heading-c ---------- --- text--- Heading-d ======= --- text---- Heading-e --- ... ...
7
votes
6answers
985 views

Joining text files with 600M+ lines

I have two files, huge.txt and small.txt. huge.txt has around 600M rows and it's 14 GB. Each line has four space separated words (tokens) and finally another space separated column with a number. ...
7
votes
2answers
2k views

Edit first line of large text file

I have a huge text file, far too big for the whole thing to be paged into memory. All I need to do with this text file is edit the first line (its a CSV file and I need to alter the titles). Is there ...
7
votes
3answers
3k views

Transforming CSV file using sed

In order to be able to import some data into a certain tool, I have to transform a CSV file from this format "data","data","data data","data","123" into this format data;data;data data;data;123 ...
7
votes
8answers
7k views

List only the device names of all available network interfaces

I want to get a list of all available Network-Device Names on my Linux server. I figured that netstat -a would do the job, however netstat produces quite much output: eth0 Link encap:Ethernet ...
6
votes
3answers
3k views

How can I delete U+200B (Zero-width space) using sed

I have a very large file that has zero-width spaces scattered throughout. It takes too long to open and edit using vi so I'd like to delete all instances of the character using sed. The problem is, ...
5
votes
3answers
1k views

Am I using sed properly to perform this conditional replace? Is there a better way?

I have a space-delimited file that's about 3200 lines long. Each line contains 7+ fields. What I would like to do is edit the file with sed so that every line containing a certain variable in field 5 ...
5
votes
2answers
6k views

Delete the first known character in a string with sed

How does one delete the first known character in a string with sed? For example, say I want to delete the first character (which is @) in the string "@ABCDEFG1234" I want sed to verify whether "@" ...
5
votes
3answers
602 views

Search for files with more than one term (grep, awk?)

I am using a command like this to find files with the word 'term' in them: grep -l term * But I now want to be able to find files which have two different words in them (let's call them termA and ...
5
votes
1answer
199 views

Non-greedy regular expression to convert command tags

Background Consider the following text: There are three types of font families: serif, sans serif, and teletype. To switch between these families, use <cmd>rm</cmd> for serif, ...
5
votes
4answers
3k views

SQL like group by and sum for text files in command line?

I have huge text files with two fields, the first is a string the second is an integer. The files are sorted by the first field. What I'd like to get in the output is one line per unique string and ...
4
votes
1answer
4k views

Using grep to remove lines from a file which contain a string from another file

I have a file containing words (one per line) such as Dog Fish Cat Shoes I have a secondary file in CSV format such as 1,shoes,red 2,apple,black 3,fog,blue I would like to use grep with ...
4
votes
1answer
1k views

Where does Macports install GNU sed when I install coreutils port?

Where does Macports install the GNU sed when I install coreutils port? I want to use GNU sed instead of BSD sed on Mac OS X.
4
votes
2answers
2k views

How append a string at the end of all lines?

I'm trying to add a string at the end of all lines in a text file, but i have a mistake... :( Example: I've this in a text file: begin--fr.a2dfp.net begin--m.fr.a2dfp.net begin--mfr.a2dfp.net ...
4
votes
3answers
4k views

sed: delete text between a string until first occurrence of another string

Imagine I have something like the following text: The quick brown fox jumps in 2012 and 2013 And I would wish to delete the part from "fox" including the four numbers but only in the first ...
4
votes
1answer
1k views

sed weirdness, unmatched {

I'm trying to use the r (read command to append the contents of data2 to data. I want to append the contents after the line containing 'two'. I want to delete the line containing 'two'. But I've hit ...
4
votes
2answers
2k views

Use sed command to replace , appearing between numbers

I have a CSV file where data are in the following format |001|,|abc,def|,123456,789,|aaa|,|bbb|,444,555,666 I want to replace only those "," that appears between numbers with some other character ...
4
votes
2answers
1k views

sed: replace first occurance of a string in every line

Imagine I have a file like this: INSERT INTO table VALUES('1','<p><em>The lazy fox jumps again</em></p>bunch of other html<p><em>Is the lazy fox ...
4
votes
3answers
564 views

sed: replace only the first range of numbers

Imagine I have an input file like this: INSERT INTO video_item_theme VALUES('9', '29'); INSERT INTO video_item_theme VALUES('19', '312'); INSERT INTO video_item_theme VALUES('414', '1'); And I wish ...
4
votes
2answers
350 views

shell script to replace a keyword in a file with the contents of another file

I'm to create a .sh to replace a specific keyword in a file using the contents of an other file. template.html contains a unique string "PLACEHOLDER" it must be replaced with the contents of file ...
4
votes
1answer
1k views

How can I use sed in /etc/apache2/sites-available?

I'm attempting to write a script that uses sed to copy the default file for apache and writes a new file, replacing /var/www ($DOC_ROOT below) with my own directory, $NEW_SITES_DIR. However, sed isn't ...
4
votes
4answers
4k views

Print back-reference in regular expression

I was hoping for a way to make sed replace the entire line with the replacement (rather than just the match) so I could do something like this: sed -e "/$some_complex_regex_with_a_backref/\1/" and ...
4
votes
1answer
2k views

How do I parse file paths separated by a space in a string?

Background: I am working in Automator on a wrapper to a command line utility. I need a way to separate an arbitrary number of file paths delimited by a single space from a single string, so that I may ...
4
votes
3answers
1k views

remove words containing non-alpha characters

Given a text file with space separated string and a tab separated integer, I'd like to get rid of all words that have non-alpha characters but keep words consisting of alpha only characters and the ...
3
votes
5answers
3k views

How to print last character of a file

In Unix, using a simple command like sed, is there a way to print the last character of a file?
3
votes
5answers
2k views

Using sed to remove digits and white space from a string

I am trying to remove the first occurence of digit(s), the dot, the second occurence of digit(s) and the space before the word. I have come up with this regex: sed 's/^[0-9]\+.[0-9]\+\s//' input.txt ...
3
votes
2answers
2k views

SED: How can I print every line after first instance of string using Sed?

I have a file with a similar format... 16:28 asdfasdf 16:29 4398upte 16:30 34liuthr 16:31 34tertio How can I use SED to print out every line including and after the line with "16:30"? The result ...
3
votes
4answers
3k views

How to remove lines from large text file using bash

I got a huge text file (log file) in my CentOS which I would like to remove top part of, probably couple of thousand lines each day. (Or probably just split into two) I have search this site and ...
3
votes
3answers
4k views

sed: replace any number of occurrences of a certain pattern

Given this input: "hell -- 'this -- world --is'-- beautiful' --thanks-- we'-- are-- here" I want to replace every '--' in between single quotes with 'X-X-X' using sed. It should give the ...
3
votes
2answers
4k views

use sed to replace nbsp, 160, Hex 00a0, Octal 240, non-breaking space

I am having some problems with config files which have the non-breaking space chars in them. How should I specify that character with sed so I can replace it with a space. sed -n 's/ / /g' ...
3
votes
3answers
1k views

combining sed with xargs to obtain a source and output file name

I have a situation where I have some input files like this: M2U0001.MPG M2U0180.MPG And I want to run a command (in a bash shell) on each similarly named file in the directory. I'd like the ...
3
votes
4answers
1k views

Search for and print only matched pattern

I have some huge xml text files. I need to write a script to find and print a specific tag only. I tried sed and grep but they both return the whole line. Using SunOS 5.x, so not all linux commands ...
3
votes
2answers
2k views

How can I use sed to alter the results of find and pass the results to cp?

In solaris, I'd like to copy all files found by the find command to a slightly different path. The following script basically executes cp for each file found by find. For example: cp ...
3
votes
2answers
2k views

Multiple search and replace actions in one large text file

I have a big text file (about 2GB). I want to do five search and replace actions on the same file, and would like to do this in one command. Normally I use vim, open the file, do one replace action, ...
3
votes
3answers
1k views

Substituting a multi-line pattern in an HTML file

I have a series of HTML files that contain two lines like this: <body> <h1>Title</h1><p> <a href="url">Description</a><br> I want to replace this text with ...
3
votes
2answers
2k views

How greedy is sed in matching patterns?

I know sed is greedy when matching patterns. But, how greedy is it? Consider these examples. $ echo 'foobar123' | sed 's/[0-9]*/(&)/' ()foobar123 $ echo 'foobar123' | sed ...