"sed" ("stream editor") is a Unix utility that parses and transforms text files.

learn more… | top users | synonyms

117
votes
4answers
142k views

How to match whitespace in sed?

How can I match whitespace in sed? In my data I want to match all of 3+ subsequent whitespace characters (tab space) and replace them by 2 spaces. How can this be done?
35
votes
6answers
37k views

use of alternation “|” in sed's regex

I am using sed, GNU sed version 4.2.1. I want to use the alternation "|" symbol in a subexpression. For example : echo "blia blib bou blf" | sed 's/bl\(ia|f\)//g' should return " blib bou " but ...
28
votes
12answers
19k views

Efficiently remove the last two lines of an extremely large text file

I have a very large file (~400 GB), and I need to remove the last 2 lines from it. I tried to use sed, but it ran for hours before I gave up. Is there a quick way of doing this, or am I stuck with ...
27
votes
6answers
8k views

Substitution in text file **without** regular expressions

I need to substitute some text inside a text file with a replacement. Usually I would do something like sed -i 's/text/replacement/g' path/to/the/file The problem is that both text and replacement ...
21
votes
6answers
13k views

How to mass prepend text to file names?

Lets say I have a directory full of .md files all named various things. Lets say I wanted to prepend "text" to the front of each file name. So for example: file a.md, b.md, and c.md would become test -...
20
votes
4answers
15k views

Newlines in sed on Mac OS X

I find that \n doesn't work in sed under Mac OS X. Specifically, say I want to break the words separated by a single space into lines: # input foo bar I use, echo "foo bar" | sed 's/ /\n/' But ...
18
votes
2answers
20k views

How to match digits followed by a dot using sed?

I'm trying to use sed to substitute all the patterns with digits followed immediately by a dot (such as 3., 355.) by an empty string. So I try: sed 's/\d+\.//g' file.txt But it doesn't work. Why is ...
18
votes
4answers
16k views

How to use sed to remove null bytes?

What is the sed incantation to remove null bytes from a file? I'm trying: s/\000//g but that is stripping out strings of zeroes. s/\x00//g seems to have no effect. I'm trying to do this in a ...
17
votes
6answers
11k views

Removing ANSI color codes from text stream

Examining the output from perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' in a text editor (e.g., vi) shows the following: ^[[37mABC ^[[0m How would one ...
15
votes
2answers
5k views

Edit first line of large text file

I have a huge text file, far too big for the whole thing to be paged into memory. All I need to do with this text file is edit the first line (its a CSV file and I need to alter the titles). Is there ...
15
votes
7answers
8k views

sed: how to replace line if found or append to end of file if not found?

With a single input file that only contains comments (starting with #) and VARIABLE=value lines, is it possible to replace a value for a single variable if found and, otherwise, append the pair to the ...
14
votes
6answers
30k views

How to get the pid of a running process using a single command that parse the output of ps?

I am looking for a single line that does return the pid of a running process. Currently I have: ps -A -o pid,cmd|grep xxx|head -n 1 And this returns the fist pid, command. I need only the first ...
12
votes
3answers
3k views

Cygwin bash sed locks my files

When I change files in cygwin bash with the sed command, the file gets locked. Reproduce: Open cmd and cd to non-user directory (f.e. temp) echo aaa > test.txt Open in texteditor, add line, try ...
12
votes
3answers
12k views

How can I make a table of contents for a markdown document with Python/AWK/SED?

I have the following markdown document: Heading-a ========== ---text--- Heading-b ------------ --- text --- Heading-c ---------- --- text--- Heading-d ======= --- text---- Heading-e --- ... ...
11
votes
4answers
9k views

Better way to do “echo $x | sed …” and “echo $x | grep …”

I often find this in scripts (and, I have to admit, write it myself): a=$(echo "$x" | sed "s/foo/bar/") or if echo "$x" | grep -q foo then ... fi Consider "foo" to include some regex stuff. ...
9
votes
8answers
18k views

List only the device names of all available network interfaces

I want to get a list of all available Network-Device Names on my Linux server. I figured that netstat -a would do the job, however netstat produces quite much output: eth0 Link encap:Ethernet ...
9
votes
3answers
7k views

How can I delete U+200B (Zero-width space) using sed

I have a very large file that has zero-width spaces scattered throughout. It takes too long to open and edit using vi so I'd like to delete all instances of the character using sed. The problem is, ...
9
votes
1answer
4k views

Can GNU sed (for Windows) handle Unicode? If so, is it a code-page/locale issue, or a switch?

I've been using GNU SED on and off for a couple of years now. It spins me out a bit sometimes, but it does a good job... for single-byte char sets! I now and then notice references to GNU SED being ...
8
votes
2answers
9k views

Properly escaping forward slash in bash script for usage with sed

I'm trying to determine the size of the files that would be newly copied when syncing two folders by running rsync in dry mode and then summing up the sizes of the files listed in the output of rsync. ...
8
votes
4answers
18k views

Using sed get substring between two double quotes

I have a file xyz... rsync: "/home/path/to/file": Permission denied (13) rsync: "/home/path/to/file1": Permission denied (13) rsync: "/home/path/to/file2": Permission denied (13) rsync: "/home/path/...
8
votes
1answer
982 views

How to make sed omit not matching lines?

In the following example, sed matches lines starting with an a or a c and prints the first character of that line (a or c): $ echo "ag bh ci dj ek fl" | sed 's/\(a\|c\)./\1/' # Matches lines starting ...
8
votes
1answer
1k views

Using sed to enable .htaccess files in Apache config

I am trying to automate the provisioning of an Ubuntu Vagrant box with a Bash script. All is well, but I am having difficulty with configuring Apache to allow overrides from .htaccess files. In my ...
7
votes
6answers
1k views

Joining text files with 600M+ lines

I have two files, huge.txt and small.txt. huge.txt has around 600M rows and it's 14 GB. Each line has four space separated words (tokens) and finally another space separated column with a number. ...
7
votes
3answers
4k views

Transforming CSV file using sed

In order to be able to import some data into a certain tool, I have to transform a CSV file from this format "data","data","data data","data","123" into this format data;data;data data;data;123 ...
7
votes
2answers
5k views

Multiple search and replace actions in one large text file

I have a big text file (about 2GB). I want to do five search and replace actions on the same file, and would like to do this in one command. Normally I use vim, open the file, do one replace action, ...
6
votes
2answers
9k views

Delete the first known character in a string with sed

How does one delete the first known character in a string with sed? For example, say I want to delete the first character (which is @) in the string "@ABCDEFG1234" I want sed to verify whether "@" ...
6
votes
4answers
5k views

SQL like group by and sum for text files in command line?

I have huge text files with two fields, the first is a string the second is an integer. The files are sorted by the first field. What I'd like to get in the output is one line per unique string and ...
5
votes
3answers
12k views

How append a string at the end of all lines?

I'm trying to add a string at the end of all lines in a text file, but I have a mistake somewhere. Example: I've this in a text file: begin--fr.a2dfp.net begin--m.fr.a2dfp.net begin--mfr.a2dfp.net ...
5
votes
2answers
4k views

Finding and deleting lines from all files recursively

I have managed to grep the occurences of a keyword by using grep "KeyWord" . -r -n -i -I but, the occurences of the KeyWord are too large. I just want to delete all the lines containing that word. ...
5
votes
1answer
13k views

Multi-Line Sed Replace

Consider the following text (incidentally, part of a MySQL dump): CREATE TABLE `table` ( `id` int(10) NOT NULL auto_increment, `name` varchar(100) NOT NULL default '', `description` text NOT ...
5
votes
3answers
2k views

Am I using sed properly to perform this conditional replace? Is there a better way?

I have a space-delimited file that's about 3200 lines long. Each line contains 7+ fields. What I would like to do is edit the file with sed so that every line containing a certain variable in field 5 ...
5
votes
1answer
2k views

Where does Macports install GNU sed when I install coreutils port?

Where does Macports install the GNU sed when I install coreutils port? I want to use GNU sed instead of BSD sed on Mac OS X.
5
votes
3answers
782 views

Search for files with more than one term (grep, awk?)

I am using a command like this to find files with the word 'term' in them: grep -l term * But I now want to be able to find files which have two different words in them (let's call them termA and ...
5
votes
2answers
2k views

How can I use sed to alter the results of find and pass the results to cp?

In solaris, I'd like to copy all files found by the find command to a slightly different path. The following script basically executes cp for each file found by find. For example: cp ./content/...
5
votes
2answers
3k views

sed, insert file before last line

The following inserting file method for sed used to be working, even before last line, but not any more. Is it a bug in the current sed? Demo of inserting file method with sed: mkdir /tmp/test ...
5
votes
1answer
405 views

Non-greedy regular expression to convert command tags

Background Consider the following text: There are three types of font families: serif, sans serif, and teletype. To switch between these families, use <cmd>rm</cmd> for serif, <cmd&...
5
votes
4answers
6k views

Print back-reference in regular expression

I was hoping for a way to make sed replace the entire line with the replacement (rather than just the match) so I could do something like this: sed -e "/$some_complex_regex_with_a_backref/\1/" and ...
5
votes
1answer
154 views

How do I prepend a string to program output without waiting for the whole line?

I have a script that runs a command on a remote server using SSH. I want to prepend the string Remote: to every line of the output, but I don't want each line to be delayed until the whole line is ...
4
votes
5answers
4k views

Using sed to remove digits and white space from a string

I am trying to remove the first occurence of digit(s), the dot, the second occurence of digit(s) and the space before the word. I have come up with this regex: sed 's/^[0-9]\+.[0-9]\+\s//' input.txt ...
4
votes
2answers
3k views

SED: How can I print every line after first instance of string using Sed?

I have a file with a similar format... 16:28 asdfasdf 16:29 4398upte 16:30 34liuthr 16:31 34tertio How can I use SED to print out every line including and after the line with "16:30"? The result ...
4
votes
2answers
10k views

use sed to replace nbsp, 160, Hex 00a0, Octal 240, non-breaking space

I am having some problems with config files which have the non-breaking space chars in them. How should I specify that character with sed so I can replace it with a space. sed -n 's/ / /g' ...
4
votes
1answer
8k views

Using grep to remove lines from a file which contain a string from another file

I have a file containing words (one per line) such as Dog Fish Cat Shoes I have a secondary file in CSV format such as 1,shoes,red 2,apple,black 3,fog,blue I would like to use grep with ...
4
votes
2answers
11k views

replace nth occurence of string in each line of a text file

I have large text files with space delimited strings (2-5). The strings can contain "'" or "-". I'd like to replace say the second space with a pipe. What's the best way to go? Using sed I was ...
4
votes
3answers
7k views

sed: delete text between a string until first occurrence of another string

Imagine I have something like the following text: The quick brown fox jumps in 2012 and 2013 And I would wish to delete the part from "fox" including the four numbers but only in the first ...
4
votes
2answers
2k views

shell script to replace a keyword in a file with the contents of another file

I'm to create a .sh to replace a specific keyword in a file using the contents of an other file. template.html contains a unique string "PLACEHOLDER" it must be replaced with the contents of file "...
4
votes
1answer
2k views

sed weirdness, unmatched {

I'm trying to use the r (read command to append the contents of data2 to data. I want to append the contents after the line containing 'two'. I want to delete the line containing 'two'. But I've hit ...
4
votes
2answers
3k views

Use sed command to replace , appearing between numbers

I have a CSV file where data are in the following format |001|,|abc,def|,123456,789,|aaa|,|bbb|,444,555,666 I want to replace only those "," that appears between numbers with some other character ...
4
votes
1answer
68 views

Can I create a “file” that, when opened, actually gives the output of a command?

I've found myself in a couple of situations now where I'd like to have such a "file" that essentially runs a script whenever it's "opened", allowing me to "read" it's output. For example, I might ...
4
votes
2answers
14k views

How do I encrypt Samsung 840 SSD with Bitlocker?

I need help encrypting my Samsung 840 Evo with Bitlocker. Here is my hardware: Samsung Evo 840 500GB hard drive (firmware EXT0BB6Q) Asus P8z77-v motherboard (BIOS v. 2104) Asus TPM chip - purchased ...
4
votes
2answers
2k views

Bulk Rename Utility

I have been using Bulk Rename utility and I have been having some issues. I have some files named things like 664722.pdf,664762.pdf, and 664722-1.pdf that I need to rename. The problem is I have to ...