AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.

learn more… | top users | synonyms

10
votes
6answers
3k views

Removing ANSI color codes from text stream

Examining the output from perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' in a text editor (e.g., vi) shows the following: ^[[37mABC ^[[0m How would one ...
8
votes
5answers
12k views

How to get the pid of a running process using a single command that parse the output of ps?

I am looking for a single line that does return the pid of a running process. Currently I have: ps -A -o pid,cmd|grep xxx|head -n 1 And this returns the fist pid, command. I need only the first ...
8
votes
3answers
7k views

Filtering top command output

I'm running a Mac and want to filter the output of the top command for a particular process such that only the PID, COMMAND and %CPU columns are displayed. After getting the PID of the process, I ...
8
votes
3answers
2k views

sed: how to replace line if found or append to end of file if not found?

With a single input file that only contains comments (starting with #) and VARIABLE=value lines, is it possible to replace a value for a single variable if found and, otherwise, append the pair to the ...
8
votes
2answers
8k views

How can I make a table of contents for a markdown document with Python/AWK/SED?

I have the following markdown document: Heading-a ========== ---text--- Heading-b ------------ --- text --- Heading-c ---------- --- text--- Heading-d ======= --- text---- Heading-e --- ... ...
7
votes
6answers
948 views

Joining text files with 600M+ lines

I have two files, huge.txt and small.txt. huge.txt has around 600M rows and it's 14 GB. Each line has four space separated words (tokens) and finally another space separated column with a number. ...
7
votes
5answers
3k views

awk, mawk, nawk, gawk… WHAT?

I've just started learning awk and I'm a little confused about all those versions around. Is there any "version" which is found on all Unix-like systems? Like, you know, plain vi? Does the standard ...
5
votes
1answer
4k views

Awk doesn't work when inside double quotes

This command doesn't work: ssh $HOST "ls -l | awk '{print $1}'" Above ignores the command "awk" I think it might be because of the double quotes? Also, how would I add another set of double quotes ...
5
votes
3answers
515 views

Search for files with more than one term (grep, awk?)

I am using a command like this to find files with the word 'term' in them: grep -l term * But I now want to be able to find files which have two different words in them (let's call them termA and ...
5
votes
4answers
3k views

SQL like group by and sum for text files in command line?

I have huge text files with two fields, the first is a string the second is an integer. The files are sorted by the first field. What I'd like to get in the output is one line per unique string and ...
5
votes
1answer
1k views

How to split binary file based on pattern?

I would like to split a binary file into smaller files based on pattern 'xÚ' (it's 78 DA in hexadecimal), so when there's an 'xÚ' in the file the splitter script splits and pastes the content into a ...
4
votes
5answers
2k views

Using awk to split text file every 10,000 lines

I have a large gzip'd text file. I'd like to something like: zcat BIGFILE.GZ | \ awk (snag 10,000 lines and redirect to...) | \ gzip -9 smallerPartFile.gz the awk part up there, I ...
4
votes
3answers
89 views

Output numbers from a file in a different order

I have a file that contains: 1 2 3 4 1 3 5 4 8 3 2 1 Each line has a different number of digits. And there are 1000 more like this. I want the output like this: 1 2 2 3 3 4 4 0 1 3 3 5 5 4 4 8 8 ...
4
votes
1answer
577 views

how to move around files using “ls -al” result in Linux?

How can I write a script to copy files from one directory to another directory according to last modified date? ls -al -rw-r--r-- 1 user user 100 2011-05-26 12:33 ...
4
votes
2answers
3k views

How can I identify non-ASCII characters from the shell?

Is there a simple way to print all non-ASCII characters and the line numbers on which they occur in a file using a command line utility such as grep, awk, perl, etc? I want to change the encoding of ...
4
votes
3answers
1k views

how to limit find command's output used with option -print0

I'd like to limit output of find command. In the past I used to use for this ls command, e.g: ls *tgz|head -100|xargs -i mv "{}" ../ but I got to know that the result may be unpredictable if name ...
4
votes
4answers
4k views

Print back-reference in regular expression

I was hoping for a way to make sed replace the entire line with the replacement (rather than just the match) so I could do something like this: sed -e "/$some_complex_regex_with_a_backref/\1/" and ...
4
votes
1answer
1k views

How do I parse file paths separated by a space in a string?

Background: I am working in Automator on a wrapper to a command line utility. I need a way to separate an arbitrary number of file paths delimited by a single space from a single string, so that I may ...
3
votes
5answers
1k views

error when using commandline as a bash alias on linux

I want to save the following commandline sequence as a bash alias: grep `date '+%d/%b/%Y'` access.logs | egrep 2765330645ae47d292c9ceac725d744e.py |awk '{print $1, $4, $5, $7, $8, $9, $10}' | sort ...
3
votes
4answers
1k views

Using grep to display second character in string?

I have a string that is always 3 characters... the first one and the last one are always the same. example: ▁▅█ Is there a simple way to only display the one in the middle? (which is the only one ...
3
votes
5answers
4k views

How to split a text file into multiple text files

I have a text file called entry.txt that contains the following: [ entry1 ] 1239 1240 1242 1391 1392 1394 1486 1487 1489 1600 1601 1603 1657 1658 1660 2075 2076 2078 2322 2323 2325 2740 2741 2743 ...
3
votes
4answers
2k views

How to remove lines from large text file using bash

I got a huge text file (log file) in my CentOS which I would like to remove top part of, probably couple of thousand lines each day. (Or probably just split into two) I have search this site and ...
3
votes
1answer
2k views

Reading output of shell command in awk script

I have a text file with a list ip addresses and other information. I am using an awk script to process this list and output various computations. I want to call dig -x from inside the awk script and ...
3
votes
1answer
3k views

Using grep to remove lines from a file which contain a string from another file

I have a file containing words (one per line) such as Dog Fish Cat Shoes I have a secondary file in CSV format such as 1,shoes,red 2,apple,black 3,fog,blue I would like to use grep with ...
3
votes
3answers
2k views

How can I insert a line in the middle of a file, with awk, given a search pattern to insert AFTER or BEFORE

I want to insert some text in the middle of a file. The text to insert would be after a specific line, say, "<begin>". I don't know the line number, nor do I know the number of lines in the text ...
3
votes
1answer
4k views

“No such file or directory” for /bin/awk on Ubuntu

How can I install awk in this version of Ubuntu 11? I installed it from Ubuntu Software Center and I cannot use it. I'm trying to run an awk script: #!/bin/awk Begin { print strftime("ora ...
3
votes
3answers
8k views

awk + print from the third field until end of line each word with one space

How do I print all $Organs words (from the third field) with one space between each Organ to another? My problem is that each $Organ param have different Organs number so I cant to guess the last ...
3
votes
2answers
4k views

replace nth occurence of string in each line of a text file

I have large text files with space delimited strings (2-5). The strings can contain "'" or "-". I'd like to replace say the second space with a pipe. What's the best way to go? Using sed I was ...
3
votes
3answers
164 views

unix command to verify span of word in text

What unix command(s) can I use to determine the line span that a word appears in text? The "span" being equal to the line number of the last instance of a word minus the line number of the first ...
3
votes
2answers
969 views

Increment one value in a text line, using script

I have a text file as input. I need to filter that through some program, SED, AWK, whatever, where i need to increment a value in a particular line every time i run the script. What's the best way ...
3
votes
3answers
287 views

Need to sed awk or grep output to extract strings

I have output which looks like this: "vancouver/news/vancouver-voices": { "status": 0, "time": 0, "timestamp": 1371298701 }, "vancouver/news/vancouvers-march-second-annual-success": { ...
3
votes
2answers
323 views

ash scripting: space-containing variable refuses to be grepped

I am trying to run the script listed at http://talk.maemo.org/showthread.php?t=70866&page=2 on its intended hardware, a Nokia Linux phone running BusyBox ash. The script receives the name of WiFi ...
3
votes
2answers
381 views

How to comment out multiple line matching a regex using sed or awk

I have a file that contains this: [...] location /static { ... multiple lines ... } [...] location /static/ { ... multiple lines ... } [...] And I want to get: [...] # ...
3
votes
1answer
148 views

Parse by Control Characters

I am trying to parse the output of a command that expects to be writing to the screen. It has data separated by move-to-origin control sequences (for the VT220, ESC[1;1H). I only need the last part ...
2
votes
3answers
1k views

How can cat/print a file except the last 2 lines?

How can get a file, except the last (for instance) 2 lines, with standard or GNU shell tools? (bash, tail, cat and so on.)
2
votes
4answers
7k views

grepping a substring from a grep result

Given a log file, I will usually do something like this: grep 'marker-1234' filter_log What is the difference in using '' or "" or nothing in the pattern? The above grep command will yield many ...
2
votes
3answers
378 views

using nawk, how to sum all numbers in $1, $2, and $3 fields?

using nawk, how to sum all numbers in $1, $2, and $3 fields? 8 6 2 7 3 5 1 9 4 I want the result to be sum = 45
2
votes
5answers
3k views

awk + print all line content except $1

I write the follwoing awk script: % echo /var/sysconfig/network/my_functions alpha beta gama | \ awk -v word=alpha '$2 == word { print $0 }' how to tell awk that I want to print all line ...
2
votes
4answers
2k views

Print line X lines before/after found line

I want to search for a particular string in a certain file. If I found the string, I also want to print the line X lines before (or after) that line. Can this with grep or awk, or do I need a ...
2
votes
3answers
355 views

How to substitute multiple lines between delimiters

Given the text example: .... text ,.. {{{python string1 = 'abcde' string2 = '12345' print(string1[[1:3]]) print(string2[[:-1]]) }}} .... text ,.. The [[ and ]] also appear outside of {{{ too. ...
2
votes
3answers
304 views

Copied directory appears to become larger at destination

I have the following code as part of a shell script: while [ $(ps -ef | awk '{print $2}' | grep -F "$CPPID") ]; do sleep 10 awk -v "usbsize=$(/bin/df | grep -F $DEVICEMOUNTPOINTQ | awk ...
2
votes
2answers
383 views

Grep, find line with minimum x words

Is there a way to find all lines with grep, that contains at least x words?
2
votes
1answer
171 views

grep the string lines from the file

csv file. 1,2,3,abc 2,3,4,def 3,4,5,abc 3,4,2,def 6,7,9,abc. I need to read that file and get the lines of file containing abc.
2
votes
2answers
265 views

Why is awk printing this line more than once?

I have the following ldif: dn: cn=Robert Smith,ou=people,dc=example,dc=com objectclass: inetOrgPerson cn: Robert Smith cn: Robert J Smith cn: bob smith sn: smith uid: rjsmith userpassword: rJsmitH ...
2
votes
1answer
30 views

Weird behavior of awk

Why do I get only "cat" when I run awk 'BEGIN { animal[three] = "hen" animal[two] = "dog" animal[one] = "cat" for (var in animal) { print animal[var] } } ?? Shouldn't it print ...
2
votes
4answers
800 views

How do you remove all occurrences of values in one list from another list?

I have a list of symbols such as... wer sfe efo How do I remove all instances of these (unique) symbols from another list of (non-unique) symbols? So in the following list, the lines starting ...
2
votes
1answer
2k views

How to use awk separator of single quote?

I'd like to use awk to pull out information from SQL output like the following: (count(distinct serverclass)='2') And need to extract the number, which is 2 in this example. I'd like to do ...
2
votes
3answers
1k views

Get LVM size using AWK?

I'm in need to find out the LVM size of image vm001. Let's say I have a LVM volume called /dev/VGgroup/vm001. Now using lvdisplay I can find out the size: --- Logical volume --- LV Name ...
2
votes
2answers
2k views

Is it possible to change the default field separator in awk?

I'm always using awk to manipulate comma separated files, so the first line in my code is always changing the field separator to "," like so: awk 'BEGIN {FS=","} $2 < 20 {print $1}' myfile.csv ...
2
votes
1answer
234 views

awk: equivalent of mysql's “\G” to print tabulated data in multiple lines

Is there an equivalent to mysql's "\G" at the end of the query but awk (or similar) that will to print tabulated data from a text file in multiple lines with the header next to it? Something that will ...