AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.

learn more… | top users | synonyms

0
votes
0answers
9 views

Oneliner to get the list of filenames which have last line (or line on position X) matching certain pattern?

How to get the list of filenames which have last line (or line on position X) matching certain pattern? We are looking for oneliner.
5
votes
3answers
753 views

Search for files with more than one term (grep, awk?)

I am using a command like this to find files with the word 'term' in them: grep -l term * But I now want to be able to find files which have two different words in them (let's call them termA and ...
1
vote
1answer
33 views

Remove a certain part in a text file

I got the file, text.txt. In this file I have written: "test" "okay" I want to remove the " part of this file, how'd I do this?
3
votes
1answer
28 views

file splitting and compression in pipeline

So I have a massive file something like this... 1/20/2016,somerandomdata 1/20/2016,somerandomdata 1/20/2016,somerandomdata 1/20/2016,somerandomdata 1/21/2016,somerandomdata 1/21/2016,somerandomdata ...
0
votes
0answers
15 views

Increase efficiency of bash loop with awk

I have to write a fairly large text file(>300,000 lines) using text extracted from the second column of one file, then prepending and appending text to each line and finally writing it out to a new ...
1
vote
1answer
43 views

How to search for a line which contains specific string in Bash?

I've got a file which contains some amount of lines, I want to look for a specific string, and if the file contains it, then I want to print a different line from the same file. for example, let's ...
0
votes
2answers
28 views

Rename CSV header using command line tools

Given a CSV file, h1 h2 h3 a b c z x b how to rename the first header to One and the third header to Three using command line tools, namely One h2 Three a b c z x b
1
vote
2answers
245 views

How to convert a column of hex to dec in gawk, strtonum in gawk gives erroneous results

I am trying to script a database from a network stream. The network stream after being heavily sed'd drops a three column file that looks something like this called file.db ...
0
votes
1answer
22 views

Transform CSV column by a function using command line tools

Given a csv file such as h1 h2 a 0 b 1 c 0 how to rename 0 to YES and 1 to NO in the second column.
2
votes
2answers
405 views

How can I sort and de-dupe a very long list of words?

I have a 400 megabyte file. The file is a list of words, each word on a single line. End of lines are a mix of cr/lf and lf. Some of the words are unicode. I'd prefer answers for Linux or Windows ...
2
votes
2answers
29 views

compare and calculate two variables with multiple values in shell script

I have written three hive queries in shell script for calculating debit-credit and target amounts which output below values: debit_amount : BOI 4760545.650000 AXIS 284.49000000 SBI 87.220000000 ...
0
votes
1answer
40 views

awk - field separator change in program, letters in the field

I want to change FS in awk, but not in BEGIN. I want to print word and the letter. awk 'BEGIN{FS=""}{print $1; FS=" ";print $1; FS=""}' Of course it does't work. Is there any function, way to show ...
0
votes
1answer
37 views

awk - Assign variable result of system function

I would like to assign variable result of system function: awk 'BEGIN{x=system("cut -d \" \" -f 1 ol");print x}' Input "ol": 1 Result: 1 0
0
votes
1answer
71 views

Multiple replace - using sed or awk

I have a code snippet like below CREATE TABLE TABLE1( ID integer NOT NULL, ROW_CREATED varchar (26) NULL, BATCH_LOG_ID integer NULL, TRANSACTION_TYPE varchar(5) NULL, PARTITION_COL integer ...
0
votes
1answer
13 views

awk print data in fixed column

I would like to print data in fixed column. I mean the column should be like in table. I used printf. Input: To find information use the search box on the top right corner of the screen, or ...
3
votes
3answers
63 views

Filtering and splitting huge file based on column Windows

I have about 2 TB worth of data files formatted like 12/20/2015 somerandomdata 12/20/2015 somerandomdata 12/20/2015 somerandomdata 12/20/2015 somerandomdata 12/21/2015 somerandomdata 12/21/2015 ...
0
votes
1answer
25 views

A way to split a text file into arbitrary blocks based on first column?

I have a (very large) text file of the form... /folder1/fileA /folder1/fileB /folder1/fileC ... /folder999/fileA /folder999/fileB /folder999/fileC ...etc with an unknown number of entries for each ...
1
vote
2answers
39 views

AWK - Add variable and result of function to variable

I want to add variable and result of function in AWK. Input: t t t t a t a ta ata ta a a Script: { key="t" print gsub(key,"")#<-it's work b=b+gsub(key,"")#<- it's something wrong } END{ ...
-1
votes
1answer
30 views
-2
votes
1answer
22 views

How to find files within a size range?

Please suggest me the way to find all the zip files which are more than 60 MB but less than 70 MB in size using find command.
0
votes
0answers
12 views

fetch required column with delimiter from files in shell script

I am storing output of pig file in a directory Which contains lines with multiple columns separated by delimiter “ \u0001” Now in shell script I want to fetch 7th column field from the output I ...
1
vote
0answers
96 views

Automatically split a huge TeX file into smaller files

How would I automatically split the sections or chapters of a huge TeX file into many smaller TeX files? This somewhat works: awk -v RS='\\chapter{' '{print "\\chapter{"$0 > (NR".tex")}' ...
0
votes
4answers
68 views

How to replace first n no. of occurrence of a string using sed and/or awk?

I got a file alphabet where a has multiple occurrence in a line. $ cat alphabet a b c d e f g h i j k a a l m n a p q r a s t u v w a x y z a k l q z where $ cat alphabet | grep -o a | wc -l 7 ...
0
votes
1answer
35 views

bash script read line by line

Hello I am trying to create a script that reads my file take all the variable it needs then use it for another command. But it seems that the variables are not being memorized, because I tried a ...
2
votes
1answer
85 views

Substitute large numbers of files according to style guide

I'd like to do a wholesale reformatting of our tests, and I'm cleaning up some inconsistent capitalization. I'm thinking of using awk to do this, since sed falls a little short, and since I need ...
1
vote
1answer
30 views

Print every line but conditionally only fragments for some lines

I have a text file that looks like this: rno-miR-344-5p miRPlus_11239/mmu-miR-383/rno-miR-383 hsa-miR-301a/mmu-miR-301a/rno-miR-301a ...
16
votes
6answers
9k views

Removing ANSI color codes from text stream

Examining the output from perl -e 'use Term::ANSIColor; print color "white"; print "ABC\n"; print color "reset";' in a text editor (e.g., vi) shows the following: ^[[37mABC ^[[0m How would one ...
0
votes
1answer
296 views

Perform a GROUP BY-like command in UNIX

I am having a text file like the following: 1 2 3 4 5 6 7 8 9 ... n <-- column numbering 1 0 0 1 0 0 0 1 0 ... 0 0 1 0 0 0 0 0 0 0 ... 1 1 0 0 0 0 0 0 1 1 ... 0 0 1 1 1 0 1 0 0 0 ... 0 0 1 0 0 ...
-1
votes
1answer
46 views

separate a large 7GB tab limited file into multiple files

I want to separate a large 7GB file into multiple files (each file size= 1 GB) without changing it's format.. How is it possible ? AWK or any script is available? My file look like this: ...
14
votes
5answers
6k views

sed: how to replace line if found or append to end of file if not found?

With a single input file that only contains comments (starting with #) and VARIABLE=value lines, is it possible to replace a value for a single variable if found and, otherwise, append the pair to the ...
0
votes
2answers
46 views

Why can't I redirect output from awk?

I've written this one liner to show MB read/written every 5 seconds. #!/bin/bash #iosum.sh iostat -dmz 5 |\ awk 'BEGIN{rx=wx=0}{if($0 == ""){printf"%.1f %.1f\n",rx,wx}else ...
1
vote
1answer
40 views

How to find the last 2 characters of Chinese names with command lines tools?

I have a list of Chinese names, some of them have 2 characters, others may have more or less, like following: I want to use some command lines tools, to find the last 2 characters of each. If ...
0
votes
1answer
66 views

Merging two files in unix, with one common column that is redundant

I have two files with one common column that is redundant.File 1 has chromosomal locations and TF's, file 2 has chromosomal locations and Refseq numbers. File 1: ...
2
votes
2answers
409 views

Why is awk printing this line more than once?

I have the following ldif: dn: cn=Robert Smith,ou=people,dc=example,dc=com objectclass: inetOrgPerson cn: Robert Smith cn: Robert J Smith cn: bob smith sn: smith uid: rjsmith userpassword: rJsmitH ...
2
votes
3answers
73 views

How to fetch address and modify the value in key/value format file?

I'm currently writing a shell-script for my Raspberry Pi so it sets up an access-point with a unique SSID. To do this I want to use the mac-address of the WiFi dongle as my SSID. For doing that I want ...
2
votes
2answers
54 views

How to add # Shebang in first line of a file which is an auto create script by another program

I am looking for a method to add a shebang #!/bin/csh -f to first line of my file , which is actually getting created by a another set of program, since this script is auto-created, it should run from ...
11
votes
3answers
11k views

How can I make a table of contents for a markdown document with Python/AWK/SED?

I have the following markdown document: Heading-a ========== ---text--- Heading-b ------------ --- text --- Heading-c ---------- --- text--- Heading-d ======= --- text---- Heading-e --- ... ...
0
votes
1answer
113 views

How to encrypt the mail in unix?

I want to encrypt the mail as that will be containing the confidential information. There are no attachments. Please help in providing the steps to encrypt the mail. I have got the reference of How ...
2
votes
2answers
210 views

Count elements in lines with awk

I have a tab-delimited text file containing data like Col 1 Col 2 1 a 1 b 1 c 1 d 1 d 2 a 2 b 3 a so on and so forth. I would like to transform this structure into ...
13
votes
6answers
21k views

How to get video duration in seconds?

How can I get video duration in seconds? What I've tried: ffmpeg -i file.flv 2>&1 | grep "Duration" Duration: 00:39:43.08, start: 0.040000, bitrate: 386 kb/s mediainfo file.flv | grep ...
1
vote
1answer
58 views

awk character differences when using if

I have an input file with this line(user data/columns stripped out) and several thousand more. The xCE is an unconverted hex value from the clients file. 412640 xCE When I run it thru this awk ...
2
votes
2answers
61 views

Sed/Awk save text between patterns if contains string

I'm facing an issue with mails. I need to get all messages between 2 people: somebody1@domain.com and person@domain.com. The file: From: somebody1@domain.com to: person@domain.com <body of the ...
1
vote
1answer
30 views

Generate CSV from NS Zone File

I have a large file that contains records in multilines. ex: domain1 NS ns1 domain1 NS ns2 domain1 NS ns3 domain2 NS dnsx What might be the fastest way to generate a CSV of the form ...
1
vote
2answers
19 views

Conditional AND in awk with environment variables

I found this command. If a line has interested word, the command will output the entire line: temp="sample" awk -F " " -v var="$temp" '$1 == var' /root/smaple.txt smaple.txt contains: sample demo ...
1
vote
3answers
83 views

linux - replace all double newlines with a space

Is it possible to replace only double newlines in a text file using sed, awk, grep, tr, or whatever is needed? this is data this is more data What I need to do is to replace all double ...
5
votes
1answer
2k views

How to split binary file based on pattern?

I would like to split a binary file into smaller files based on pattern 'xÚ' (it's 78 DA in hexadecimal), so when there's an 'xÚ' in the file the splitter script splits and pastes the content into a ...
1
vote
1answer
430 views

How to run the awk command in Windows 7?

I have the following adb + awk command which will work fine in Linux environment [Ref]. adb shell dumpsys package | awk -v RS='\n +Package' '/android\.permission\.CAMERA/{print $1}' But I need to ...
0
votes
1answer
25 views

Awk eqivalent to grep -C <num>

I want to print 5 lines of context around each line that matches "foo" but not "foobar". I can do the match easily enough via <output> | awk '/foo/ && !/foobar/' but I can't figure out ...
2
votes
2answers
3k views

How can I find my awk version?

If I want to know the version of awk I get the following: $ awk --version awk: not an option: --version Checking in man awk I see that my awk is mawk - pattern scanning and text processing ...
0
votes
2answers
48 views

AWK formatting Disk Utility command formatting incorrectly

I am trying to learn awk at the moment and so I am trying it on the disk utility command. Here is the output when I run Disk Utility list disk0 /dev/disk0 (internal, physical): #: ...