Questions tagged [awk]
AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.
578
questions
0
votes
1answer
16 views
sed/awk: convertion of multi-column data
I am operating with multi-column data file procuded by some AWK script:
# output.csv
lig12, dG(rescored)
1, 0.596625
2, 1.05873
3, 1.11285
4, 0.697402
I need to modify this output.csv to one line ...
1
vote
1answer
43 views
Use sed/awk to add specific text from one file to beginning of another file
I'm trying to use sed/awk to copy a range of text from the beginning of file1 to a specific line. Then add that output to the beginning of file 2 for a script I'm writing.
The following command does ...
0
votes
2answers
16 views
Extracting data from text file by specific characters
I have a program that reads from a sensor and prints on stdout. The interesting data for me is just the X,Y and Z values:
[2021-05-07 10:47:41.993] [info] Acceleromter (avrg 100): X=+6696.080 Y=-5577....
0
votes
2answers
63 views
awk Removing Double Quotes from CSV File Quoted Strings separated by comma
I have a csv file I would like to remove any rouge " or , in it. The problem is all my fields look something like this "***********","*********", where * = can be any ...
0
votes
0answers
28 views
Error filtering 2 files using AWK for a given condition
First of all, thank you for your help. I have a problem filtering 2 files using AWK conditionals. The two files I want to filter are these one: Fasta.fa
>SiiA lcl|NC_003197.2_prot_NP_463122....
0
votes
0answers
16 views
Get Characters up to the last given pattern
need help. I want to get all the CREATE statements on a file until ; but the only problem if there is a ; in the comment.. so I need to get the CREATE statement until the ); pattern...
....
CREATE ...
0
votes
1answer
63 views
Shuffle rows independently in a large file
I need to generate a random shuffle a very large csv (where I don't know in advance how many columns would be) in this way. So I have to go from this form
1,a,...
2,b,...
3,c,...
to something like ...
1
vote
2answers
23 views
Extract key from key-value pair
I have a file looking like that.
mke_version = "3.3.7"
msr_version = "2.8.5"
mcr_version = "19.03.14"
I want to print from file only key ...
0
votes
1answer
19 views
Combining two txt files containing name and last name on command line
I have two text files. First file have 20 Names listed in seperate lines and 2nd file have 20 surnames listed in seperated lines. How I can combine those files. So final text file have both first name ...
1
vote
1answer
18 views
Print sections in text file using awk and shell variables
I have a text file, materials.txt, with a large number of material definitions similar to this:
*FILESTART
line_a
line_b
line_c
*MAT_PLASTIC
1 property1 property2 property3
property4 ...
0
votes
0answers
59 views
How to slice array of awk-output in bash?
Summary
Depending on what bash or awk version I have, I get different results when I try to slice an array that holds awk output. On older versions of bash and awk the "slice operation" ( e....
0
votes
1answer
25 views
Replacing words with specified words using AWK
My code generates several json file all of which look similar to
{
"instance": "sample_instance_05",
"steps": [
{
"129": "North&...
13
votes
5answers
3k views
Add to every end of line in Notepad++
I have a long text file
gallery-dl -g -i w4b027.txt >
gallery-dl -g -i a4b028.txt >
gallery-dl -g -i b4b029.txt >
gallery-dl -g -i c4b030.txt >
gallery-dl -g -i d4b031.txt >
gallery-dl -...
0
votes
1answer
25 views
(awk) writing content between pattern 1 and second occurrence of pattern 2 into separate files
errors=$(mktemp -d) || exit 1
cat "$logFile" | awk -v d=$errors '/ ERROR /, /********************************************************************************/ {if ($0 ~ / ERROR /) {n++} ; ...
1
vote
0answers
44 views
grep and awk are not displaying output in terminal
When trying to grep Record in .txt file simply
$ grep 'Record' file.txt
$
it goes to the next line without any output or error (I've tried to write output in file, but it is empty)
I've tried also to ...
0
votes
1answer
24 views
awk multiple commands and multiple args
I'm trying to turn this output:
❯ hg branches -T'{node} {branch}\n'
4b0826975cb29da26a6dd13f463241463c43c6a7 maria
5c31751cdb774f8b6337b6f64232b15903530997 default
Into these commands:
hg bookmark -r ...
2
votes
1answer
69 views
Shell special chars blowing up regexp(s) in all: sed, (g)awk, and perl [closed]
I would like to change the PS1 in my ~/.bashrc programmatically with any of the the most popular regexp tools { sed, awk or perl }. However, I have problems with special characters. Note that in the ...
2
votes
2answers
112 views
Filtering a CSV and adding a new tag column in powershell
I have a CSV-file in.txt with 3 columns (Doc_Title, Doc_Date, Doc_URL) and I want to use PS to add another column Doc_Category using another CSV-file tag.txt with 2 columns (keyword, tag). I already ...
0
votes
1answer
25 views
Extract values following specific substrings in string containing spaces using awk
I have this single line string.
WordPress version: 5.5.3 Database revision: 48748 TinyMCE version: 4.9100 (49100-20200624) Package language: en_US
I would like to extract 5.5.3 and en_US
This is the ...
0
votes
3answers
119 views
find duplicate record in Text file
I have text file with this kind of structure
mv /XXX/20000/XXX-18245 /XXX/20000/XXX-28042
mv /XXX/10000/XXX-9942 /XXX/10000/XXX-18166
mv /XXX/10000/XXX-9962 /XXX/10000/XXX-18189
mv /XXX/20000/XXX-...
2
votes
2answers
198 views
Remove lines where either column is empty
I have a large colon-delimited text file containing two columns like this:
valueA:valueB
valueC:
valueD:valueE
:valueG
If there is no data to the left or right of the colon, I want to remove the ...
3
votes
1answer
204 views
Remove spaces from first column of delimited file
I have a very large colon-delimited file with two columns:
dhsj fdksjdk:dndkdlkfkd
shfdosod dkjsosjd:dkj dosjeod
dhswisoaidodisbw: dhsoi wjdos
fjfdododjfideefkjso sn:fhfdkd dkdjdf
If the first ...
0
votes
0answers
20 views
Reading output of shell command (awk)
I need advice on creating a script using awk, it's that I have a text file where I have a keyword and if it's at the beginning of a line, I move the whole line to the end of the line.
example:
end ...
0
votes
1answer
31 views
Transpose vertically with a catch
I've been working through some issues preparing data for logstash ingestion. I have a decent chunk of the process worked out but I'm wondering if there's a better way to do this. Basically I need to ...
2
votes
2answers
3k views
Why am I getting this error in my script? awk: script.awk:19: “ syntax error
I have 2 text files
1.txt
AA;00000;
BB;11111;
GG;22222;
2.txt
KK;WW;55555;11111;
KK;FF;ZZ;11111;
KK;RR;YY;11111;
I try to generate this 3.txt output:
AA;00000;
BB;11111;KK;WW;55555;KK;FF;ZZ;KK;RR;YY;...
1
vote
0answers
33 views
merging with format ;string;
I repost my question hoping to provide now a minimal reproducible example
Is basically doing a merge of the two files with format NAME;URL;?
Not exactly, string is not required to be a URL, it is ...
0
votes
0answers
43 views
When script carries common strings from rows, lines are truncated
I have this script
BEGIN {
OFS=FS=";"
}
ARGIND == 1 {
map[$2] = ($2 in map ? map[$2] OFS : "") $1
next
}
{
r = $0;
for (i=1; i<=NF; ++i)
if ($i in map)
...
0
votes
0answers
22 views
Cat awk command additional code implementation
I want to understand how to use awk cat command because I think to do not understand
In this page he suggest me to use his code
You should process g2.txt first like this:
cat join2.awk
BEGIN {
OFS=...
0
votes
0answers
26 views
Not the same output of small bash-script on different Linux systems
I have a small script, that makes JOIN of LSBLK and DF commands.
Script:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT | egrep -v "fd0|ram|loop|sr0|hdc|cdrom|[SWAP]" |
while read kname ...
0
votes
1answer
37 views
Helping with Join of output from DF and LSBLK Linux commands via bash
I already asked this question, but without update. I think this information should be related to this site.
I need to merge two outputs in Linux. This:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT |...
0
votes
1answer
39 views
How would I use sed and/or awk/grep to edit the line following a matched string?
I have a large text file that needs some changes. I need to do this by first locating lines that have a particular common string, and then editing the line directly after that. So for example, if I ...
0
votes
2answers
42 views
How to decrypt octal encoding from zfs diff output?
zfs diff replaces some of the characters in files names with \xxxx (in an octal encoding). Can I decrypt it back with awk or sed or any other common utility (without resorting to a perl or a python ...
0
votes
1answer
15 views
Tail log and output top processes upon encountering string fails
I am running the following command to tail a log file and output top processes to a file upon encountering a given string:
tail -f mylogfile.log | awk '
/server on fire/ { system("...
1
vote
1answer
52 views
Putty formatting
When I run the following one-liner, I get run-on output in PuTTY like so: (\ added below to make it easier to read, which is not included in the actual one-liner script)
fileName=10M.csv ; \
for k ...
0
votes
0answers
351 views
How to use variables in awk if condition in shell script?
When I tried to cut columns with key words from a csv file, I applied a variable from the for loop of a shell script in the awk if condition. But the if cannot recognize my string. So there's no ...
0
votes
1answer
30 views
grep for same occurrence more than once
Is it possible to grep or awk for more than 1 occurrence of the same string from a file?
awk '/$STRING|$STRING/' file
grep -e $STRING -e $STRING file
I tried other options but still not able to get ...
1
vote
1answer
56 views
AWK/SED/GREP - print M lines after the Nth occurrence of a match
I am looking to display M lines after the Nth occurrence of a match. Actually I need to do this in order to split a file containing Nth <regex> always followed by M lines into Nth different ...
-1
votes
2answers
12 views
How to remove 3 last numbers, seperated by space, in each line of a file - and remove everything exept said three numbers
I need to split a file containing data of this type;
Jazz Guitar 1 16 3 4
Carlos Guitar1 16 3 5
Overdrive Gtr1 16 3 6
Western Gtr&Whis 16 3 7
Pat Guitar Synth 16 3 8
Shadow Guitar 16 3 9
Clean El....
0
votes
1answer
32 views
The start of string token ^ is ignored into an awk script?
For learning purpose I made a very simple awk script.
It should find the name and the size of some binaries inside /usr/bin.
Here's an example of the script :
#!/usr/bin/awk -f
func round(n) {
...
2
votes
1answer
323 views
Remove all characters after first non-numeric character using sed or awk
TL;DR:
How can I create an expression using either sed or awk that will remove all characters after (and including) the first character that is not either numeric (i.e. [0-9]) or a period (.). I've ...
1
vote
2answers
40 views
Missing value when summing in awk
I have two-column files, with first column sorted, such that
0 107
1 1
1 141
2 22
3 1
3 222
4 65
5 1
5 53
6 79
I want an output with no repeated entries in the first column, and as the second column ...
0
votes
2answers
18 views
How to execute bash command on nth flat file column and print it out to output?
I need just basename cmd to be executed on 2'nd column
[ora@rac TEST]$ cat test.log | awk '{print $1 " " $2}'
9780c69e13e3dda64c3c6ddce2b37803 /test/test_202007140422_01.zip
...
0
votes
2answers
34 views
How can I change the end of a line depending on what is in next line?
I have a file full of text like:
CREATE TABLE `wild_adventure_game` (
`game_instance_id` bigint(20) unsigned NOT NULL,
`free_game_config_id` bigint(20) unsigned DEFAULT NULL,
`respin_instance_id`...
1
vote
2answers
67 views
pattern match special charecters and display contents between the pattern
My apache log format is as below
192.168.10.1 <"www.mytest.com"> <-"-"-> <--“192.168.10.90"--> [2020-07-10 03:35:46 PDT] >1008< 2470 3006 "https&...
2
votes
2answers
29 views
Combine strings with same first word preserving their order
I have a file that contain many lines consecutive and non consecutive with same first word and i want to merge all those lines.Not finding any way. please suggest someone.
eg.
cat file
X blah blah ...
1
vote
1answer
47 views
recursively sort files with string “SENT_” somewhere in filename by the substring immediately after SENT_, and then display them. Linux
So I wish to run this from any directory, say ~/ and have it work for subdirectories. Example would be;
ls ~/:
/home/me/FILES/DIR_1/DIR_a/SENT_2222_....
/home/me/FILES/DIR2/SENT_3333....
/home/me/...
2
votes
3answers
47 views
AWK is printing all columns
I want to isolate the second column.
If I do:
echo "1 2 3" | awk "{ print $2 }"
the result is:
1 2 3
The expected result is:
2
I've tried with -F' ' option, but the result is ...
0
votes
3answers
188 views
Bash case statement using awk commands in a for loop not working
I have the following bash script:
#!/bin/bash
OUTPUT=NULL,2,3,NULL,5,6,7,8
pvar1=$(echo $OUTPUT | awk -F ',' '{print $1}')
pvar2=$(echo $OUTPUT | awk -F ',' '{print $2}')
pvar3=$(echo $OUTPUT | awk -...
0
votes
2answers
34 views
How to change third column of a variable so it is encased with single quotation marks using AWK or SED
I have a variable $out:
out=1,2,3,4,5
I want to change the third column so the value gets single quotation marks as in (I talk of columns here as $out represents the values of an insert statement for ...
1
vote
2answers
2k views
How to replace a column value in a variable with a variable using awk
I have two variables:
var=x
out=1,2,3,4,5
I want to replace the third column in $out with $var. I am able to substitute x for 3 in column 3 with the following command (I speak of columns in $out as ...
