Questions tagged [awk]
AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.
595
questions
0
votes
1
answer
28
views
Summing real numbers from a text file
We have the following file:
more /home/avrg.bc.txt
3.4
54.2
4.5
3.5
2.3
89.6
.
.
.
We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
- 575
0
votes
1
answer
8
views
keep each line containing x only if column y is the first containing x. remove the rest containing x
I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query.
Using bash and bash tools like awk, sed, etc. I would ...
- 11
0
votes
1
answer
31
views
Removing Duplicate from File
I have two files fileA and fileb
FILEA
1.1.1.1
2.2.2.2
3.3.3.3
4.4.4.4
5.5.5.5
FILEB
4.4.4.4
2.2.2.2
Output should be
1.1.1.1
3.3.3.3
5.5.5.5
Size of File A is around 60GB and Size of File B is 1GB
...
- 15
0
votes
1
answer
21
views
Enclose the year in parenthesis in a filename
I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis.
"Ghostbusters 3 2016.avi" ...
2
votes
2
answers
50
views
Find and delete near duplicate lines with certain punctuation
I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
- 35
1
vote
1
answer
27
views
Extract last 5mins log from log file using shell
I need to extract last 5mins log from log file.
Here is my log file
[2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com....
- 11
0
votes
0
answers
44
views
using awk with single quote in a shell script will output error [duplicate]
we are on Debian 11.
I'd like to use grep and awk to retrieve information from the content of a single file, like the following:
file: /home/user/directory/otherdirectory/data.txt
file content:
$db['...
-1
votes
2
answers
80
views
I have a list of searching word. How to replace them in another file?
I'm sure this question is asked multiple times, but I fail to find it. Basically I have a text file, and a list of all search strings (e.g. cat, dog, banana, etc). I want all occurrences of those ...
- 1,761
0
votes
1
answer
44
views
Printf in gawk with the correct encoding?
I'm wondering: can gawk printf in any format besides ASCII?
Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
- 1
0
votes
2
answers
65
views
Remove the file name from a messy path in a text file awk or sed
I have an xml file with many lines containing something like this:
Save="C:\Users\Administrator\Desktop\my files\1020\A54f\Drawing965.DWG" Module="0"
Save="C:\Users\...
- 35
0
votes
0
answers
23
views
replace values of a column if it falls within the range of two values
i have a text file that contain four number of columns. I want to modify only the 4th column data with some conditions.The 4th column of the input file contains both negative and positive values data....
- 1
0
votes
2
answers
44
views
How can erase first character of all string in file
How can erase first character of all string in file if this is zero?
e.g
I have file
$ cat file
01111111 01222222 6666066
09999011 02222044 0743333
01000000 30000000 2222220
Try with sed command ...
- 1
0
votes
0
answers
29
views
Using awk to find out number of users that spent less than an hour
The ouput of last -wi command is like
xyz pts/0 24.131.154.199 Mon Apr 12 09:39 - 11:40 (02:01)
I want to use awk to find out how many of users spent less than an hour
Any ideas how to go ...
- 11
1
vote
0
answers
22
views
Awk stopping printing at "
I'm making an awk script to convert a csv of aliases into a shell script to apply them, but I'm having an issue where awk seems to be stopping at a " in the second field. The command I'm running ...
- 11
0
votes
2
answers
42
views
bash + gawk: set the number of the processed lines in CSV
As a part of my bash routine, I am using the following gawk code integrated to bash function which loop the CSVs and do their analysis:
test_ranking44 () {
for csv in "${rescore}"/*str*....
- 147
0
votes
1
answer
29
views
Post-processing of multi-column CSV: remove dublicate lines + sort
I am dealing with the csv produced by the concatenation (via cat) of several CSVs:
ID(Prot), ID(lig), ID(cluster), dG(rescored), dG(before), POP(before)
1000, lig40, 1, 0.805136, -5....
- 147
0
votes
1
answer
142
views
how to replace commented line with sed
I want to replace below line with sed so I am trying below command which is giving me error "sed: -e expression #1, char 59: unknown option to `s'"
I have not known to sed much can you guys ...
0
votes
1
answer
303
views
Remove double quotes embracing variable strings from terminal
I have several files that contain this kind of line:
from="$variable_name" and other stuff
$variable name can change, so I could have
from="$myArray" and other stuff
from="$...
- 113
1
vote
1
answer
282
views
awk: compute min/max values for multi-column data using big number of input files
I am dealing with the post-processing of big number of csv data files located in the distinct directories. Each csv file has the following 3 column format:
ID, POP, dG
1, 24, -6.8100
2, 22, -6.7900
3, ...
- 147
0
votes
1
answer
67
views
Using bash variables in the awk code
The following function written in bash incorporates AWK code carying out math operation on the multi-column data and eventually saves the results in the output file for all processed CSVs.
home="$...
- 147
1
vote
1
answer
29
views
sed/awk: convertion of multi-column data
I am operating with multi-column data file procuded by some AWK script:
# output.csv
lig12, dG(rescored)
1, 0.596625
2, 1.05873
3, 1.11285
4, 0.697402
I need to modify this output.csv to one line ...
- 147
1
vote
1
answer
148
views
Use sed/awk to add specific text from one file to beginning of another file
I'm trying to use sed/awk to copy a range of text from the beginning of file1 to a specific line. Then add that output to the beginning of file 2 for a script I'm writing.
The following command does ...
- 11
0
votes
2
answers
22
views
Extracting data from text file by specific characters
I have a program that reads from a sensor and prints on stdout. The interesting data for me is just the X,Y and Z values:
[2021-05-07 10:47:41.993] [info] Acceleromter (avrg 100): X=+6696.080 Y=-5577....
- 105
0
votes
2
answers
309
views
awk Removing Double Quotes from CSV File Quoted Strings separated by comma
I have a csv file I would like to remove any rouge " or , in it. The problem is all my fields look something like this "***********","*********", where * = can be any ...
- 1
0
votes
1
answer
56
views
Error filtering 2 files using AWK for a given condition
First of all, thank you for your help. I have a problem filtering 2 files using AWK conditionals. The two files I want to filter are these one: Fasta.fa
>SiiA lcl|NC_003197.2_prot_NP_463122....
- 1
0
votes
1
answer
184
views
Shuffle rows independently in a large file
I need to generate a random shuffle a very large csv (where I don't know in advance how many columns would be) in this way. So I have to go from this form
1,a,...
2,b,...
3,c,...
to something like ...
- 311
1
vote
2
answers
260
views
Extract key from key-value pair
I have a file looking like that.
mke_version = "3.3.7"
msr_version = "2.8.5"
mcr_version = "19.03.14"
I want to print from file only key ...
- 23
0
votes
1
answer
68
views
Combining two txt files containing name and last name on command line
I have two text files. First file have 20 Names listed in seperate lines and 2nd file have 20 surnames listed in seperated lines. How I can combine those files. So final text file have both first name ...
- 340
1
vote
1
answer
43
views
Print sections in text file using awk and shell variables
I have a text file, materials.txt, with a large number of material definitions similar to this:
*FILESTART
line_a
line_b
line_c
*MAT_PLASTIC
1 property1 property2 property3
property4 ...
- 23
0
votes
0
answers
261
views
How to slice array of awk-output in bash?
Summary
Depending on what bash or awk version I have, I get different results when I try to slice an array that holds awk output. On older versions of bash and awk the "slice operation" ( e....
- 101
0
votes
1
answer
119
views
Replacing words with specified words using AWK
My code generates several json file all of which look similar to
{
"instance": "sample_instance_05",
"steps": [
{
"129": "North&...
- 1,651
14
votes
5
answers
4k
views
Add to every end of line in Notepad++
I have a long text file
gallery-dl -g -i w4b027.txt >
gallery-dl -g -i a4b028.txt >
gallery-dl -g -i b4b029.txt >
gallery-dl -g -i c4b030.txt >
gallery-dl -g -i d4b031.txt >
gallery-dl -...
- 340
0
votes
1
answer
31
views
(awk) writing content between pattern 1 and second occurrence of pattern 2 into separate files
errors=$(mktemp -d) || exit 1
cat "$logFile" | awk -v d=$errors '/ ERROR /, /********************************************************************************/ {if ($0 ~ / ERROR /) {n++} ; ...
- 1
1
vote
0
answers
129
views
grep and awk are not displaying output in terminal
When trying to grep Record in .txt file simply
$ grep 'Record' file.txt
$
it goes to the next line without any output or error (I've tried to write output in file, but it is empty)
I've tried also to ...
- 11
0
votes
1
answer
75
views
awk multiple commands and multiple args
I'm trying to turn this output:
❯ hg branches -T'{node} {branch}\n'
4b0826975cb29da26a6dd13f463241463c43c6a7 maria
5c31751cdb774f8b6337b6f64232b15903530997 default
Into these commands:
hg bookmark -r ...
- 11.3k
2
votes
1
answer
81
views
Shell special chars blowing up regexp(s) in all: sed, (g)awk, and perl [closed]
I would like to change the PS1 in my ~/.bashrc programmatically with any of the the most popular regexp tools { sed, awk or perl }. However, I have problems with special characters. Note that in the ...
- 21
2
votes
2
answers
268
views
Filtering a CSV and adding a new tag column in powershell
I have a CSV-file in.txt with 3 columns (Doc_Title, Doc_Date, Doc_URL) and I want to use PS to add another column Doc_Category using another CSV-file tag.txt with 2 columns (keyword, tag). I already ...
- 204
0
votes
1
answer
131
views
Extract values following specific substrings in string containing spaces using awk
I have this single line string.
WordPress version: 5.5.3 Database revision: 48748 TinyMCE version: 4.9100 (49100-20200624) Package language: en_US
I would like to extract 5.5.3 and en_US
This is the ...
- 101
0
votes
3
answers
4k
views
find duplicate record in Text file
I have text file with this kind of structure
mv /XXX/20000/XXX-18245 /XXX/20000/XXX-28042
mv /XXX/10000/XXX-9942 /XXX/10000/XXX-18166
mv /XXX/10000/XXX-9962 /XXX/10000/XXX-18189
mv /XXX/20000/XXX-...
- 35
2
votes
2
answers
217
views
Remove lines where either column is empty
I have a large colon-delimited text file containing two columns like this:
valueA:valueB
valueC:
valueD:valueE
:valueG
If there is no data to the left or right of the colon, I want to remove the ...
- 10.3k
3
votes
1
answer
257
views
Remove spaces from first column of delimited file
I have a very large colon-delimited file with two columns:
dhsj fdksjdk:dndkdlkfkd
shfdosod dkjsosjd:dkj dosjeod
dhswisoaidodisbw: dhsoi wjdos
fjfdododjfideefkjso sn:fhfdkd dkdjdf
If the first ...
- 10.3k
0
votes
1
answer
35
views
Transpose vertically with a catch
I've been working through some issues preparing data for logstash ingestion. I have a decent chunk of the process worked out but I'm wondering if there's a better way to do this. Basically I need to ...
2
votes
2
answers
3k
views
Why am I getting this error in my script? awk: script.awk:19: “ syntax error
I have 2 text files
1.txt
AA;00000;
BB;11111;
GG;22222;
2.txt
KK;WW;55555;11111;
KK;FF;ZZ;11111;
KK;RR;YY;11111;
I try to generate this 3.txt output:
AA;00000;
BB;11111;KK;WW;55555;KK;FF;ZZ;KK;RR;YY;...
- 47
1
vote
0
answers
36
views
merging with format ;string;
I repost my question hoping to provide now a minimal reproducible example
Is basically doing a merge of the two files with format NAME;URL;?
Not exactly, string is not required to be a URL, it is ...
- 53
0
votes
0
answers
46
views
When script carries common strings from rows, lines are truncated
I have this script
BEGIN {
OFS=FS=";"
}
ARGIND == 1 {
map[$2] = ($2 in map ? map[$2] OFS : "") $1
next
}
{
r = $0;
for (i=1; i<=NF; ++i)
if ($i in map)
...
- 67
0
votes
0
answers
29
views
Cat awk command additional code implementation
I want to understand how to use awk cat command because I think to do not understand
In this page he suggest me to use his code
You should process g2.txt first like this:
cat join2.awk
BEGIN {
OFS=...
- 47
0
votes
0
answers
31
views
Not the same output of small bash-script on different Linux systems
I have a small script, that makes JOIN of LSBLK and DF commands.
Script:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT | egrep -v "fd0|ram|loop|sr0|hdc|cdrom|[SWAP]" |
while read kname ...
- 153
0
votes
1
answer
79
views
Helping with Join of output from DF and LSBLK Linux commands via bash
I already asked this question, but without update. I think this information should be related to this site.
I need to merge two outputs in Linux. This:
lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT |...
- 153
0
votes
1
answer
66
views
How would I use sed and/or awk/grep to edit the line following a matched string?
I have a large text file that needs some changes. I need to do this by first locating lines that have a particular common string, and then editing the line directly after that. So for example, if I ...
- 1,258
0
votes
2
answers
79
views
How to decrypt octal encoding from zfs diff output?
zfs diff replaces some of the characters in files names with \xxxx (in an octal encoding). Can I decrypt it back with awk or sed or any other common utility (without resorting to a perl or a python ...
- 109