Questions tagged [awk]

AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.

Filter by
Sorted by
Tagged with
0 votes
1 answer
28 views

Summing real numbers from a text file

We have the following file: more /home/avrg.bc.txt 3.4 54.2 4.5 3.5 2.3 89.6 . . . We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
user avatar
0 votes
1 answer
8 views

keep each line containing x only if column y is the first containing x. remove the rest containing x

I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query. Using bash and bash tools like awk, sed, etc. I would ...
user avatar
0 votes
1 answer
31 views

Removing Duplicate from File

I have two files fileA and fileb FILEA 1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5 FILEB 4.4.4.4 2.2.2.2 Output should be 1.1.1.1 3.3.3.3 5.5.5.5 Size of File A is around 60GB and Size of File B is 1GB ...
user avatar
  • 15
0 votes
1 answer
21 views

Enclose the year in parenthesis in a filename

I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis. "Ghostbusters 3 2016.avi" ...
user avatar
2 votes
2 answers
50 views

Find and delete near duplicate lines with certain punctuation

I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
user avatar
  • 35
1 vote
1 answer
27 views

Extract last 5mins log from log file using shell

I need to extract last 5mins log from log file. Here is my log file [2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com....
user avatar
  • 11
0 votes
0 answers
44 views

using awk with single quote in a shell script will output error [duplicate]

we are on Debian 11. I'd like to use grep and awk to retrieve information from the content of a single file, like the following: file: /home/user/directory/otherdirectory/data.txt file content: $db['...
user avatar
-1 votes
2 answers
80 views

I have a list of searching word. How to replace them in another file?

I'm sure this question is asked multiple times, but I fail to find it. Basically I have a text file, and a list of all search strings (e.g. cat, dog, banana, etc). I want all occurrences of those ...
user avatar
  • 1,761
0 votes
1 answer
44 views

Printf in gawk with the correct encoding?

I'm wondering: can gawk printf in any format besides ASCII? Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
user avatar
  • 1
0 votes
2 answers
65 views

Remove the file name from a messy path in a text file awk or sed

I have an xml file with many lines containing something like this: Save="C:\Users\Administrator\Desktop\my files\1020\A54f\Drawing965.DWG" Module="0" Save="C:\Users\...
user avatar
  • 35
0 votes
0 answers
23 views

replace values of a column if it falls within the range of two values

i have a text file that contain four number of columns. I want to modify only the 4th column data with some conditions.The 4th column of the input file contains both negative and positive values data....
user avatar
0 votes
2 answers
44 views

How can erase first character of all string in file

How can erase first character of all string in file if this is zero? e.g I have file $ cat file 01111111 01222222 6666066 09999011 02222044 0743333 01000000 30000000 2222220 Try with sed command ...
user avatar
0 votes
0 answers
29 views

Using awk to find out number of users that spent less than an hour

The ouput of last -wi command is like xyz pts/0 24.131.154.199 Mon Apr 12 09:39 - 11:40 (02:01) I want to use awk to find out how many of users spent less than an hour Any ideas how to go ...
user avatar
  • 11
1 vote
0 answers
22 views

Awk stopping printing at "

I'm making an awk script to convert a csv of aliases into a shell script to apply them, but I'm having an issue where awk seems to be stopping at a " in the second field. The command I'm running ...
user avatar
  • 11
0 votes
2 answers
42 views

bash + gawk: set the number of the processed lines in CSV

As a part of my bash routine, I am using the following gawk code integrated to bash function which loop the CSVs and do their analysis: test_ranking44 () { for csv in "${rescore}"/*str*....
user avatar
  • 147
0 votes
1 answer
29 views

Post-processing of multi-column CSV: remove dublicate lines + sort

I am dealing with the csv produced by the concatenation (via cat) of several CSVs: ID(Prot), ID(lig), ID(cluster), dG(rescored), dG(before), POP(before) 1000, lig40, 1, 0.805136, -5....
user avatar
  • 147
0 votes
1 answer
142 views

how to replace commented line with sed

I want to replace below line with sed so I am trying below command which is giving me error "sed: -e expression #1, char 59: unknown option to `s'" I have not known to sed much can you guys ...
user avatar
0 votes
1 answer
303 views

Remove double quotes embracing variable strings from terminal

I have several files that contain this kind of line: from="$variable_name" and other stuff $variable name can change, so I could have from="$myArray" and other stuff from="$...
user avatar
  • 113
1 vote
1 answer
282 views

awk: compute min/max values for multi-column data using big number of input files

I am dealing with the post-processing of big number of csv data files located in the distinct directories. Each csv file has the following 3 column format: ID, POP, dG 1, 24, -6.8100 2, 22, -6.7900 3, ...
user avatar
  • 147
0 votes
1 answer
67 views

Using bash variables in the awk code

The following function written in bash incorporates AWK code carying out math operation on the multi-column data and eventually saves the results in the output file for all processed CSVs. home="$...
user avatar
  • 147
1 vote
1 answer
29 views

sed/awk: convertion of multi-column data

I am operating with multi-column data file procuded by some AWK script: # output.csv lig12, dG(rescored) 1, 0.596625 2, 1.05873 3, 1.11285 4, 0.697402 I need to modify this output.csv to one line ...
user avatar
  • 147
1 vote
1 answer
148 views

Use sed/awk to add specific text from one file to beginning of another file

I'm trying to use sed/awk to copy a range of text from the beginning of file1 to a specific line. Then add that output to the beginning of file 2 for a script I'm writing. The following command does ...
user avatar
  • 11
0 votes
2 answers
22 views

Extracting data from text file by specific characters

I have a program that reads from a sensor and prints on stdout. The interesting data for me is just the X,Y and Z values: [2021-05-07 10:47:41.993] [info] Acceleromter (avrg 100): X=+6696.080 Y=-5577....
user avatar
  • 105
0 votes
2 answers
309 views

awk Removing Double Quotes from CSV File Quoted Strings separated by comma

I have a csv file I would like to remove any rouge " or , in it. The problem is all my fields look something like this "***********","*********", where * = can be any ...
user avatar
  • 1
0 votes
1 answer
56 views

Error filtering 2 files using AWK for a given condition

First of all, thank you for your help. I have a problem filtering 2 files using AWK conditionals. The two files I want to filter are these one: Fasta.fa >SiiA lcl|NC_003197.2_prot_NP_463122....
user avatar
0 votes
1 answer
184 views

Shuffle rows independently in a large file

I need to generate a random shuffle a very large csv (where I don't know in advance how many columns would be) in this way. So I have to go from this form 1,a,... 2,b,... 3,c,... to something like ...
user avatar
  • 311
1 vote
2 answers
260 views

Extract key from key-value pair

I have a file looking like that. mke_version = "3.3.7" msr_version = "2.8.5" mcr_version = "19.03.14" I want to print from file only key ...
user avatar
  • 23
0 votes
1 answer
68 views

Combining two txt files containing name and last name on command line

I have two text files. First file have 20 Names listed in seperate lines and 2nd file have 20 surnames listed in seperated lines. How I can combine those files. So final text file have both first name ...
user avatar
  • 340
1 vote
1 answer
43 views

Print sections in text file using awk and shell variables

I have a text file, materials.txt, with a large number of material definitions similar to this: *FILESTART line_a line_b line_c *MAT_PLASTIC 1 property1 property2 property3 property4 ...
user avatar
  • 23
0 votes
0 answers
261 views

How to slice array of awk-output in bash?

Summary Depending on what bash or awk version I have, I get different results when I try to slice an array that holds awk output. On older versions of bash and awk the "slice operation" ( e....
user avatar
  • 101
0 votes
1 answer
119 views

Replacing words with specified words using AWK

My code generates several json file all of which look similar to { "instance": "sample_instance_05", "steps": [ { "129": "North&...
user avatar
14 votes
5 answers
4k views

Add to every end of line in Notepad++

I have a long text file gallery-dl -g -i w4b027.txt > gallery-dl -g -i a4b028.txt > gallery-dl -g -i b4b029.txt > gallery-dl -g -i c4b030.txt > gallery-dl -g -i d4b031.txt > gallery-dl -...
user avatar
  • 340
0 votes
1 answer
31 views

(awk) writing content between pattern 1 and second occurrence of pattern 2 into separate files

errors=$(mktemp -d) || exit 1 cat "$logFile" | awk -v d=$errors '/ ERROR /, /********************************************************************************/ {if ($0 ~ / ERROR /) {n++} ; ...
user avatar
  • 1
1 vote
0 answers
129 views

grep and awk are not displaying output in terminal

When trying to grep Record in .txt file simply $ grep 'Record' file.txt $ it goes to the next line without any output or error (I've tried to write output in file, but it is empty) I've tried also to ...
user avatar
  • 11
0 votes
1 answer
75 views

awk multiple commands and multiple args

I'm trying to turn this output: ❯ hg branches -T'{node} {branch}\n' 4b0826975cb29da26a6dd13f463241463c43c6a7 maria 5c31751cdb774f8b6337b6f64232b15903530997 default Into these commands: hg bookmark -r ...
user avatar
  • 11.3k
2 votes
1 answer
81 views

Shell special chars blowing up regexp(s) in all: sed, (g)awk, and perl [closed]

I would like to change the PS1 in my ~/.bashrc programmatically with any of the the most popular regexp tools { sed, awk or perl }. However, I have problems with special characters. Note that in the ...
user avatar
  • 21
2 votes
2 answers
268 views

Filtering a CSV and adding a new tag column in powershell

I have a CSV-file in.txt with 3 columns (Doc_Title, Doc_Date, Doc_URL) and I want to use PS to add another column Doc_Category using another CSV-file tag.txt with 2 columns (keyword, tag). I already ...
user avatar
  • 204
0 votes
1 answer
131 views

Extract values following specific substrings in string containing spaces using awk

I have this single line string. WordPress version: 5.5.3 Database revision: 48748 TinyMCE version: 4.9100 (49100-20200624) Package language: en_US I would like to extract 5.5.3 and en_US This is the ...
user avatar
0 votes
3 answers
4k views

find duplicate record in Text file

I have text file with this kind of structure mv /XXX/20000/XXX-18245 /XXX/20000/XXX-28042 mv /XXX/10000/XXX-9942 /XXX/10000/XXX-18166 mv /XXX/10000/XXX-9962 /XXX/10000/XXX-18189 mv /XXX/20000/XXX-...
user avatar
  • 35
2 votes
2 answers
217 views

Remove lines where either column is empty

I have a large colon-delimited text file containing two columns like this: valueA:valueB valueC: valueD:valueE :valueG If there is no data to the left or right of the colon, I want to remove the ...
user avatar
  • 10.3k
3 votes
1 answer
257 views

Remove spaces from first column of delimited file

I have a very large colon-delimited file with two columns: dhsj fdksjdk:dndkdlkfkd shfdosod dkjsosjd:dkj dosjeod dhswisoaidodisbw: dhsoi wjdos fjfdododjfideefkjso sn:fhfdkd dkdjdf If the first ...
user avatar
  • 10.3k
0 votes
1 answer
35 views

Transpose vertically with a catch

I've been working through some issues preparing data for logstash ingestion. I have a decent chunk of the process worked out but I'm wondering if there's a better way to do this. Basically I need to ...
user avatar
2 votes
2 answers
3k views

Why am I getting this error in my script? awk: script.awk:19: “ syntax error

I have 2 text files 1.txt AA;00000; BB;11111; GG;22222; 2.txt KK;WW;55555;11111; KK;FF;ZZ;11111; KK;RR;YY;11111; I try to generate this 3.txt output: AA;00000; BB;11111;KK;WW;55555;KK;FF;ZZ;KK;RR;YY;...
user avatar
1 vote
0 answers
36 views

merging with format ;string;

I repost my question hoping to provide now a minimal reproducible example Is basically doing a merge of the two files with format NAME;URL;? Not exactly, string is not required to be a URL, it is ...
user avatar
0 votes
0 answers
46 views

When script carries common strings from rows, lines are truncated

I have this script BEGIN { OFS=FS=";" } ARGIND == 1 { map[$2] = ($2 in map ? map[$2] OFS : "") $1 next } { r = $0; for (i=1; i<=NF; ++i) if ($i in map) ...
user avatar
  • 67
0 votes
0 answers
29 views

Cat awk command additional code implementation

I want to understand how to use awk cat command because I think to do not understand In this page he suggest me to use his code You should process g2.txt first like this: cat join2.awk BEGIN { OFS=...
user avatar
0 votes
0 answers
31 views

Not the same output of small bash-script on different Linux systems

I have a small script, that makes JOIN of LSBLK and DF commands. Script: lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT | egrep -v "fd0|ram|loop|sr0|hdc|cdrom|[SWAP]" | while read kname ...
user avatar
  • 153
0 votes
1 answer
79 views

Helping with Join of output from DF and LSBLK Linux commands via bash

I already asked this question, but without update. I think this information should be related to this site. I need to merge two outputs in Linux. This: lsblk -n -b --output KNAME,NAME,SIZE,MOUNTPOINT |...
user avatar
  • 153
0 votes
1 answer
66 views

How would I use sed and/or awk/grep to edit the line following a matched string?

I have a large text file that needs some changes. I need to do this by first locating lines that have a particular common string, and then editing the line directly after that. So for example, if I ...
user avatar
  • 1,258
0 votes
2 answers
79 views

How to decrypt octal encoding from zfs diff output?

zfs diff replaces some of the characters in files names with \xxxx (in an octal encoding). Can I decrypt it back with awk or sed or any other common utility (without resorting to a perl or a python ...
user avatar
  • 109

1
2 3 4 5
12