As of May 31, 2023, we have updated our Code of Conduct.

Questions tagged [awk]

AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.

Filter by
Sorted by
Tagged with
3 votes
2 answers
475 views

How do I delete every ocurrence of a word that is followed by another, without altering the whitespace?

Suppose my file looks like this: foo bar foo bar bar foo foo bar foo What I want to do is delete every foo that is followed by a bar (i.e. there is only whitespace between them), without ...
vim_overlord's user avatar
1 vote
0 answers
78 views

pipe awk output in sed command

I have multiple lines in a file names.txt, let's say my name is Jim your name please what is good Harry potter how is he Jhony and me and so on , more than 500 lines. I use awk command to search/ ...
Rizwan.A's user avatar
0 votes
0 answers
87 views

bash loop to create m3u file with sed awk grep

I am trying to auto create m3u file for my IPTV software with the help of sed/awk or grep command,but don't have much experience in this. expert help required to do this. working on Windows-10 PC ...
Rizwan.A's user avatar
0 votes
2 answers
75 views

Are linux commands (grep, awk etc) part of the system, or the shell?

I'm a developer, so I know my way around a terminal, but I never really dove into it. I can write bash scripts (albeit on a beginner level, not my area of expertise), but I now need to understand ...
romand's user avatar
  • 3
0 votes
1 answer
28 views

Linux - Find string and then end of section

I have an XMLTV file. I need to get all of the programme sections for a specific channel. An example snippet: <programme start="2023031305000 -0400" end="2023031305300 -0400&...
Dan Marcoux's user avatar
0 votes
1 answer
43 views

AWK: "invalid regexp: Invalid collation character" -- how do I make it valid?

I have an awk script that must process millions of records, but I need to remove any containing a multibyte character. In one environment where I work, the following simplified shell sequence ...
Kyle Banerjee's user avatar
0 votes
1 answer
123 views

How to find the "pattern" in FASTA file and record the cordinates along with header

I am looking for a solution to search for a string of 17 base pairs in the FASTA file that is the human reference genome. To clarify, in simple words and put the resources, i have tried grep function ...
Santosh's user avatar
1 vote
2 answers
125 views

Delete lines between two patterns (with multiple occurrences) around a third pattern (single occurrence)

I'm trying to edit an uncompressed PDF on macOS Monterey, and I want to delete objects that contain a certain pattern in their description. Such objects start with "X 0 obj" (where X is the ...
Lucuma13's user avatar
0 votes
1 answer
372 views

Awk exact string match in a specific column only returns header row

I have a 6-7M row CSV file that end users wish to review with Excel or Numbers. Each user only needs to review a subset of the file, so I was hoping to offer an awk one-liner that they can use until ...
Dan Wagner's user avatar
0 votes
2 answers
61 views

How to group a flat list into categories and create a tab-separated value file using awk?

Crosspost: https://www.nixcraft.com/t/converting-a-list-into-a-tab-separated-file-grouped-by-values/4517 I have a text file with a list of values. The goal is to create a tab separated values file, ...
substancev's user avatar
0 votes
3 answers
99 views

Get a valid path with random directory name from a text file

How to to get the valid path from a string? For example I have the following text in file.txt that will always generate with a random path using the following patern: Some error occurred. rerun the ...
Kalib Zen's user avatar
  • 103
0 votes
0 answers
51 views

Simplify NETSTAT monitoring output

I'm monitoring suspicious incoming connections using sudo watch -n 30 "netstat -antp" which provides more information than I need, so based on the answer I found here, I've tried to ...
marcusdavanco's user avatar
1 vote
1 answer
287 views

How to copy first line of text file to start or end of all another lines?

I am currently editing down multiple large text files (maybe 500 different text files) , and need to copy first line of text files to start or end of all another lines For example this is one of my ...
alex leiy's user avatar
-1 votes
2 answers
425 views

Convert CSV to text and automatically remove quotes and a specific word - Linux

I need to convert .csv to .txt format via CL (Command line) and automatically remove quotes and a specific word - Linux. Example: I have multiple exported csv files that have: At first row I have: &...
Marius's user avatar
  • 1
0 votes
0 answers
15 views

How to discard empty fields after separating a string using awk?

For example, I want to grab "name", "age", and corresponding values from string{"name":"tom","age":11'}. Using command: $ echo '{"name":&...
fiedel's user avatar
  • 35
0 votes
0 answers
8 views

Bash, pattern substitution for json-like multiline string

I wrote a bash script that essentially turns df -a command into multiple, new-line separated, Json formatted objects. PATTERN='!/loop/ && /^\// {printf \ "\"disk.device\":\"...
darkpirate's user avatar
2 votes
0 answers
265 views

Find and replace multi-line block in multiple files (bash/linux)

I'm trying to find a way to find a text block that meets the criteria: block starts with "google = {" block ends with next "}" character found - ignoring everything else. Example ...
jbl-toom's user avatar
0 votes
0 answers
279 views

Awk regex: how to remove all punctuation except when in the middle of a word?

Following information gathered on this thread, I wanted to use this code to fill my needs, however the code provided wasn't working for more than one punctuation character (e.g. '...'). I thus wanted ...
Michael Gauthier's user avatar
1 vote
1 answer
82 views

get the top urls from the logs

I have a numbers of log files: adsfs.demo.com_2022-07-11-0000-0001_cn.tgz adsfs.demo.com_2022-07-11-0000-0002_cn.tgz adsfs.demo.com_2022-07-11-0000-0003_cn.tgz adsfs.demo.com_2022-07-11-0000-...
244boy's user avatar
  • 525
0 votes
1 answer
43 views

Insert new line if file contain string

I need to insert 'import packagename' to the line after imports. In files which contains the specific word. What I try: find iOS/Chat \ -type f \ -name "*.swift" \ -exec gsed -e "/...
Bimawa's user avatar
  • 101
0 votes
1 answer
651 views

Inserting string between two known strings with sed or awk

I've used sed a bit before in the past for updating strings but I haven't been successful with inserting a string in between two known strings. I attempted to use this but it didn't update anything. ...
rk92's user avatar
  • 103
0 votes
2 answers
2k views

How to group and count all lines in a file that contain specific string

I want to filter all lines from a file that contain mySearchString and after that group them together and count them. Example find all lines that contain 9791 AB-9791___Foo AB-9791___Foo DE-9791___Bar ...
surfmuggle's user avatar
0 votes
1 answer
118 views

Summing real numbers from a text file

We have the following file: more /home/avrg.bc.txt 3.4 54.2 4.5 3.5 2.3 89.6 . . . We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
King David's user avatar
0 votes
1 answer
17 views

keep each line containing x only if column y is the first containing x. remove the rest containing x

I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query. Using bash and bash tools like awk, sed, etc. I would ...
DJ Champion's user avatar
0 votes
1 answer
75 views

Removing Duplicate from File

I have two files fileA and fileb FILEA 1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5 FILEB 4.4.4.4 2.2.2.2 Output should be 1.1.1.1 3.3.3.3 5.5.5.5 Size of File A is around 60GB and Size of File B is 1GB ...
ph3ro's user avatar
  • 143
0 votes
1 answer
158 views

Enclose the year in parenthesis in a filename

I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis. "Ghostbusters 3 2016.avi" ...
user88710's user avatar
2 votes
2 answers
92 views

Find and delete near duplicate lines with certain punctuation

I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
Priora's user avatar
  • 35
1 vote
1 answer
632 views

Extract last 5mins log from log file using shell

I need to extract last 5mins log from log file. Here is my log file [2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com....
Anirudh's user avatar
  • 11
0 votes
0 answers
446 views

using awk with single quote in a shell script will output error [duplicate]

we are on Debian 11. I'd like to use grep and awk to retrieve information from the content of a single file, like the following: file: /home/user/directory/otherdirectory/data.txt file content: $db['...
iodurocarburo's user avatar
-1 votes
2 answers
744 views

I have a list of searching word. How to replace them in another file?

I'm sure this question is asked multiple times, but I fail to find it. Basically I have a text file, and a list of all search strings (e.g. cat, dog, banana, etc). I want all occurrences of those ...
Ooker's user avatar
  • 1,897
0 votes
1 answer
385 views

Printf in gawk with the correct encoding?

I'm wondering: can gawk printf in any format besides ASCII? Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
ixns's user avatar
  • 1
0 votes
2 answers
452 views

Remove the file name from a messy path in a text file awk or sed

I have an xml file with many lines containing something like this: Save="C:\Users\Administrator\Desktop\my files\1020\A54f\Drawing965.DWG" Module="0" Save="C:\Users\...
Priora's user avatar
  • 35
0 votes
2 answers
295 views

How can erase first character of all string in file

How can erase first character of all string in file if this is zero? e.g I have file $ cat file 01111111 01222222 6666066 09999011 02222044 0743333 01000000 30000000 2222220 Try with sed command ...
aav1307's user avatar
1 vote
0 answers
23 views

Awk stopping printing at "

I'm making an awk script to convert a csv of aliases into a shell script to apply them, but I'm having an issue where awk seems to be stopping at a " in the second field. The command I'm running ...
dwdwdan's user avatar
  • 11
0 votes
2 answers
47 views

bash + gawk: set the number of the processed lines in CSV

As a part of my bash routine, I am using the following gawk code integrated to bash function which loop the CSVs and do their analysis: test_ranking44 () { for csv in "${rescore}"/*str*....
Hot JAMS's user avatar
  • 147
0 votes
1 answer
34 views

Post-processing of multi-column CSV: remove dublicate lines + sort

I am dealing with the csv produced by the concatenation (via cat) of several CSVs: ID(Prot), ID(lig), ID(cluster), dG(rescored), dG(before), POP(before) 1000, lig40, 1, 0.805136, -5....
Hot JAMS's user avatar
  • 147
0 votes
1 answer
561 views

how to replace commented line with sed

I want to replace below line with sed so I am trying below command which is giving me error "sed: -e expression #1, char 59: unknown option to `s'" I have not known to sed much can you guys ...
user1373123's user avatar
0 votes
1 answer
921 views

Remove double quotes embracing variable strings from terminal

I have several files that contain this kind of line: from="$variable_name" and other stuff $variable name can change, so I could have from="$myArray" and other stuff from="$...
splunk's user avatar
  • 123
1 vote
1 answer
1k views

awk: compute min/max values for multi-column data using big number of input files

I am dealing with the post-processing of big number of csv data files located in the distinct directories. Each csv file has the following 3 column format: ID, POP, dG 1, 24, -6.8100 2, 22, -6.7900 3, ...
Hot JAMS's user avatar
  • 147
0 votes
1 answer
91 views

Using bash variables in the awk code

The following function written in bash incorporates AWK code carying out math operation on the multi-column data and eventually saves the results in the output file for all processed CSVs. home="$...
Hot JAMS's user avatar
  • 147
1 vote
1 answer
34 views

sed/awk: convertion of multi-column data

I am operating with multi-column data file procuded by some AWK script: # output.csv lig12, dG(rescored) 1, 0.596625 2, 1.05873 3, 1.11285 4, 0.697402 I need to modify this output.csv to one line ...
Hot JAMS's user avatar
  • 147
1 vote
1 answer
530 views

Use sed/awk to add specific text from one file to beginning of another file

I'm trying to use sed/awk to copy a range of text from the beginning of file1 to a specific line. Then add that output to the beginning of file 2 for a script I'm writing. The following command does ...
grizzly's user avatar
  • 11
0 votes
2 answers
208 views

Extracting data from text file by specific characters

I have a program that reads from a sensor and prints on stdout. The interesting data for me is just the X,Y and Z values: [2021-05-07 10:47:41.993] [info] Acceleromter (avrg 100): X=+6696.080 Y=-5577....
DEKKER's user avatar
  • 135
0 votes
2 answers
1k views

awk Removing Double Quotes from CSV File Quoted Strings separated by comma

I have a csv file I would like to remove any rouge " or , in it. The problem is all my fields look something like this "***********","*********", where * = can be any ...
Alex's user avatar
  • 1
0 votes
1 answer
233 views

Error filtering 2 files using AWK for a given condition

First of all, thank you for your help. I have a problem filtering 2 files using AWK conditionals. The two files I want to filter are these one: Fasta.fa >SiiA lcl|NC_003197.2_prot_NP_463122....
quik1399's user avatar
0 votes
1 answer
327 views

Shuffle rows independently in a large file

I need to generate a random shuffle a very large csv (where I don't know in advance how many columns would be) in this way. So I have to go from this form 1,a,... 2,b,... 3,c,... to something like ...
emanuele's user avatar
  • 331
1 vote
2 answers
3k views

Extract key from key-value pair

I have a file looking like that. mke_version = "3.3.7" msr_version = "2.8.5" mcr_version = "19.03.14" I want to print from file only key ...
Lotarc's user avatar
  • 23
0 votes
1 answer
282 views

Combining two txt files containing name and last name on command line

I have two text files. First file have 20 Names listed in seperate lines and 2nd file have 20 surnames listed in seperated lines. How I can combine those files. So final text file have both first name ...
Roxion's user avatar
  • 344
1 vote
1 answer
263 views

Print sections in text file using awk and shell variables

I have a text file, materials.txt, with a large number of material definitions similar to this: *FILESTART line_a line_b line_c *MAT_PLASTIC 1 property1 property2 property3 property4 ...
Coporob's user avatar
  • 23
0 votes
0 answers
447 views

How to slice array of awk-output in bash?

Summary Depending on what bash or awk version I have, I get different results when I try to slice an array that holds awk output. On older versions of bash and awk the "slice operation" ( e....
gomfy's user avatar
  • 101

1
2 3 4 5
13