Questions tagged [awk]
AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.
615
questions
3
votes
2
answers
475
views
How do I delete every ocurrence of a word that is followed by another, without altering the whitespace?
Suppose my file looks like this:
foo bar
foo bar bar
foo foo bar foo
What I want to do is delete every foo that is followed by a bar (i.e. there is only whitespace between them), without ...
- 61
1
vote
0
answers
78
views
pipe awk output in sed command
I have multiple lines in a file names.txt, let's say
my name is Jim your name please
what is good Harry potter
how is he Jhony and me
and so on , more than 500 lines.
I use awk command to search/ ...
- 21
0
votes
0
answers
87
views
bash loop to create m3u file with sed awk grep
I am trying to auto create m3u file for my IPTV software with the help of sed/awk or grep command,but don't have much experience in this. expert help required to do this. working on Windows-10 PC ...
- 21
0
votes
2
answers
75
views
Are linux commands (grep, awk etc) part of the system, or the shell?
I'm a developer, so I know my way around a terminal, but I never really dove into it. I can write bash scripts (albeit on a beginner level, not my area of expertise), but I now need to understand ...
- 3
0
votes
1
answer
28
views
Linux - Find string and then end of section
I have an XMLTV file. I need to get all of the programme sections for a specific channel. An example snippet:
<programme start="2023031305000 -0400" end="2023031305300 -0400&...
0
votes
1
answer
43
views
AWK: "invalid regexp: Invalid collation character" -- how do I make it valid?
I have an awk script that must process millions of records, but I need to remove any containing a multibyte character.
In one environment where I work, the following simplified shell sequence ...
- 181
0
votes
1
answer
123
views
How to find the "pattern" in FASTA file and record the cordinates along with header
I am looking for a solution to search for a string of 17 base pairs in the FASTA file that is the human reference genome.
To clarify, in simple words and put the resources, i have tried grep function ...
- 3
1
vote
2
answers
125
views
Delete lines between two patterns (with multiple occurrences) around a third pattern (single occurrence)
I'm trying to edit an uncompressed PDF on macOS Monterey, and I want to delete objects that contain a certain pattern in their description. Such objects start with "X 0 obj" (where X is the ...
- 11
0
votes
1
answer
372
views
Awk exact string match in a specific column only returns header row
I have a 6-7M row CSV file that end users wish to review with Excel or Numbers. Each user only needs to review a subset of the file, so I was hoping to offer an awk one-liner that they can use until ...
- 105
0
votes
2
answers
61
views
How to group a flat list into categories and create a tab-separated value file using awk?
Crosspost: https://www.nixcraft.com/t/converting-a-list-into-a-tab-separated-file-grouped-by-values/4517
I have a text file with a list of values. The goal is to create a tab separated values file, ...
0
votes
3
answers
99
views
Get a valid path with random directory name from a text file
How to to get the valid path from a string?
For example I have the following text in file.txt that will always generate with a random path using the following patern:
Some error occurred. rerun the ...
- 103
0
votes
0
answers
51
views
Simplify NETSTAT monitoring output
I'm monitoring suspicious incoming connections using
sudo watch -n 30 "netstat -antp"
which provides more information than I need, so based on the answer I found here,
I've tried to ...
1
vote
1
answer
287
views
How to copy first line of text file to start or end of all another lines?
I am currently editing down multiple large text files (maybe 500 different text files) , and need to copy first line of text files to start or end of all another lines
For example this is one of my ...
- 19
-1
votes
2
answers
425
views
Convert CSV to text and automatically remove quotes and a specific word - Linux
I need to convert .csv to .txt format via CL (Command line) and automatically remove quotes and a specific word - Linux.
Example:
I have multiple exported csv files that have:
At first row I have: &...
- 1
0
votes
0
answers
15
views
How to discard empty fields after separating a string using awk?
For example, I want to grab "name", "age", and corresponding values from string{"name":"tom","age":11'}. Using command:
$ echo '{"name":&...
- 35
0
votes
0
answers
8
views
Bash, pattern substitution for json-like multiline string
I wrote a bash script that essentially turns df -a command into multiple, new-line separated, Json formatted objects.
PATTERN='!/loop/ && /^\// {printf \
"\"disk.device\":\"...
- 101
2
votes
0
answers
265
views
Find and replace multi-line block in multiple files (bash/linux)
I'm trying to find a way to find a text block that meets the criteria:
block starts with "google = {"
block ends with next "}" character found - ignoring everything else.
Example
...
- 21
0
votes
0
answers
279
views
Awk regex: how to remove all punctuation except when in the middle of a word?
Following information gathered on this thread, I wanted to use this code to fill my needs, however the code provided wasn't working for more than one punctuation character (e.g. '...'). I thus wanted ...
1
vote
1
answer
82
views
get the top urls from the logs
I have a numbers of log files:
adsfs.demo.com_2022-07-11-0000-0001_cn.tgz
adsfs.demo.com_2022-07-11-0000-0002_cn.tgz
adsfs.demo.com_2022-07-11-0000-0003_cn.tgz
adsfs.demo.com_2022-07-11-0000-...
- 525
0
votes
1
answer
43
views
Insert new line if file contain string
I need to insert 'import packagename' to the line after imports. In files which contains the specific word.
What I try:
find iOS/Chat \
-type f \
-name "*.swift" \
-exec gsed -e "/...
- 101
0
votes
1
answer
651
views
Inserting string between two known strings with sed or awk
I've used sed a bit before in the past for updating strings but I haven't been successful with inserting a string in between two known strings.
I attempted to use this but it didn't update anything.
...
- 103
0
votes
2
answers
2k
views
How to group and count all lines in a file that contain specific string
I want to filter all lines from a file that contain mySearchString and after that group them together and count them.
Example find all lines that contain 9791
AB-9791___Foo
AB-9791___Foo
DE-9791___Bar
...
- 275
0
votes
1
answer
118
views
Summing real numbers from a text file
We have the following file:
more /home/avrg.bc.txt
3.4
54.2
4.5
3.5
2.3
89.6
.
.
.
We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
- 723
0
votes
1
answer
17
views
keep each line containing x only if column y is the first containing x. remove the rest containing x
I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query.
Using bash and bash tools like awk, sed, etc. I would ...
- 11
0
votes
1
answer
75
views
Removing Duplicate from File
I have two files fileA and fileb
FILEA
1.1.1.1
2.2.2.2
3.3.3.3
4.4.4.4
5.5.5.5
FILEB
4.4.4.4
2.2.2.2
Output should be
1.1.1.1
3.3.3.3
5.5.5.5
Size of File A is around 60GB and Size of File B is 1GB
...
- 143
0
votes
1
answer
158
views
Enclose the year in parenthesis in a filename
I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis.
"Ghostbusters 3 2016.avi" ...
2
votes
2
answers
92
views
Find and delete near duplicate lines with certain punctuation
I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
- 35
1
vote
1
answer
632
views
Extract last 5mins log from log file using shell
I need to extract last 5mins log from log file.
Here is my log file
[2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com....
- 11
0
votes
0
answers
446
views
using awk with single quote in a shell script will output error [duplicate]
we are on Debian 11.
I'd like to use grep and awk to retrieve information from the content of a single file, like the following:
file: /home/user/directory/otherdirectory/data.txt
file content:
$db['...
-1
votes
2
answers
744
views
I have a list of searching word. How to replace them in another file?
I'm sure this question is asked multiple times, but I fail to find it. Basically I have a text file, and a list of all search strings (e.g. cat, dog, banana, etc). I want all occurrences of those ...
- 1,897
0
votes
1
answer
385
views
Printf in gawk with the correct encoding?
I'm wondering: can gawk printf in any format besides ASCII?
Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
- 1
0
votes
2
answers
452
views
Remove the file name from a messy path in a text file awk or sed
I have an xml file with many lines containing something like this:
Save="C:\Users\Administrator\Desktop\my files\1020\A54f\Drawing965.DWG" Module="0"
Save="C:\Users\...
- 35
0
votes
2
answers
295
views
How can erase first character of all string in file
How can erase first character of all string in file if this is zero?
e.g
I have file
$ cat file
01111111 01222222 6666066
09999011 02222044 0743333
01000000 30000000 2222220
Try with sed command ...
- 1
1
vote
0
answers
23
views
Awk stopping printing at "
I'm making an awk script to convert a csv of aliases into a shell script to apply them, but I'm having an issue where awk seems to be stopping at a " in the second field. The command I'm running ...
- 11
0
votes
2
answers
47
views
bash + gawk: set the number of the processed lines in CSV
As a part of my bash routine, I am using the following gawk code integrated to bash function which loop the CSVs and do their analysis:
test_ranking44 () {
for csv in "${rescore}"/*str*....
- 147
0
votes
1
answer
34
views
Post-processing of multi-column CSV: remove dublicate lines + sort
I am dealing with the csv produced by the concatenation (via cat) of several CSVs:
ID(Prot), ID(lig), ID(cluster), dG(rescored), dG(before), POP(before)
1000, lig40, 1, 0.805136, -5....
- 147
0
votes
1
answer
561
views
how to replace commented line with sed
I want to replace below line with sed so I am trying below command which is giving me error "sed: -e expression #1, char 59: unknown option to `s'"
I have not known to sed much can you guys ...
0
votes
1
answer
921
views
Remove double quotes embracing variable strings from terminal
I have several files that contain this kind of line:
from="$variable_name" and other stuff
$variable name can change, so I could have
from="$myArray" and other stuff
from="$...
- 123
1
vote
1
answer
1k
views
awk: compute min/max values for multi-column data using big number of input files
I am dealing with the post-processing of big number of csv data files located in the distinct directories. Each csv file has the following 3 column format:
ID, POP, dG
1, 24, -6.8100
2, 22, -6.7900
3, ...
- 147
0
votes
1
answer
91
views
Using bash variables in the awk code
The following function written in bash incorporates AWK code carying out math operation on the multi-column data and eventually saves the results in the output file for all processed CSVs.
home="$...
- 147
1
vote
1
answer
34
views
sed/awk: convertion of multi-column data
I am operating with multi-column data file procuded by some AWK script:
# output.csv
lig12, dG(rescored)
1, 0.596625
2, 1.05873
3, 1.11285
4, 0.697402
I need to modify this output.csv to one line ...
- 147
1
vote
1
answer
530
views
Use sed/awk to add specific text from one file to beginning of another file
I'm trying to use sed/awk to copy a range of text from the beginning of file1 to a specific line. Then add that output to the beginning of file 2 for a script I'm writing.
The following command does ...
- 11
0
votes
2
answers
208
views
Extracting data from text file by specific characters
I have a program that reads from a sensor and prints on stdout. The interesting data for me is just the X,Y and Z values:
[2021-05-07 10:47:41.993] [info] Acceleromter (avrg 100): X=+6696.080 Y=-5577....
- 135
0
votes
2
answers
1k
views
awk Removing Double Quotes from CSV File Quoted Strings separated by comma
I have a csv file I would like to remove any rouge " or , in it. The problem is all my fields look something like this "***********","*********", where * = can be any ...
- 1
0
votes
1
answer
233
views
Error filtering 2 files using AWK for a given condition
First of all, thank you for your help. I have a problem filtering 2 files using AWK conditionals. The two files I want to filter are these one: Fasta.fa
>SiiA lcl|NC_003197.2_prot_NP_463122....
- 1
0
votes
1
answer
327
views
Shuffle rows independently in a large file
I need to generate a random shuffle a very large csv (where I don't know in advance how many columns would be) in this way. So I have to go from this form
1,a,...
2,b,...
3,c,...
to something like ...
- 331
1
vote
2
answers
3k
views
Extract key from key-value pair
I have a file looking like that.
mke_version = "3.3.7"
msr_version = "2.8.5"
mcr_version = "19.03.14"
I want to print from file only key ...
- 23
0
votes
1
answer
282
views
Combining two txt files containing name and last name on command line
I have two text files. First file have 20 Names listed in seperate lines and 2nd file have 20 surnames listed in seperated lines. How I can combine those files. So final text file have both first name ...
- 344
1
vote
1
answer
263
views
Print sections in text file using awk and shell variables
I have a text file, materials.txt, with a large number of material definitions similar to this:
*FILESTART
line_a
line_b
line_c
*MAT_PLASTIC
1 property1 property2 property3
property4 ...
- 23
0
votes
0
answers
447
views
How to slice array of awk-output in bash?
Summary
Depending on what bash or awk version I have, I get different results when I try to slice an array that holds awk output. On older versions of bash and awk the "slice operation" ( e....
- 101