Skip to main content

Questions tagged [awk]

AWK is a text-processing language. It is mainly used to manipulate and process structured data, and to generate reports.

Filter by
Sorted by
Tagged with
0 votes
2 answers
105 views

bash + how to insert date before line

lets say I want to pipe the date output in the beggining of some text for example echo "this line is test line" | date and expected output should be Wed May 22 14:55:10 UTC 2024 this line ...
King David's user avatar
0 votes
2 answers
54 views

How to extract a text in between two patterns (potentially containing floating lookbehind) with only one invocation?

I'm trying to match any characters (except line break characters) in between xxx<1+ whitespaces>" and whitespace. The format of the input string is the following: xxx<1+ whitespaces>&...
pmor's user avatar
  • 368
0 votes
0 answers
20 views

Read git label in version format on Windows

I have the following bash script which read the last label from git which is in version format (1.2.3.4). git tag --sort=-refname | awk 'match($0, /^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$/)' | head -n 1 I ...
Tomas Kubes's user avatar
1 vote
3 answers
79 views

Awk command to find the most common word in a file?

Lets say I have a file that is simply a list full of dates and names, And I need to look for the "team name" that is repeated most in that list? Player name is located at position $1 and ...
Andre Freitas's user avatar
2 votes
1 answer
302 views

How to Print, when columns have spaces - awk?

Problem: desired columns have text with spaces SAMSUNG MZNLN128HCGR-000H1 undesired columns have text with spaces, see VENDOR = M.2 SSD awk only printing the first word. awk not printing the column ...
joseph22's user avatar
  • 367
0 votes
1 answer
68 views

AWK replace some special new line to one line

i have some log like this: 2023-11-15T08:59:28.000000+00:00 database-1 # Time: 231115 8:59:28 # User@Host: rdsadmin[rdsadmin] @ localhost [] # Thread_id: 3 Schema: QC_hit: No # Query_time: 0....
dvthanh's user avatar
0 votes
2 answers
457 views

awk string replacement - properly escape '&' character

I am working on one of our applications to update the docker image from using static configs for each environment to a single template using dynamic variable insertion with awk. Our Dockerfile calls a ...
kat.is.konfused's user avatar
2 votes
2 answers
335 views

I want to get every command that has 3 letters and only three letters

I don't get the output I expect using apropos -k . |awk '/^.../ {print $1}' am I completely overthinking this? In this sample output only zsh should have come back. I also tried this apropos -k . | ...
Mark Scheck's user avatar
0 votes
0 answers
150 views

Convert a 1 row (comma separated) text file to multiple rows

How can I convert text files with the following format: ["word01","word02","word03","word04","word05","word06","word07","...
pablorosatti's user avatar
0 votes
1 answer
577 views

Moving files to a directory with spaces in its name

I'm trying to create a directory that has in its name the beginning of the names of the files it contains. I need this in order to archive it later. I have run into problems when moving files if the ...
user avatar
1 vote
1 answer
98 views

How to parse upload and download speed with ifstat and printf?

In bash I'm trying to display rounded (no decimals) download and upload speed in readable format i.e something like that: Download 16KB Upload 200KB I got this but its not quite working: ifstat 1 1 |...
okoolo's user avatar
  • 15
0 votes
2 answers
1k views

Find and replace string in a file using 'sed' or 'awk'

Let's say I have the following input file: yhara.runner.mng1.nna1.X9HCJG.1 yhara.runner.mng1.nna1.Z2HCJG.1 100.000 100 yhara.runner.mng1.nna1.X9HCJG.1 yhara.runner.mng1.nna1.AUEM0K.1 94.144 72 ...
denny's user avatar
  • 103
0 votes
1 answer
559 views

How can I combine "Search for multiple strings (patterns)" with "Print rows after a hit" in a string/search and output a common file?

I am using the windows version of grep "GNU Grep". syntax: grep [OPTIONS] PATTERN [FILE...] How can I combine different options with different patterns? Both lines listed here work on their ...
minu's user avatar
  • 3
0 votes
1 answer
336 views

Loop with awk output field

the output of my script grep 'Interface GigabitEthernet' * | \ sed 's/104-8-20-23.lightspeed.gnvlsc.sbcglobal.net/104.8.20.23/g' | \ sed 's/0.0.0.0: //g' | \ sed 's/.log:/ /g' | \ sed 's/Line protocol ...
Lafayette's user avatar
2 votes
0 answers
48 views

SFTP user login details real-time filtering

I have enabled the SFTP login log into the default logfile /var/log/syslog and tried to filter the login time of each user and insert it into the database. But the filtering is not worked as I ...
Amal P Ramesh's user avatar
1 vote
2 answers
309 views

Multiply numerical value after a string in a json file by 3?

I'm trying to replace a number in a set of text files. The number will be different for each file, but it will always occur after a specific string. Here's the original file: "structures&...
Herdo's user avatar
  • 11
3 votes
4 answers
332 views

How to remove all newlines from a file?

I have the command cat file.txt | gawk '{print $2}' > test which displays: 1 1 0 0 1 1 Afterwards: vi test :%s/\n// Which outputs: 110011 My question is I don't know how I could incorporate this ...
Antonio De Angelis's user avatar
-1 votes
1 answer
169 views

write a efficient for loop in bash for multiple files (.PDB files)

I have problem with this script: #!/bin/bash for filename in /home/hb/pg/ensemble/pdbs/P24941/raw_pdb/*.pdb; do grep COMPND $filename | grep "CHAIN:" -B 1 | sed 's/COMPND//g' | sed 's/...
shahed's user avatar
  • 9
3 votes
3 answers
576 views

How do I delete every ocurrence of a word that is followed by another, without altering the whitespace?

Suppose my file looks like this: foo bar foo bar bar foo foo bar foo What I want to do is delete every foo that is followed by a bar (i.e. there is only whitespace between them), without ...
vim_overlord's user avatar
1 vote
0 answers
239 views

pipe awk output in sed command

I have multiple lines in a file names.txt, let's say my name is Jim your name please what is good Harry potter how is he Jhony and me and so on , more than 500 lines. I use awk command to search/ ...
Rizwan.A's user avatar
0 votes
0 answers
258 views

bash loop to create m3u file with sed awk grep

I am trying to auto create m3u file for my IPTV software with the help of sed/awk or grep command,but don't have much experience in this. expert help required to do this. working on Windows-10 PC ...
Rizwan.A's user avatar
0 votes
2 answers
529 views

Are linux commands (grep, awk etc) part of the system, or the shell?

I'm a developer, so I know my way around a terminal, but I never really dove into it. I can write bash scripts (albeit on a beginner level, not my area of expertise), but I now need to understand ...
romand's user avatar
  • 3
0 votes
1 answer
54 views

Linux - Find string and then end of section

I have an XMLTV file. I need to get all of the programme sections for a specific channel. An example snippet: <programme start="2023031305000 -0400" end="2023031305300 -0400&...
Dan Marcoux's user avatar
0 votes
1 answer
196 views

AWK: "invalid regexp: Invalid collation character" -- how do I make it valid?

I have an awk script that must process millions of records, but I need to remove any containing a multibyte character. In one environment where I work, the following simplified shell sequence ...
Kyle Banerjee's user avatar
1 vote
1 answer
398 views

How to find the "pattern" in FASTA file and record the cordinates along with header

I am looking for a solution to search for a string of 17 base pairs in the FASTA file that is the human reference genome. To clarify, in simple words and put the resources, i have tried grep function ...
Santosh's user avatar
  • 13
1 vote
2 answers
590 views

Delete lines between two patterns (with multiple occurrences) around a third pattern (single occurrence)

I'm trying to edit an uncompressed PDF on macOS Monterey, and I want to delete objects that contain a certain pattern in their description. Such objects start with "X 0 obj" (where X is the ...
Lucuma13's user avatar
0 votes
1 answer
1k views

Awk exact string match in a specific column only returns header row

I have a 6-7M row CSV file that end users wish to review with Excel or Numbers. Each user only needs to review a subset of the file, so I was hoping to offer an awk one-liner that they can use until ...
Dan Wagner's user avatar
0 votes
2 answers
166 views

How to group a flat list into categories and create a tab-separated value file using awk?

Crosspost: https://www.nixcraft.com/t/converting-a-list-into-a-tab-separated-file-grouped-by-values/4517 I have a text file with a list of values. The goal is to create a tab separated values file, ...
substancev's user avatar
0 votes
3 answers
258 views

Get a valid path with random directory name from a text file

How to to get the valid path from a string? For example I have the following text in file.txt that will always generate with a random path using the following patern: Some error occurred. rerun the ...
Kalib Zen's user avatar
  • 103
0 votes
0 answers
179 views

Simplify NETSTAT monitoring output

I'm monitoring suspicious incoming connections using sudo watch -n 30 "netstat -antp" which provides more information than I need, so based on the answer I found here, I've tried to ...
marcusdavanco's user avatar
1 vote
1 answer
451 views

How to copy first line of text file to start or end of all another lines?

I am currently editing down multiple large text files (maybe 500 different text files) , and need to copy first line of text files to start or end of all another lines For example this is one of my ...
alex leiy's user avatar
-1 votes
2 answers
1k views

Convert CSV to text and automatically remove quotes and a specific word - Linux

I need to convert .csv to .txt format via CL (Command line) and automatically remove quotes and a specific word - Linux. Example: I have multiple exported csv files that have: At first row I have: &...
Marius's user avatar
  • 1
2 votes
0 answers
926 views

Find and replace multi-line block in multiple files (bash/linux)

I'm trying to find a way to find a text block that meets the criteria: block starts with "google = {" block ends with next "}" character found - ignoring everything else. Example ...
jbl-toom's user avatar
0 votes
0 answers
407 views

Awk regex: how to remove all punctuation except when in the middle of a word?

Following information gathered on this thread, I wanted to use this code to fill my needs, however the code provided wasn't working for more than one punctuation character (e.g. '...'). I thus wanted ...
Michael Gauthier's user avatar
1 vote
1 answer
161 views

get the top urls from the logs

I have a numbers of log files: adsfs.demo.com_2022-07-11-0000-0001_cn.tgz adsfs.demo.com_2022-07-11-0000-0002_cn.tgz adsfs.demo.com_2022-07-11-0000-0003_cn.tgz adsfs.demo.com_2022-07-11-0000-...
244boy's user avatar
  • 615
0 votes
1 answer
159 views

Insert new line if file contain string

I need to insert 'import packagename' to the line after imports. In files which contains the specific word. What I try: find iOS/Chat \ -type f \ -name "*.swift" \ -exec gsed -e "/...
Bimawa's user avatar
  • 101
0 votes
1 answer
922 views

Inserting string between two known strings with sed or awk

I've used sed a bit before in the past for updating strings but I haven't been successful with inserting a string in between two known strings. I attempted to use this but it didn't update anything. ...
rk92's user avatar
  • 103
0 votes
2 answers
4k views

How to group and count all lines in a file that contain specific string

I want to filter all lines from a file that contain mySearchString and after that group them together and count them. Example find all lines that contain 9791 AB-9791___Foo AB-9791___Foo DE-9791___Bar ...
surfmuggle's user avatar
0 votes
1 answer
432 views

Summing real numbers from a text file

We have the following file: more /home/avrg.bc.txt 3.4 54.2 4.5 3.5 2.3 89.6 . . . We want to calculate the sum of the numbers in the file as 3.4 + 54.2 + 4.5 + ....
King David's user avatar
0 votes
1 answer
60 views

keep each line containing x only if column y is the first containing x. remove the rest containing x

I'm trying to reduce a results file that contains multiple results for sometimes the same query and I only want the top result from each query. Using bash and bash tools like awk, sed, etc. I would ...
DJ Champion's user avatar
0 votes
1 answer
220 views

Removing Duplicate from File

I have two files fileA and fileb FILEA 1.1.1.1 2.2.2.2 3.3.3.3 4.4.4.4 5.5.5.5 FILEB 4.4.4.4 2.2.2.2 Output should be 1.1.1.1 3.3.3.3 5.5.5.5 Size of File A is around 60GB and Size of File B is 1GB ...
ph3ro's user avatar
  • 143
0 votes
1 answer
283 views

Enclose the year in parenthesis in a filename

I have a group of files in a folder that I want to rename; each file has a 4-digit year as part of the filename, I would like to enclose the years in parenthesis. "Ghostbusters 3 2016.avi" ...
user88710's user avatar
2 votes
2 answers
126 views

Find and delete near duplicate lines with certain punctuation

I have a text file with millions of lines. Some lines contain the same alphanumeric sequence, but differ in case and punctuation. I consider these lines duplicates. I want to remove any duplicate line ...
Priora's user avatar
  • 35
1 vote
1 answer
1k views

Extract last 5mins log from log file using shell

I need to extract last 5mins log from log file. Here is my log file [2022-02-08 13:26:21:352] [ERROR] [iBus Connection LifeCycle - CCMHost_DummyDevice_Backup_AD2::Management:::NRMCMO_FLPROD2] [com....
Anirudh's user avatar
  • 11
0 votes
0 answers
519 views

using awk with single quote in a shell script will output error [duplicate]

we are on Debian 11. I'd like to use grep and awk to retrieve information from the content of a single file, like the following: file: /home/user/directory/otherdirectory/data.txt file content: $db['...
iodurocarburo's user avatar
-1 votes
2 answers
1k views

I have a list of searching word. How to replace them in another file?

I'm sure this question is asked multiple times, but I fail to find it. Basically I have a text file, and a list of all search strings (e.g. cat, dog, banana, etc). I want all occurrences of those ...
Ooker's user avatar
  • 2,131
0 votes
1 answer
906 views

Printf in gawk with the correct encoding?

I'm wondering: can gawk printf in any format besides ASCII? Currently, I'm using gawk match() to search through some UTF-8 text. When I go ahead and print out the matches gawk finds, it ends up like ...
ixns's user avatar
  • 1
0 votes
2 answers
640 views

Remove the file name from a messy path in a text file awk or sed

I have an xml file with many lines containing something like this: Save="C:\Users\Administrator\Desktop\my files\1020\A54f\Drawing965.DWG" Module="0" Save="C:\Users\...
Priora's user avatar
  • 35
0 votes
2 answers
614 views

How can erase first character of all string in file

How can erase first character of all string in file if this is zero? e.g I have file $ cat file 01111111 01222222 6666066 09999011 02222044 0743333 01000000 30000000 2222220 Try with sed command ...
aav1307's user avatar
1 vote
0 answers
59 views

Awk stopping printing at "

I'm making an awk script to convert a csv of aliases into a shell script to apply them, but I'm having an issue where awk seems to be stopping at a " in the second field. The command I'm running ...
dwdwdan's user avatar
  • 11

1
2 3 4 5
13