I have a tab-delimited text file containing Column1...Columnn columns and R1 to Rn rows. Within some of the columns, there are multiple nested fields with an identifier and then separated by semicolon (shown in the attached sample image file in red colour). To be precise, I am attaching a sample snapshot of the file.

enter image description here

Here, the data in the Column6 INFO has multiple nested fields such as DP; RPB; AF1; AC1; DP4;... etc.

Task: From inputfile.txt, in Column6 INFO, to choose field DP4=a,b,c,d and need to do simple arithmetic operations on the DP4=a,b,c,d values (marked in red in the image), such as (c+d)/(a+b+c+d) -> and paste the results for each row as new column INFOextra in the same file.

How can it be done on e unix shell script?

  • Please add your sample input to the question. I'm not going to waste my time retyping it to test my code. – glenn jackman Mar 12 '16 at 22:27
awk '$1=="ID" {print $0 "\tINFOextra"; next}; NF { info=$6; gsub(/.*;DP4=|;MQ=.*/, "", info); split(info, a, /,/); print $0 "\t" (a[3]+a[4])/(a[1]+a[2]+a[3]+a[4])}' inputfile.txt > outputfile.txt
# then check the content outputfile.txt and rename it if important

As you can see, the solution is very similar to the answer given to your previous question, and not longer at all. It's because awk is extremely fine-tuned to serve in problems like yours. I do suggest having a look at its manual page (http://linux.die.net/man/1/awk) to see how simple it is compared to other (and more general-purpose) languages.

If multiple input files are to be processed into multiple output files with the appropriate names, the options are the following:

  • create a loop in shell and starting an awk process for each file, one by one
  • make awk write the output in files whose names are dependant on the name of the current input file which information is stored in the FILENAME variable of awk that is automatically set during the process. Inside the awk code redirection can be used with the same syntax and similar outcome as the shell uses:

    awk '$1=="ID" {print $0 "\tINFOextra" > FILENAME ".out"; next}; NF { info=$6; gsub(/.*;DP4=|;MQ=.*/, "", info); split(info, a, /,/); print $0 "\t" (a[3]+a[4])/(a[1]+a[2]+a[3]+a[4]) > FILENAME ".out"}' inputfile1.txt inputfile2.txt

Here each instance of inputfileN.txt will have a corresponding inputfileN.txt.out file. FILENAME is a simple string so any kind of manipulation of the output files are valid.

When the specification gets complicated such that the additional field must appear at an inner position (not at the beginning or the end), creating a subroutine (called function in awk) should be considered to create the output lines. That function here iterates through all the fields, print them as usual, but where the additional field should appeare, it writes that after the n-1-th and before the n-th field, making it the n-th one. Now, it's worth to put the awk code in its own file:

$ cat bio.awk

function myprint( str) {
   for (i=1; i<=NF; ++i) {
      printf "%s", $i  > FILENAME ".out"
      if (i==44)
         printf "\t%s", str  >> FILENAME ".out"
      if (i!=NF)
         printf "\t"  >> FILENAME ".out"
   print ""  >> FILENAME ".out"

$1=="ID" {
   myprint( "INFOextra")

NF {
   gsub(/.*;DP4=|;MQ=.*/, "", info)
   split(info, a, /,/)
   myprint( (a[3]+a[4])/(a[1]+a[2]+a[3]+a[4]) )

Calling it then results in a shorter and clean command line:

awk -f bio.awk inputfile1.txt inputfile2.txt
| improve this answer | |
  • Thanks a lot. Awk is awesome. I will explore it further. Seems to be of very useful for my tasks. How to print the column name 'INFOextra' instead of '0'? Secondly, if I want to run this over multiple input files in batch, {input}_1.txt to {input}_100.txt, how could I do it? – panbar Mar 12 '16 at 22:35
  • It should (must) write "INFOextra" literally in the first line if you did not forget to put the quotes as written, that is like this "\tINFOextra". If you wish to process several input files to one output, then just write each input files (using wild cards at your will) BEFORE the "> outputfile.txt" part. If you want N output files for N inputs, then I will have to modify the answer. Should I? – Gombai Sándor Mar 12 '16 at 22:52
  • Yes, I want N output files for N inputs separately. I tried several times with print $0 "\tINFOextra". However, it did not print the column name 'INFOextra' instead of '0' in the outfile. Otherwise it performs the arithmetics correctly and print the values in the last column. – panbar Mar 12 '16 at 23:07
  • Answer modified. I tried the code with all the 3 kinds awk I know to exist on linux but could not catch your 0-problem. If you can copy some sample in text (not image) somewhere, I may be able show you a picture of befores and afters. – Gombai Sándor Mar 12 '16 at 23:34
  • Thanks, the single input file single out file script is working fine and giving correct output for (a[3]+a[4])/(a[1]+a[2]+a[3]+a[4]) including the column header. However, for the multiple input files and multiple output files script the (a[3]+a[4])/(a[1]+a[2]+a[3]+a[4]) calculated values are wrong. I used same 3 input files for the single script and multiple script. – panbar Mar 14 '16 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.