Bash: How to remove 0x0A character from start of file

Question

Somewhere before the line count=~echo "$conflict" | wc -l~ (~ are used in place of back quote), the hex character 0x0A is being added. So, when the files $DA and $DB are the same (meaning the script should return a null set), 0x0A is being added. So, instead of the line count being zero in the next command, the count is 1. It is being added to the end the output for each iteration of the loop. It isn't a big deal for loops returning other content as well but is an annoyance for loops in which a null set should be returned (and accordingly, a new file should not be created). I have experimented with inserting tr -d '\r' and tr -d '\n' at the start and end of the conflict=~cat "$DA" "$DB" line but it hasn't helped.

How can I simply delete any line that contains only this character (or really any non-printable character except for \n)?

Your help is appreciated.

# Files "DA" and "DB" are titled as such:
# 10M_$i_$j_$m_OtherNonRelevantChars or 10M_$i_$k_$n_OtherNonRelevantChars
# Field one is an integer; fields 2 and 3 are alphanumeric (checksum hashes)

IFS=$'\n'

for i in {1..7}
    do for j in {P,B,R}
        do for k in {P,B,R}
            do for m in {3,9}
                do for n in {3,9}
                    do 
                        DA=`ls "10M_${i}_${j}_${m}"*`
                        DB=`ls "10M_${i}_${k}_${n}"*`
                        dos2unix "$DA" "$DB"
                
                        conflict=`cat "$DA" "$DB" | \
                            awk  -v OFS=',' -F',' '{print $1,substr($2,0,8),substr($3,0,8)}' | \
                            sort -t',' -k1,1n | uniq -u | \
                            awk -v i=$i -v j=$j -v k=$k -v m=$m -v n=$n -v OFS=',' -F',' '{print i,j,m,k,n,$1}' | \
                            uniq -d`

                        # The unwanted character appears somewhere before here
                        count=`echo "$conflict" | wc -l`
                        if [[ "$count" -gt 0 ]]
                            then 
                                echo $conflict  >> "C_${i}_${j}_${m}_${k}_${n}.txt" # output of conflicting hashes for given matching unique first field ID
                        fi
                        echo "$i $j $m $k $n"
                    done
                done
            done
        done
    done

Answer 1

$count is always greater than zero because echo adds a newline character. Each of these commands prints 1:

conflict=""; echo "$conflict" | wc -l
             echo ""          | wc -l
             echo             | wc -l

Echoing the content of a variable that contains newline character(s) can only increase the count.

This is what happens:

• Command substitution (conflict=…) removes all trailing newline characters.

• Therefore we can consider few cases:

  1. If there is a newline character followed by a non-newline character the the variable will contain at least one newline character.
  2. If the first line is non-empty and all (zero or more) following lines are empty then the variable will be non-empty but it will contain exactly zero newline characters.
  3. If there is only an incomplete line then the variable will be non-empty but it will contain exactly zero newline characters.
  4. If all the lines are empty then the variable will be empty; as such it will contain exactly zero newline characters.
  5. If the output is empty then the variable will be empty; as such it will contain exactly zero newline characters.

• As already stated, echo adds exactly one newline character.

• wc -l counts newline characters. When we say it counts lines, we mean formally defined lines. An incomplete line is not a line in these terms.

So after echo there is at least one newline character and [[ "$count" -gt 0 ]] is always true. Worse, using printf or echo -n to avoid adding an extra newline (xor taking it into account with -gt 1) is not a solution because "exactly one line" and "empty output" (and few other cases) generate the same number of newlines in the variable: zero.

In your case probably it's enough to test if $conflict is not empty:

[ -n "$conflict" ]

In general you may want to keep trailing newlines. It seems your second awk cannot generate empty lines though.

---

Note if you calculated count directly, without the intermediate conflict variable

count="$(cat … | … | wc -l)"

then there would be no problem. I notice you use $conflict later (don't you want to quote it?), so this approach is not really an option (unless you decide to obtain count and conflict independently, which is sub-optimal and in general may lead to inconsistent results when input data changes in between).

---

Let's get back to incomplete lines. You're using dos2unix. In the DOS/Windows world a text file may end with an incomplete line (in terms of POSIX). If this happens then after dos2unix the incomplete line will still be incomplete. You have at least two potential problems when you run cat "$DA" "$DB" | …:

1. If the first file ends with an incomplete line then it will be concatenated with the beginning of the second file without any newline in between.
2. If the second file ends with an incomplete line then the output from cat will end with it and the tools later in the pipe may "misbehave". Text processing tools expect text, lines. In general, upon encountering an incomplete line such tool may:
   • ignore the incomplete line as if it wasn't there;
   • or accept it and pass it (think about a filter like grep) to its output as an incomplete line;
   • or accept it, fix (add a newline character) and pass it to its output as a complete line;
   • or throw an error (example).

So pay attention to your input files.