Having trouble with what should be a simple bash script

Question

I'm having a problem with one segment of the following script. Each echo->grep->was written in to help me figure out where the script was failing to perform as I expect it to. It is designed to be part of a larger loop, but for sanity, I only have here the part that is relevant to my question.

I have files in /var/www/acpog/upload that are named 0SCAN2013 followed by the rest of the date (YYYYMMDDHHmmSS)

And the file ./alt_upcs.csv has 2 columns, one being sku (1419), the second being UPC (9781844484201)

If I can't find a match in the /var/www/acpog/uploads/ for sku, I want to attempt to find a match for the UPC. If that fails I want to attempt to find a match for sku with zeros padding the left hand side.

Here's the snippet:

st="0"
id="1419"
sd="SCAN2013"

echo "grep \"^$id,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
grep "^$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
dt=$(grep "^$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

if [ "$dt" == "" ]
then
  echo "grep \"^$id,\" alt_upcs.csv | cut -d',' -f 2"
  grep "^$id," alt_upcs.csv | cut -d',' -f 2
  u=$(grep "^$id," alt_upcs.csv | cut -d',' -f 2)
  for j in $(grep "^$id," alt_upcs.csv | cut -d',' -f 2)
  do
    if [ "$dt" == "" ]
    then
      echo "grep \"^$j,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
      grep "$j" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
      dt=$(grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)
    fi
  done
fi

if [ "$dt" == "" ]
then
  echo "grep \"^0$id,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
  grep "^0$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
  dt=$(grep "^0$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)
fi

and here's the output:

grep "^1419," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1
grep "^1419," alt_upcs.csv | cut -d',' -f 2
9781844484201
," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1
grep "^01419," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1

Lines 1, 2, 3, and 5 are as expected. Line 4 does not have the beginning of the line I'm trying to echo up to just before the comma. If I echo the variable $j after this, it contains the UPC as expected, but the grep command using that variable isn't returning anything either.

EDIT: Also, there are 4 files that should match in the upload directory, all starting with 0SCAN2013

Here's some further output from manually running commands from the prompt:

# echo "grep \"^$j,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
grep "^9781844484201," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1

# grep $j /var/www/acpog/upload/$st$sd*
/var/www/acpog/upload/0SCAN20130821213905:9781844484201,0,2,8:21,13:48:496

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 1 | tail -1
/var/www/acpog/upload/4SCAN20130805204626:9781844484201

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 2 | tail -1
2

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 3 | tail -1
-1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
8:21,13:48:496

To test, these files should work. In the directory where the script resides: filename alt_upcs.csv containing 1 line:

1419,9781844484201

In the /var/www/acpog/upload/ directory, filename 0SCAN20130821213905 contining 1 line:

9781844484201,0,2,8:21,13:48:496

Answer 1

Your csv file has windows "CRLF" line endings. (CR is "carriage return", also written \r or \0x0D; LF is "line feed", \n or \0x0A). The unix utilities you're using don't recognize the CR as part of the line ending, so it becomes part of the looked-up UPC. That will prevent the UPC from being found. It also creates misleading output (your line 4), because the CR does a carriage-return on output, causing the part of the echo following it (starting with the ,") to overwrite the beginning.

Use dos2unix or some similar utility to fix your csv file.