I'm having a problem with one segment of the following script. Each echo->grep->was written in to help me figure out where the script was failing to perform as I expect it to. It is designed to be part of a larger loop, but for sanity, I only have here the part that is relevant to my question.

I have files in /var/www/acpog/upload that are named 0SCAN2013 followed by the rest of the date (YYYYMMDDHHmmSS)

And the file ./alt_upcs.csv has 2 columns, one being sku (1419), the second being UPC (9781844484201)

If I can't find a match in the /var/www/acpog/uploads/ for sku, I want to attempt to find a match for the UPC. If that fails I want to attempt to find a match for sku with zeros padding the left hand side.

Here's the snippet:


echo "grep \"^$id,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
grep "^$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
dt=$(grep "^$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

if [ "$dt" == "" ]
  echo "grep \"^$id,\" alt_upcs.csv | cut -d',' -f 2"
  grep "^$id," alt_upcs.csv | cut -d',' -f 2
  u=$(grep "^$id," alt_upcs.csv | cut -d',' -f 2)
  for j in $(grep "^$id," alt_upcs.csv | cut -d',' -f 2)
    if [ "$dt" == "" ]
      echo "grep \"^$j,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
      grep "$j" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
      dt=$(grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

if [ "$dt" == "" ]
  echo "grep \"^0$id,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
  grep "^0$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
  dt=$(grep "^0$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

and here's the output:

grep "^1419," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1
grep "^1419," alt_upcs.csv | cut -d',' -f 2
," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1
grep "^01419," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1

Lines 1, 2, 3, and 5 are as expected. Line 4 does not have the beginning of the line I'm trying to echo up to just before the comma. If I echo the variable $j after this, it contains the UPC as expected, but the grep command using that variable isn't returning anything either.

EDIT: Also, there are 4 files that should match in the upload directory, all starting with 0SCAN2013

Here's some further output from manually running commands from the prompt:

# echo "grep \"^$j,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
grep "^9781844484201," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1

# grep $j /var/www/acpog/upload/$st$sd*

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 1 | tail -1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 2 | tail -1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 3 | tail -1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1

To test, these files should work. In the directory where the script resides: filename alt_upcs.csv containing 1 line:


In the /var/www/acpog/upload/ directory, filename 0SCAN20130821213905 contining 1 line:

  • Can you post a minimum working example of your input files so we can test it?
    – terdon
    Sep 2, 2013 at 17:04

1 Answer 1


Your csv file has windows "CRLF" line endings. (CR is "carriage return", also written \r or \0x0D; LF is "line feed", \n or \0x0A). The unix utilities you're using don't recognize the CR as part of the line ending, so it becomes part of the looked-up UPC. That will prevent the UPC from being found. It also creates misleading output (your line 4), because the CR does a carriage-return on output, causing the part of the echo following it (starting with the ,") to overwrite the beginning.

Use dos2unix or some similar utility to fix your csv file.

  • Thanks, now I feel like a nooby for not realizing this sooner. I am used to being able to see the extra characters in vi, but my files showed fine. I created a new alt_upcs.csv file from scratch and it worked. Sep 2, 2013 at 17:52
  • @DavidWilkins: vim will normally autodetect line-ending style; you need to check to see whether something like [DOS] appears after the filename in your status bar. You can use set ff=unix to dedosify the file.
    – rici
    Sep 2, 2013 at 17:54
  • Thanks, that's easier to remember than tr -d '\15\32 < dosfile > unixfile and no I don't know why my server doesn't have dos2unix. It's the first one I ever worked on that doesn't Sep 2, 2013 at 18:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .