I'm having a problem with one segment of the following script. Each echo->grep->was written in to help me figure out where the script was failing to perform as I expect it to. It is designed to be part of a larger loop, but for sanity, I only have here the part that is relevant to my question.

I have files in /var/www/acpog/upload that are named 0SCAN2013 followed by the rest of the date (YYYYMMDDHHmmSS)

And the file ./alt_upcs.csv has 2 columns, one being sku (1419), the second being UPC (9781844484201)

If I can't find a match in the /var/www/acpog/uploads/ for sku, I want to attempt to find a match for the UPC. If that fails I want to attempt to find a match for sku with zeros padding the left hand side.

Here's the snippet:


echo "grep \"^$id,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
grep "^$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
dt=$(grep "^$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

if [ "$dt" == "" ]
  echo "grep \"^$id,\" alt_upcs.csv | cut -d',' -f 2"
  grep "^$id," alt_upcs.csv | cut -d',' -f 2
  u=$(grep "^$id," alt_upcs.csv | cut -d',' -f 2)
  for j in $(grep "^$id," alt_upcs.csv | cut -d',' -f 2)
    if [ "$dt" == "" ]
      echo "grep \"^$j,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
      grep "$j" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
      dt=$(grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

if [ "$dt" == "" ]
  echo "grep \"^0$id,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
  grep "^0$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1
  dt=$(grep "^0$id," /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1)

and here's the output:

grep "^1419," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1
grep "^1419," alt_upcs.csv | cut -d',' -f 2
," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1
grep "^01419," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1

Lines 1, 2, 3, and 5 are as expected. Line 4 does not have the beginning of the line I'm trying to echo up to just before the comma. If I echo the variable $j after this, it contains the UPC as expected, but the grep command using that variable isn't returning anything either.

EDIT: Also, there are 4 files that should match in the upload directory, all starting with 0SCAN2013

Here's some further output from manually running commands from the prompt:

# echo "grep \"^$j,\" /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1"
grep "^9781844484201," /var/www/acpog/upload/0SCAN2013* | cut -d',' -f 4-5 | tail -1

# grep $j /var/www/acpog/upload/$st$sd*

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 1 | tail -1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 2 | tail -1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 3 | tail -1

# grep $j /var/www/acpog/upload/$st$sd* | cut -d',' -f 4-5 | tail -1

To test, these files should work. In the directory where the script resides: filename alt_upcs.csv containing 1 line:


In the /var/www/acpog/upload/ directory, filename 0SCAN20130821213905 contining 1 line:

  • Can you post a minimum working example of your input files so we can test it?
    – terdon
    Sep 2, 2013 at 17:04

1 Answer 1


Your csv file has windows "CRLF" line endings. (CR is "carriage return", also written \r or \0x0D; LF is "line feed", \n or \0x0A). The unix utilities you're using don't recognize the CR as part of the line ending, so it becomes part of the looked-up UPC. That will prevent the UPC from being found. It also creates misleading output (your line 4), because the CR does a carriage-return on output, causing the part of the echo following it (starting with the ,") to overwrite the beginning.

Use dos2unix or some similar utility to fix your csv file.

  • Thanks, now I feel like a nooby for not realizing this sooner. I am used to being able to see the extra characters in vi, but my files showed fine. I created a new alt_upcs.csv file from scratch and it worked. Sep 2, 2013 at 17:52
  • @DavidWilkins: vim will normally autodetect line-ending style; you need to check to see whether something like [DOS] appears after the filename in your status bar. You can use set ff=unix to dedosify the file.
    – rici
    Sep 2, 2013 at 17:54
  • Thanks, that's easier to remember than tr -d '\15\32 < dosfile > unixfile and no I don't know why my server doesn't have dos2unix. It's the first one I ever worked on that doesn't Sep 2, 2013 at 18:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.