0

I created a script that processes files in a loop, using a process call. I check the exit code of said call to see if I should move the files (on success.) Problem is, that when the process fails with an exception, it never exits. How do I go about detecting the exception occurred, so I can make the script move on to the next files?

Relevant part of script

# Stream data
sstableloader -d $3 $tablepathfull

# On success, move data to target dir
if [[ $? != 0 ]]; then
    echo "Error: Table failed - $tablepathfull"
else
    echo "Table OK - $tablepathfull"
    trgtdir="$2/$hostname/$keyspacename/$typename/$timestamp/$keyspacename/$tablename"
    mkdir -p $trgtdir
    mv $tablepathfull/* $trgtdir
    rmdir $tablepathfull
fi

If there is no 'official' way, is it perhaps possible to capture the output (see below) of the process call, and simply kill the process once/if the exception occurs?

Exception output

Exception in thread "STREAM-OUT-/XX.XX.XXX.88" Exception in thread "STREAM-OUT-/XX.XX.XXX.92" java.lang.NullPointerException
    at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
    at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
    at java.lang.Thread.run(Thread.java:744)
java.lang.NullPointerException
    at org.apache.cassandra.streaming.ConnectionHandler$MessageHandler.signalCloseDone(ConnectionHandler.java:249)
    at org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:375)
    at java.lang.Thread.run(Thread.java:744)
10
  • That exception should be terminal. Please report this as a bug.
    – Dan D.
    Mar 2, 2014 at 9:55
  • @DanD. To who, and where? And isn't it possible the java application that I am calling prints the exception, but then doesn't exit? Sounds probable to me...
    – natli
    Mar 2, 2014 at 10:31
  • Can you see the process with ps after it gives that exception? It should be a child process to your script.
    – user127350
    Mar 2, 2014 at 10:32
  • @Radoo I can see it in htop
    – natli
    Mar 2, 2014 at 10:32
  • Then it should be a bug of that Java app. The normal behaviour of a simple process tool after throwing an exception is to exit with an error code. Why would it be locked forever doing nothing?
    – user127350
    Mar 2, 2014 at 10:34

1 Answer 1

2

The only workaround I could come with is using subprocesses and files:

TEMP_FILE='/tmp/some_file.txt'

function load_table() {
  if [ $# -lt 2 ]; then
    printf "1" > "${TEMP_FILE}"
    return 1
  fi

  local param1="$1"
  local table_full_path="$2"
  local exit_code

  # Stream data
  sstableloader -d "${param1}" "${table_full_path}" >> "${TEMP_FILE}"
  exit_code=$?

  printf "\n%s" "${exit_code}" >> "${TEMP_FILE}"
}

function is_process_running() {
  if [ $# -eq 0 ]; then
    return 1
  fi
  local process_id="$1"

  ps aux | sed -r 's/[ ]+/ /g' | cut -d' ' -f2 | grep -q "${process_id}"
  return $?
}

function exceptions_count() {
  local count=$(tail -10 "${TEMP_FILE}" | grep -c "Exception")
  return $count
}

…

load_table "$3" "${tablepathfull}" &

# Given you have one subprocess only.. get the pid of the first subprocess in the list
job_pids=( $(jobs -p) )
load_table_job_pid=${job_pids[0]}

while is_process_running "${load_table_job_pid}" && exceptions_count -eq 0; do
  sleep 5
done

exit_code=0
if is_process_running "${load_table_job_pid}"; then
  local load_table_job_gid=$(ps x -o  "%p %r %y %x %c " | sed -r -e 's/[ ]+/ /g' -e 's/^[ ]+//g' | grep -E "^${load_table_job_pid} " | cut -d' ' -f2)
  kill -TERM -$load_table_job_gid >/dev/null 2>&1
  exit_code=1
else
  exit_code=$(tail -1 "${TEMP_FILE}")
fi

rm -f "${TEMP_FILE}"

# Your code
# On success, move data to target dir
if [ $exit_code -ne 0 ]; then
    echo "Error: Table failed - $tablepathfull"
else
    echo "Table OK - $tablepathfull"
    trgtdir="$2/$hostname/$keyspacename/$typename/$timestamp/$keyspacename/$tablename"
    mkdir -p $trgtdir
    mv $tablepathfull/* $trgtdir
    rmdir $tablepathfull
fi

You can improve the code by adding a retry count or something.

11
  • Retry isn't going to do any good, since the exception seems to occur if there's something wrong with the table files. I'll test this script & accept the answer as soon as the issue occurs again, but it looks good & taught me a thing or two about bash, so thanks!
    – natli
    Mar 2, 2014 at 17:26
  • Oh, just out of curiosity; does grep -c "Exception" "${TEMP_FILE}" read the whole file every time? Because the output file can get quite big. If so, I'm going to replace it with tail -n 10 because the last 10 lines would definitely include the Exception if it did occur.
    – natli
    Mar 2, 2014 at 17:32
  • Yes. Given the part of the code you wrote, I thought that java tool doesn't output anything. local count=$(tail -10 "${TEMP_FILE}" | grep -c "Exception")
    – user127350
    Mar 2, 2014 at 21:56
  • I've updated the code with this last update.
    – user127350
    Mar 2, 2014 at 21:58
  • It appears to be working just fine, but an exception hasn't occurred yet so I can't be 100% sure. But theoretically it should work so I expect no issues, thanks!
    – natli
    Mar 2, 2014 at 21:59

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .