Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I have script [1] that is executed by Postgres daemon (WAL archiver) and fails with exit code 12; see [2]. But if I execute the same script on a terminal/ssh session, it succeeds; see [3].

User BasketCase on freenode IRC channel #rsync tried to diagnose it, but couldn't get far. See [4] for the conversation.

This does not happen on all the machines its used on, but this is the second time I have faced this.

Any help will be highly appreciated.

Thanks in advance.

[1] WAL archiving script

#!/bin/bash
# $1 is the %p substituted by postgres in archive_command
# $2 is the %f substituted by postgres in archive_command
# This script backs up the WAL file to every replica, and
# exits with the last failure code, if any.
final_exit_code=0
replicas=$(grep REPLICA /some/file | sort | uniq | cut -d = -f 2-)

for replica_url in $replicas; do
     echo Sending WAL file to $replica_url
     rsync --timeout=10 -avz -e 'ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -i /opt/PostgresPlus/CloudDB/data/cluster_ssh.key' "$1" root@$replica_url:/mnt/pcs/wal_archive/"$2"
     exit_code=$?
     if [ $exit_code -ne 0 ] ; then final_exit_code=$exit_code ; fi
  done
exit $final_exit_code

[2] Sending WAL file to 10.33.177.184 rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6] LOG: archive command failed with exit code 12 DETAIL: The failed archive command was: ./wal_archive.sh pg_xlog/0000000100000005000000EE 0000000100000005000000EE

[3] $ ./wal_archive.sh pg_xlog/0000000100000005000000EE 0000000100000005000000EE Sending WAL file to 10.33.177.184 Warning: Permanently added '10.33.177.184' (RSA) to the list of known hosts. sending incremental file list 0000000100000005000000EE

sent 5180930 bytes received 31 bytes 941992.91 bytes/sec total size is 16777216 speedup is 3.24

[4] http://gurjeet.privatepaste.com/dc98277db3

share|improve this question
    
Trawling through all the chat logs ... what is the content of /opt/PostgresPlus/9.1AS/lib ? –  tink Jan 11 '13 at 2:38
    
(1) When you run from terminal, are you using(su) the postgres daemon account? (2) Which Linux distro are you using? –  John Siu Jan 11 '13 at 5:31
    
@tink The gurjeet.privatepaste.com/e75c925ffd Setting LD_LIBRARY_PATH in terminal to be same as Daemon's caused the same error in the terminal too! Thanks for the tip. –  Gurjeet Singh Jan 11 '13 at 17:54

1 Answer 1

The problem was the LD_LIBRARY_PATH difference between the terminal and the Postgres daemon's environment.

If I use the same LD_LIBRARY_PATH in the terminal, then the rsync in terminal fails too:

$ export LD_LIBRARY_PATH=/opt/PostgresPlus/9.1AS/lib:
$ ./wal_archive.sh pg_xlog/0000000100000005000000EE 0000000100000005000000EE
Sending WAL file to 10.33.177.184
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

ssh was using the libraries from /opt/PostgresPlus/9.1AS/lib, which are probably incompatible with ssh binary.

Here's ldd output for ssh after setting that export LD_LIBRARY_PATH

$ ldd `which ssh`
    linux-vdso.so.1 =>  (0x00007fff3fa28000)
    libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007fe726907000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe7266e7000)
    libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fe72634d000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fe72614a000)
    libz.so.1 => /opt/PostgresPlus/9.1AS/lib/libz.so.1 (0x00007fe725f34000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fe725d1b000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fe725ae4000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fe7258c9000)
    libgssapi_krb5.so.2 => /opt/PostgresPlus/9.1AS/lib/libgssapi_krb5.so.2 (0x00007fe725690000)
    libkrb5.so.3 => /opt/PostgresPlus/9.1AS/lib/libkrb5.so.3 (0x00007fe7253d3000)
    libk5crypto.so.3 => /opt/PostgresPlus/9.1AS/lib/libk5crypto.so.3 (0x00007fe7251aa000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fe724fa6000)
    libnss3.so => /usr/lib64/libnss3.so (0x00007fe724c6a000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe7248d6000)
    libplc4.so => /lib64/libplc4.so (0x00007fe7246d1000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fe7244cd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe726d76000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007fe72426a000)
    libcom_err.so.3 => /opt/PostgresPlus/9.1AS/lib/libcom_err.so.3 (0x00007fe724067000)
    libkrb5support.so.0 => /opt/PostgresPlus/9.1AS/lib/libkrb5support.so.0 (0x00007fe723e60000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe723c42000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fe723a1c000)
    libplds4.so => /lib64/libplds4.so (0x00007fe723818000)
    libnspr4.so => /lib64/libnspr4.so (0x00007fe7235da000)

And here's the same command without the LD_LIBRARY_PATH set

$ ldd `which ssh`
    linux-vdso.so.1 =>  (0x00007fff941ff000)
    libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007f93b2ab2000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f93b2893000)
    libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f93b24f8000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f93b22f5000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f93b20df000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f93b1ec5000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f93b1c8e000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f93b1a74000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f93b1831000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f93b1552000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f93b1326000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f93b1121000)
    libnss3.so => /usr/lib64/libnss3.so (0x00007f93b0de5000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f93b0a52000)
    libplc4.so => /lib64/libplc4.so (0x00007f93b084c000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f93b0648000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f93b2f21000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007f93b03e6000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f93b01da000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f93affd7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f93afdba000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f93afb93000)
    libplds4.so => /lib64/libplds4.so (0x00007f93af98f000)
    libnspr4.so => /lib64/libnspr4.so (0x00007f93af752000)
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.