0

I have script [1] that is executed by Postgres daemon (WAL archiver) and fails with exit code 12; see [2]. But if I execute the same script on a terminal/ssh session, it succeeds; see [3].

User BasketCase on freenode IRC channel #rsync tried to diagnose it, but couldn't get far. See [4] for the conversation.

This does not happen on all the machines its used on, but this is the second time I have faced this.

Any help will be highly appreciated.

Thanks in advance.

[1] WAL archiving script

#!/bin/bash
# $1 is the %p substituted by postgres in archive_command
# $2 is the %f substituted by postgres in archive_command
# This script backs up the WAL file to every replica, and
# exits with the last failure code, if any.
final_exit_code=0
replicas=$(grep REPLICA /some/file | sort | uniq | cut -d = -f 2-)

for replica_url in $replicas; do
     echo Sending WAL file to $replica_url
     rsync --timeout=10 -avz -e 'ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o ConnectTimeout=10 -i /opt/PostgresPlus/CloudDB/data/cluster_ssh.key' "$1" root@$replica_url:/mnt/pcs/wal_archive/"$2"
     exit_code=$?
     if [ $exit_code -ne 0 ] ; then final_exit_code=$exit_code ; fi
  done
exit $final_exit_code

[2] Sending WAL file to 10.33.177.184 rsync: connection unexpectedly closed (0 bytes received so far) [sender] rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6] LOG: archive command failed with exit code 12 DETAIL: The failed archive command was: ./wal_archive.sh pg_xlog/0000000100000005000000EE 0000000100000005000000EE

[3] $ ./wal_archive.sh pg_xlog/0000000100000005000000EE 0000000100000005000000EE Sending WAL file to 10.33.177.184 Warning: Permanently added '10.33.177.184' (RSA) to the list of known hosts. sending incremental file list 0000000100000005000000EE

sent 5180930 bytes received 31 bytes 941992.91 bytes/sec total size is 16777216 speedup is 3.24

[4] http://gurjeet.privatepaste.com/dc98277db3

3
  • Trawling through all the chat logs ... what is the content of /opt/PostgresPlus/9.1AS/lib ?
    – tink
    Jan 11, 2013 at 2:38
  • (1) When you run from terminal, are you using(su) the postgres daemon account? (2) Which Linux distro are you using?
    – John Siu
    Jan 11, 2013 at 5:31
  • @tink The gurjeet.privatepaste.com/e75c925ffd Setting LD_LIBRARY_PATH in terminal to be same as Daemon's caused the same error in the terminal too! Thanks for the tip. Jan 11, 2013 at 17:54

1 Answer 1

1

The problem was the LD_LIBRARY_PATH difference between the terminal and the Postgres daemon's environment.

If I use the same LD_LIBRARY_PATH in the terminal, then the rsync in terminal fails too:

$ export LD_LIBRARY_PATH=/opt/PostgresPlus/9.1AS/lib:
$ ./wal_archive.sh pg_xlog/0000000100000005000000EE 0000000100000005000000EE
Sending WAL file to 10.33.177.184
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(600) [sender=3.0.6]

ssh was using the libraries from /opt/PostgresPlus/9.1AS/lib, which are probably incompatible with ssh binary.

Here's ldd output for ssh after setting that export LD_LIBRARY_PATH

$ ldd `which ssh`
    linux-vdso.so.1 =>  (0x00007fff3fa28000)
    libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007fe726907000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007fe7266e7000)
    libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007fe72634d000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007fe72614a000)
    libz.so.1 => /opt/PostgresPlus/9.1AS/lib/libz.so.1 (0x00007fe725f34000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007fe725d1b000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007fe725ae4000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007fe7258c9000)
    libgssapi_krb5.so.2 => /opt/PostgresPlus/9.1AS/lib/libgssapi_krb5.so.2 (0x00007fe725690000)
    libkrb5.so.3 => /opt/PostgresPlus/9.1AS/lib/libkrb5.so.3 (0x00007fe7253d3000)
    libk5crypto.so.3 => /opt/PostgresPlus/9.1AS/lib/libk5crypto.so.3 (0x00007fe7251aa000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007fe724fa6000)
    libnss3.so => /usr/lib64/libnss3.so (0x00007fe724c6a000)
    libc.so.6 => /lib64/libc.so.6 (0x00007fe7248d6000)
    libplc4.so => /lib64/libplc4.so (0x00007fe7246d1000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007fe7244cd000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fe726d76000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007fe72426a000)
    libcom_err.so.3 => /opt/PostgresPlus/9.1AS/lib/libcom_err.so.3 (0x00007fe724067000)
    libkrb5support.so.0 => /opt/PostgresPlus/9.1AS/lib/libkrb5support.so.0 (0x00007fe723e60000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fe723c42000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007fe723a1c000)
    libplds4.so => /lib64/libplds4.so (0x00007fe723818000)
    libnspr4.so => /lib64/libnspr4.so (0x00007fe7235da000)

And here's the same command without the LD_LIBRARY_PATH set

$ ldd `which ssh`
    linux-vdso.so.1 =>  (0x00007fff941ff000)
    libfipscheck.so.1 => /lib64/libfipscheck.so.1 (0x00007f93b2ab2000)
    libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f93b2893000)
    libcrypto.so.10 => /usr/lib64/libcrypto.so.10 (0x00007f93b24f8000)
    libutil.so.1 => /lib64/libutil.so.1 (0x00007f93b22f5000)
    libz.so.1 => /lib64/libz.so.1 (0x00007f93b20df000)
    libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f93b1ec5000)
    libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f93b1c8e000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f93b1a74000)
    libgssapi_krb5.so.2 => /lib64/libgssapi_krb5.so.2 (0x00007f93b1831000)
    libkrb5.so.3 => /lib64/libkrb5.so.3 (0x00007f93b1552000)
    libk5crypto.so.3 => /lib64/libk5crypto.so.3 (0x00007f93b1326000)
    libcom_err.so.2 => /lib64/libcom_err.so.2 (0x00007f93b1121000)
    libnss3.so => /usr/lib64/libnss3.so (0x00007f93b0de5000)
    libc.so.6 => /lib64/libc.so.6 (0x00007f93b0a52000)
    libplc4.so => /lib64/libplc4.so (0x00007f93b084c000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00007f93b0648000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f93b2f21000)
    libfreebl3.so => /lib64/libfreebl3.so (0x00007f93b03e6000)
    libkrb5support.so.0 => /lib64/libkrb5support.so.0 (0x00007f93b01da000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f93affd7000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f93afdba000)
    libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f93afb93000)
    libplds4.so => /lib64/libplds4.so (0x00007f93af98f000)
    libnspr4.so => /lib64/libnspr4.so (0x00007f93af752000)

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .