22

We have a Python fabric command that runs in parallel across several hosts, something like this:

$ fab --hosts=prod1.server,prod2.server,prod3.server --parallel copy_cache

This will copy cache to the production servers listed in parallel. There is various logging that occurs throughout the process to indicate how far along we are since it can take hours for the XXgig cache directories. Since the copying is happening concurrently, the output when run on the command line comes back interlaced in real-time, like so:

[prod1.server] Executing task 'nginx_cache_copy'
[prod2.server] Executing task 'nginx_cache_copy'
[prod3.server] Executing task 'nginx_cache_copy'
2014-09-16 10:02:29.688243
[prod1.server] INFO: rsyncing cache dir
[prod1.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" \
                                             "repo/cache/."
2014-09-16 10:02:29.716345
[prod2.server] INFO: rsyncing cache dir
[prod2.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" \
                                             "repo/cache/."
2014-09-16 10:02:29.853275
[prod3.server] INFO: rsyncing cache dir
[prod3.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" \
                                             "repo/cache/."
2014-09-16 10:02:29.984154
[prod1.server] INFO: Reloading nginx config
[prod1.server] run: sbin/nginx -s reload -c "repo/nginx.conf"
2014-09-16 10:02:30.025155
[prod2.server] INFO: Reloading nginx config
[prod2.server] run: sbin/nginx -s reload -c "repo/nginx.conf"
2014-09-16 10:02:30.100169
[prod1.server] SUCCESS: CACHE COPY COMPLETE

2014-09-16 10:02:30.181938
[prod2.server] SUCCESS: CACHE COPY COMPLETE

2014-09-16 10:02:30.331402
[prod3.server] INFO: Reloading nginx config
[prod3.server] run: sbin/nginx -s reload -c "repo/nginx.conf"
2014-09-16 10:02:30.559271
[prod3.server] SUCCESS: CACHE COPY COMPLETE

Done.

However, when the task is run through Jenkins, the console output does not display until all tasks are done because Jenkins groups the output AFTER the threads are joined when all threads are complete. So, once all commands are complete the output looks like this:

[prod1.server] Executing task 'nginx_cache_copy'
2014-09-16 10:02:29.688243
[prod1.server] INFO: rsyncing cache dir
[prod1.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" \
                                             "repo/cache/."
2014-09-16 10:02:29.984154
[prod1.server] INFO: Reloading nginx config
[prod1.server] run: sbin/nginx -s reload -c "repo/nginx.conf"
2014-09-16 10:02:30.100169
[prod1.server] SUCCESS: CACHE COPY COMPLETE

[prod2.server] Executing task 'nginx_cache_copy'
2014-09-16 10:02:29.716345
[prod2.server] INFO: rsyncing cache dir
[prod2.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" \
                                             "repo/cache/."
2014-09-16 10:02:30.025155
[prod2.server] INFO: Reloading nginx config
[prod2.server] run: sbin/nginx -s reload -c "repo/nginx.conf"
2014-09-16 10:02:30.181938
[prod2.server] SUCCESS: CACHE COPY COMPLETE

[prod3.server] Executing task 'nginx_cache_copy'
2014-09-16 10:02:29.853275
[prod3.server] INFO: rsyncing cache dir
[prod3.server] run: rsync -a -q cache.server:"repo/cache/some.site.com" \
                                             "repo/cache/."
2014-09-16 10:02:30.331402
[prod3.server] INFO: Reloading nginx config
[prod3.server] run: sbin/nginx -s reload -c "repo/nginx.conf"
2014-09-16 10:02:30.559271
[prod3.server] SUCCESS: CACHE COPY COMPLETE

Done.

While this is more readable, it's not ideal because we'd like to keep track of the state of the process by reading the console output in realtime. Note that when this fabric command is run without the --parallel option the console output does occur in realtime, however obviously this is not workable because the serial process takes much longer to run.

I haven't been able to find a setting in Jenkins that will disable this thread grouping. Does anyone have any ideas?

  • 4
    Never found a complete solution for this, but running PYTHONUNBUFFERED=1 && fab ... helps a bit by showing each server as it completes instead of waiting for all to complete. Still would like to find a completely realtime solution though. – mVChr Oct 22 '14 at 0:59
  • 3
    I'm voting to close this question as off-topic because it seems more fit for Server Fault – Tamara Wijsman Jun 23 '16 at 9:54
  • 1
    have you tried preceding the unix commands with "stdbuf -oL"? – strobelight Sep 26 '16 at 1:07
  • Mr. VarCharbroiled Charcoal Black - How satisfied are you with Superuser.com helping provide an answer to this question? Are you in need of additional attention still? Perhaps there are further options we can explore for you. You might consider changing the title to something more appealing to draw attention for example as one quick thought tactic you could apply; what's the harm in trying. – Pimp Juice IT Feb 5 '17 at 23:49
  • 4
    I'm voting to close this question as off-topic because it has not gotten a satisfactory answer here and may receive more/better attention at ServerFault. – music2myear Apr 17 '17 at 21:48

Browse other questions tagged or ask your own question.