Streaming log file contents (piping)

Question

I am trying to understand how piping works and am stuck at this point.

A log file is being streamed and its output is being piped to pigz. What I am trying to do is, redirect the pigz content to a python script as stdin.

I can't figure out how to redirect the data to script

This currently works:

./customread file_name | pigz -dc

This shows the log contents to stdout, how do I pipe it to the script from here?

I've tried

python script.py < `./customread file_name | pigz -dc`

I want to redirect the data as it comes in..

Answer 1

Not being too familiar with what you're trying to get to, but I think you can just use another nested pipe. I've done similar in bash and zsh. With pigz outputting to stdout with -c, just pipe that to your python script looking for stdin.

./customread file_name | pigz -dc | python script.py

Have you tried something like above? It worked for me when I tried with;

tail -f /var/log/syslog | grep `hostname` | python test.py

Here's the Python I used courtesy of this post from Stackoverflow:

#!/usr/bin/python                                                   
import sys 
import os

while True:
    line = sys.stdin.readline()
    if not line:
        break
    sys.stdout.write(line) # or print, doesn't matter.