Modify 2>&1 so that output goes to a specific file and any errors or other output goes to a log file I can monitor?

Question

I'm reading this post and I don't fully follow the answer.

I wondered if I could modify this, what's it called, command redirect?

What I would like:

1. Don't hang, use ampersand & to move on to the next command
2. Output the data generated by the command to file data.json
3. Any problems, error messages or issues running the command, output them to bqlog.txt

Is this possible?

E.g. I currently have this command:

bq query --max_rows=1000000 --use_legacy_sql=false --format=json --batch=true --headless=true 'select * from `ga4-extract.analytics_123456789.events_'"$RUNDATE"'`;' > data.json

It's failing after about 30 minutes and I cannot see the output or what the issue is. This is on a github actions runner and the console output of the workflow just gives a failed status but, unique to this workflow, no console output indicating the error.

I wanted to modify my command so that I can output to data.json like I already have, but then also move on to the next step of monitoring the bqlog.txt file with tail -F bqlog.txt

Answer 1

Yes, this is possible.

The 2>&1 means the following:

2 : stderr
> : output redirect
& : copy file descriptor
1 : file descriptor 1

So: stderr output redirect to wherever file descriptor 1 (stdout) is directed. Default: the screen.

This means, stderr is outputted to the screen.

Your question is, how can I redirect the output of my command to file 1, and the stderr to another?

Your command becomes:

command >output1.txt 2>output2.txt

To use your own code, you would write (I've added enters to avoid a scrollbar and making it readable. The same command without the enters and scrollbar is below:

bq query --max_rows=1000000 --use_legacy_sql=false --format=json 
   --batch=true --headless=true 
   'select * from `ga4-extract.analytics_123456789.events_'"$RUNDATE"'`;
   ' >data.json 2>bglog.txt

or without enters:

bq query --max_rows=1000000 --use_legacy_sql=false --format=json --batch=true --headless=true 'select * from `ga4-extract.analytics_123456789.events_'"$RUNDATE"'`;' >data.json 2>bglog.txt

Do note, this command will create an empty file if there is no output, and it will overwrite the file on every command. If you want to append, use >> instead. Eg:

>>data.json 2>>bglog.txt

In addition you may want to add the following:

echo --------------------- >> data.json
echo --------------------- >> bglog.txt

This will write ---- to both files, helping you to see when an old execution and new exection takes place.