How to run commands as in a queue

Question

I need to do a lot of copying of various files to various folders. I can add all my copy commands to a bash script and then run that, but then I must wait until it finishes if I want to add more commands to that copying "queue".

Is there a way I can run commands as a queue and sort of add more commands to that queue while things are running?

Explained in a different way, I want to start a long running task. While that is running I want to start another one that does not actually start until the first one is done. And then add another after that last one and so on. Is this possible somehow?

Answer 1

The at utility, best known for running commands at a specified time, also has a queue feature and can be asked to start running commands now. It reads the command to run from standard input.

echo 'command1 --option arg1 arg2' | at -q myqueue -t now
echo 'command2 ...' | at -q myqueue -t now

The batch command is equivalent to at -q b -m now (-m meaning the command output, if any, will be mailed to you, like cron does). Not all unix variants support queue names (-q myqueue); you may be limited to a single queue called b.

Answer 2

Both Brad and Mankoff's solutions are good suggestions. Another that's similar to a combination of both them would be to use GNU Screen to implement your queue. This has the advantage of being able to run in the background, you can check on it whenever, and queueing up new commands just pastes them into the buffer to be executed after the previous commands exit.

First, run:

$ screen -d -m -S queue

(incidentally, now's a good time to play with some awesome .screenrc files)

That will spawn up a background screen session for you named queue.

Now, queue up as many commands as you like:

screen -S queue -X stuff "echo first; sleep 4; echo second^M"

I'm doing multiple commands in the above just for testing. Your use case would probably look more like:

screen -S queue -X stuff "echo first^M"
screen -S queue -X stuff "echo second^M"

Note that the "^M" in my line above is a way to get an embedded newline that will be interpreted later after screen stuffs it into your existing bash shell. Use "CTL-V, " to get that sequence.

It'd be pretty easy to make some simple shell-scripts to automate that and queue up commands. Then, whenever you want to check the status of your background queue, you re-attach via:

screen -S queue -r

Technically, you don't even need to name your screen session and it will work fine, but once you get hooked on it, you're going to want to leave one running all the time anyway. ;-)

Of course, if you do that, another good way to do it would be to name one of the current windows "queue" and use:

screen -S queue -p queue -X stuff "command"

Answer 3

the hackiest way i can think of is to maintain the queue "state" with simple lock-files in /tmp. when file1.sh is doing its cp commands, it will keep a lock file (or hardlink) in /tmp. when it is done, erase the file. every other script will have to look in /tmp for lock files with the naming scheme you choose. naturally this is subject to all sorts of failures, but it is trivial to set up and is doable completely within bash.

Answer 4

You can simply add commands to the command line of shell that is already running a command.

Either type carefully, or type it elsewhere, verify, and cut-and-paste. Even if the current command is spewing lines of output, you can still type something new, hit enter, and it will run when all commands running or entered before it complete.

Example follows. You can cut-and-paste the whole thing, or enter the subsequent commands while the 1st is running (if you type really really fast), or more likely while the 2nd is running.:

echo "foo"
sleep 10; echo "bar"
sleep 3
echo "baz"