Take the 2-minute tour ×
Super User is a question and answer site for computer enthusiasts and power users. It's 100% free, no registration required.

I need to do a lot of copying of various files to various folders. I can add all my copy commands to a bash script and then run that, but then I must wait until it finishes if I want to add more commands to that copying "queue".

Is there a way I can run commands as a queue and sort of add more commands to that queue while things are running?

Explained in a different way, I want to start a long running task. While that is running I want to start another one that does not actually start until the first one is done. And then add another after that last one and so on. Is this possible somehow?

share|improve this question
1  
May I suggest that you accept an answer that solves your problem. It seems that there are a couple of them that would work for you. –  holmb Feb 22 '13 at 14:49
1  
Yes you may, and I have wanted to. Problem is that I asked this when I worked at a company that were using Macs. I am no longer working there, nor do I have any Macs myself so I have no way of testing any of these. Don't feel comfortable with marking one as an answer when I am unable to test that it actually works, so for now I have hoped that people vote up those answers they find work well for them so the "answer" can emerge in that way instead. –  Svish Feb 24 '13 at 10:55
add comment

8 Answers

The at utility, best known for running commands at a specified time, also has a queue feature and can be asked to start running commands now. It reads the command to run from standard input.

echo 'command1 --option arg1 arg2' | at -q myqueue -t now
echo 'command2 ...' | at -q myqueue -t now

The batch command is equivalent to at -q b -m now (-m meaning the command output, if any, will be mailed to you, like cron does). Not all unix variants support queue names (-q myqueue); you may be limited to a single queue called b.

share|improve this answer
add comment

Append & to the end of your command to send it to the background, and then wait on it before running the next. For example:

$ command1 &
$ wait; command2 &
$ wait; command3 &
$ ...
share|improve this answer
1  
yay! this syntax is awesome when you've kicked something off and then go looking at stackoverflow for a way to queue up something after it –  Erik Aronesty Oct 16 '12 at 18:34
    
If you change your mind on any of those commands in the meantime, what can be done to abort the wait? It seems impervious to all my killshots. –  Ken Williams Dec 14 '12 at 23:36
add comment

Both Brad and Mankoff's solutions are good suggestions. Another that's similar to a combination of both them would be to use GNU Screen to implement your queue. This has the advantage of being able to run in the background, you can check on it whenever, and queueing up new commands just pastes them into the buffer to be executed after the previous commands exit.

First, run:

$ screen -d -m -S queue

(incidentally, now's a good time to play with some awesome .screenrc files)

That will spawn up a background screen session for you named queue.

Now, queue up as many commands as you like:

screen -S queue -X stuff "echo first; sleep 4; echo second^M"

I'm doing multiple commands in the above just for testing. Your use case would probably look more like:

screen -S queue -X stuff "echo first^M"
screen -S queue -X stuff "echo second^M"

Note that the "^M" in my line above is a way to get an embedded newline that will be interpreted later after screen stuffs it into your existing bash shell. Use "CTL-V, " to get that sequence.

It'd be pretty easy to make some simple shell-scripts to automate that and queue up commands. Then, whenever you want to check the status of your background queue, you re-attach via:

screen -S queue -r

Technically, you don't even need to name your screen session and it will work fine, but once you get hooked on it, you're going to want to leave one running all the time anyway. ;-)

Of course, if you do that, another good way to do it would be to name one of the current windows "queue" and use:

screen -S queue -p queue -X stuff "command"
share|improve this answer
    
It seems like this basically just "types" the command in the running screen session, so that when the shell gains control again after the first command finishes, this command will start. In other words, it's equivalent to @mankoff's solution, but uses a fancier tool to do the typing. –  Ken Williams Dec 14 '12 at 23:04
    
Pretty much -- the main difference is that the screen solution supports better automation. You can do some stuff by hand, other things with scripts, all with a simple interface. –  Jordan Jan 29 '13 at 6:12
add comment

I need stuff like this fairly frequently too. I wrote a little utility called after that executes a command whenever some other process has finished. It looks like this:

#!/usr/bin/perl

my $pid = shift;
die "Usage: $0 <pid> <command...>" unless $pid =~ /^\d+$/ && @ARGV;

print STDERR "Queueing process $$ after process $pid\n";
sleep 1 while -e "/proc/$pid";
exec @ARGV;

You then run it like so:

% command1 arg1 arg2 ...  # creates pid=2853
% after 2853 command2 arg1 arg2 ... # creates pid=9564
% after 9564 command3 arg1 arg2 ...

The big advantage to this over some other approaches is that the first job doesn't need to be run in any special way, you can follow any process with your new job.

share|improve this answer
    
Nice script @ken. Though I'd recommend trying out Task Spooler since you seem to require this from time to time. See my answer. –  holmb Feb 21 '13 at 14:57
    
Looks neat, thanks. The one thing missing from TS seems to be the ability to follow jobs that weren't started under TS. Though I guess you could start a new TS job whose purpose is just to wait for an existing process to finish. –  Ken Williams Feb 21 '13 at 18:57
    
Indeed. That is a useful aspect of your solution. –  holmb Feb 22 '13 at 14:46
add comment

You can simply add commands to the command line of shell that is already running a command.

Either type carefully, or type it elsewhere, verify, and cut-and-paste. Even if the current command is spewing lines of output, you can still type something new, hit enter, and it will run when all commands running or entered before it complete.

Example follows. You can cut-and-paste the whole thing, or enter the subsequent commands while the 1st is running (if you type really really fast), or more likely while the 2nd is running.:

echo "foo"
sleep 10; echo "bar"
sleep 3
echo "baz"
share|improve this answer
add comment

There is a utility that I have used with great success for exactly the use case that you are describing. Recently I moved my primary laptop to new hardware, which involved moving some files to a NAS and the rest of them to the new machine.

This is how I did it.

  1. Set up all machines involved with a network connection so that they can reach each other.

  2. On the machine where you are moving files from (hereafter called source machine), install rsync with apt-get install rsync and the secret-sauce Task Spooler (website http://vicerveza.homeunix.net/~viric/soft/ts/). If you are on Debian the package name for ts is task-spooler and the executable is renamed to tsp to avoid name clash with the ts executable from the moreutils package. The Debian package linked from the website is perfectly installable on Ubuntu with dpkg -i task-spooler_0.7.3-1_amd64.deb or similar.

  3. Also make sure that all machines have SSH installed with apt-get install openssh-server. On the source machine you need to set up SSH to allow passwordless login to the target machines. The method most will use is public-key authentication with ssh-agent (see https://www.google.se/search?q=ssh+public+key+authentication for examples of that), but I sometimes use an easier method which works just as well with password authentication. Add the following to your SSH client configuration (either ~/.ssh/config or /etc/ssh/ssh_config):

    Host *
         ControlMaster auto
         ControlPath ~/.ssh/master-%r@%h:%p
    

    Then open one terminal on the source machine, login with ssh target1, authenticate as usual, and then leave this terminal open. Notice that there is a socket file named ~/.ssh/master-user@target1:22, this is the file that will keep an authenticated master session open and allow subsequent passwordless connections for user (as long as the connection uses the same target hostname and port).

    At this point you need to verify that you can login without being prompted for authentication to the target machines.

  4. Now run ts rsync -ave ssh bigfile user@target1: for a single file or ts rsync -ave ssh bigdir user@target1: for a directory. With rsync it is important not to include a trailing slash on the directory (bigdir compared to bigdir/) or rsync will assume that you meant the equivalent of bigdir/* in most other tools.

    Task Spooler will return the prompt and let you queue many of these commands in succession. Inspect the run queue with ts without arguments.

Task Spooler has many features such as rearranging the run queue, run a specific job only if another job ran successfully, etc. View the help with ts -h. I sometimes inspect the command output while it is running with ts -c.

There are other methods of doing this but for your use case they all include Task Spooler. I choose to use rsync over SSH to preserve file timestamps, which copying over SMB would not have.

share|improve this answer
add comment

the hackiest way i can think of is to maintain the queue "state" with simple lock-files in /tmp. when file1.sh is doing its cp commands, it will keep a lock file (or hardlink) in /tmp. when it is done, erase the file. every other script will have to look in /tmp for lock files with the naming scheme you choose. naturally this is subject to all sorts of failures, but it is trivial to set up and is doable completely within bash.

share|improve this answer
    
Difficult to use in practice, since it requires that every job you want to run needs to be modified to keep lock files, and needs to know (or accept as parameters) which lock files to use. Whenever possible, it's best to have the queueing external to the job. –  Ken Williams Dec 17 '12 at 16:25
add comment

Command1 && Command2

  • the "&&" means that command2 only runs after command1 finishes with a return code of 0
  • Note: a || means that command2 only runs after command1 finishes with a return code Other than 0
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.