54

I need to do a lot of copying of various files to various folders. I can add all my copy commands to a bash script and then run that, but then I must wait until it finishes if I want to add more commands to that copying "queue".

Is there a way I can run commands as a queue and sort of add more commands to that queue while things are running?

Explained in a different way, I want to start a long running task. While that is running I want to start another one that does not actually start until the first one is done. And then add another after that last one and so on. Is this possible somehow?

2
  • 3
    May I suggest that you accept an answer that solves your problem. It seems that there are a couple of them that would work for you.
    – holmb
    Feb 22, 2013 at 14:49
  • 2
    Yes you may, and I have wanted to. Problem is that I asked this when I worked at a company that were using Macs. I am no longer working there, nor do I have any Macs myself so I have no way of testing any of these. Don't feel comfortable with marking one as an answer when I am unable to test that it actually works, so for now I have hoped that people vote up those answers they find work well for them so the "answer" can emerge in that way instead.
    – Svish
    Feb 24, 2013 at 10:55

8 Answers 8

31

Append & to the end of your command to send it to the background, and then wait on it before running the next. For example:

$ command1 &
$ wait; command2 &
$ wait; command3 &
$ ...
4
  • 2
    yay! this syntax is awesome when you've kicked something off and then go looking at stackoverflow for a way to queue up something after it Oct 16, 2012 at 18:34
  • 4
    If you change your mind on any of those commands in the meantime, what can be done to abort the wait? It seems impervious to all my killshots. Dec 14, 2012 at 23:36
  • 1
    You just kill the command. The wait simply stops the job until previous ones finish. Apr 23, 2015 at 10:04
  • 1
    Will this work with nohup?
    – Michael
    Feb 22, 2018 at 14:45
28

The at utility, best known for running commands at a specified time, also has a queue feature and can be asked to start running commands now. It reads the command to run from standard input.

echo 'command1 --option arg1 arg2' | at -q myqueue now
echo 'command2 ...' | at -q myqueue now

The batch command is equivalent to at -q b -m now (-m meaning the command output, if any, will be mailed to you, like cron does). Not all unix variants support queue names (-q myqueue); you may be limited to a single queue called b. Linux's at is limited to single-letter queue names.

7
  • 4
    On linux at least this does not seem to wait for the previous command to finish.
    – wds
    Feb 16, 2015 at 10:16
  • @wds Works for me on Debian wheezy. Where and how did it break for you? Note that at only waits for the command to finish; if the command launches subprocesses that outlive their parent, at doesn't wait for the subprocesses. Feb 16, 2015 at 13:27
  • I tried on CentOS 6. I tested with just a simple "sleep x". I'm not sure what at does to run that but I assume it starts a subshell and that subshell should wait for the result (a sleep call does not return immediately).
    – wds
    Feb 17, 2015 at 8:43
  • As well as queue names, 'now' doesn't appear standard either: bash on ubuntu is complaining about it...
    – winwaed
    Jan 18, 2016 at 3:15
  • @winwaed Oh, it's now, not -t now, sorry. I hadn't noticed I had it wrong in the sample code. Jan 18, 2016 at 10:55
14

Both Brad and Mankoff's solutions are good suggestions. Another that's similar to a combination of both them would be to use GNU Screen to implement your queue. This has the advantage of being able to run in the background, you can check on it whenever, and queueing up new commands just pastes them into the buffer to be executed after the previous commands exit.

First, run:

$ screen -d -m -S queue

(incidentally, now's a good time to play with some awesome .screenrc files)

That will spawn up a background screen session for you named queue.

Now, queue up as many commands as you like:

screen -S queue -X stuff "echo first; sleep 4; echo second^M"

I'm doing multiple commands in the above just for testing. Your use case would probably look more like:

screen -S queue -X stuff "echo first^M"
screen -S queue -X stuff "echo second^M"

Note that the "^M" in my line above is a way to get an embedded newline that will be interpreted later after screen stuffs it into your existing bash shell. Use "CTL-V, " to get that sequence.

It'd be pretty easy to make some simple shell-scripts to automate that and queue up commands. Then, whenever you want to check the status of your background queue, you re-attach via:

screen -S queue -r

Technically, you don't even need to name your screen session and it will work fine, but once you get hooked on it, you're going to want to leave one running all the time anyway. ;-)

Of course, if you do that, another good way to do it would be to name one of the current windows "queue" and use:

screen -S queue -p queue -X stuff "command"
5
  • It seems like this basically just "types" the command in the running screen session, so that when the shell gains control again after the first command finishes, this command will start. In other words, it's equivalent to @mankoff's solution, but uses a fancier tool to do the typing. Dec 14, 2012 at 23:04
  • Pretty much -- the main difference is that the screen solution supports better automation. You can do some stuff by hand, other things with scripts, all with a simple interface.
    – Jordan
    Jan 29, 2013 at 6:12
  • @Jordan Very good, it works for me. But what is "stuff" for after -X ?
    – Konstantin
    Feb 19, 2019 at 2:12
  • @Konstantin gnu.org/software/screen/manual/html_node/Paste.html#Paste (TL;DR -- stuff whatever follows into the terminal as if you typed it)
    – Jordan
    Feb 28, 2019 at 4:26
  • @Jordan Oh, I see now. It is a command. I thought it is an arbitrary string.
    – Konstantin
    Mar 1, 2019 at 11:53
10

There is a utility that I have used with great success for exactly the use case that you are describing. Recently I moved my primary laptop to new hardware, which involved moving some files to a NAS and the rest of them to the new machine.

This is how I did it.

  1. Set up all machines involved with a network connection so that they can reach each other.

  2. On the machine where you are moving files from (hereafter called source machine), install rsync with apt-get install rsync and the secret-sauce Task Spooler (website http://vicerveza.homeunix.net/~viric/soft/ts/). If you are on Debian the package name for ts is task-spooler and the executable is renamed to tsp to avoid name clash with the ts executable from the moreutils package. The Debian package linked from the website is perfectly installable on Ubuntu with dpkg -i task-spooler_0.7.3-1_amd64.deb or similar.

  3. Also make sure that all machines have SSH installed with apt-get install openssh-server. On the source machine you need to set up SSH to allow passwordless login to the target machines. The method most will use is public-key authentication with ssh-agent (see https://www.google.se/search?q=ssh+public+key+authentication for examples of that), but I sometimes use an easier method which works just as well with password authentication. Add the following to your SSH client configuration (either ~/.ssh/config or /etc/ssh/ssh_config):

    Host *
         ControlMaster auto
         ControlPath ~/.ssh/master-%r@%h:%p
    

    Then open one terminal on the source machine, login with ssh target1, authenticate as usual, and then leave this terminal open. Notice that there is a socket file named ~/.ssh/master-user@target1:22, this is the file that will keep an authenticated master session open and allow subsequent passwordless connections for user (as long as the connection uses the same target hostname and port).

    At this point you need to verify that you can login without being prompted for authentication to the target machines.

  4. Now run ts rsync -ave ssh bigfile user@target1: for a single file or ts rsync -ave ssh bigdir user@target1: for a directory. With rsync it is important not to include a trailing slash on the directory (bigdir compared to bigdir/) or rsync will assume that you meant the equivalent of bigdir/* in most other tools.

    Task Spooler will return the prompt and let you queue many of these commands in succession. Inspect the run queue with ts without arguments.

Task Spooler has many features such as rearranging the run queue, run a specific job only if another job ran successfully, etc. View the help with ts -h. I sometimes inspect the command output while it is running with ts -c.

There are other methods of doing this but for your use case they all include Task Spooler. I choose to use rsync over SSH to preserve file timestamps, which copying over SMB would not have.

4
  • thank you for your answer I didn't know about ts, seem very powerful and easy to use, just forked on github autotoolized and debianized.
    – Alex
    Sep 23, 2014 at 22:03
  • @Alex I had a look at your fork and was especially interested in what was done with regards to autotooling and debianizing the original source, and looking at your commits it was not readily available as a diff to the original source. Would you mind force pushing new commits which in step 1. imports original source and 2. commit with your additions? It will help someone browsing your source to (more easily) trust your additions to the original source. This is one of the few packages I do not install from a PPA or similar so it would be nice to build it myself with your fork.
    – holmb
    Sep 24, 2014 at 15:35
  • you know I've to change my glasses, I lost the part of your answer about the existing task-spooler :-) ... anyway, that's in your comment is a very good suggestion, I'll do very soon.
    – Alex
    Sep 24, 2014 at 15:52
  • done, deleted and re-created the repository with progressive commits, only a couple of mistakes (should metabolize more local commits before push)
    – Alex
    Sep 25, 2014 at 2:29
4

I need stuff like this fairly frequently too. I wrote a little utility called after that executes a command whenever some other process has finished. It looks like this:

#!/usr/bin/perl

my $pid = shift;
die "Usage: $0 <pid> <command...>" unless $pid =~ /^\d+$/ && @ARGV;

print STDERR "Queueing process $$ after process $pid\n";
sleep 1 while -e "/proc/$pid";
exec @ARGV;

You then run it like so:

% command1 arg1 arg2 ...  # creates pid=2853
% after 2853 command2 arg1 arg2 ... # creates pid=9564
% after 9564 command3 arg1 arg2 ...

The big advantage to this over some other approaches is that the first job doesn't need to be run in any special way, you can follow any process with your new job.

3
  • Nice script @ken. Though I'd recommend trying out Task Spooler since you seem to require this from time to time. See my answer.
    – holmb
    Feb 21, 2013 at 14:57
  • Looks neat, thanks. The one thing missing from TS seems to be the ability to follow jobs that weren't started under TS. Though I guess you could start a new TS job whose purpose is just to wait for an existing process to finish. Feb 21, 2013 at 18:57
  • Indeed. That is a useful aspect of your solution.
    – holmb
    Feb 22, 2013 at 14:46
2

Command1 && Command2

  • the "&&" means that command2 only runs after command1 finishes with a return code of 0
  • Note: a || means that command2 only runs after command1 finishes with a return code Other than 0
2
  • And what if I don't care what's the return code?
    – Sam Sirry
    Mar 9, 2020 at 21:04
  • Oh, I found it: Semicolon separates commands on a single line.
    – Sam Sirry
    Mar 9, 2020 at 21:19
1

You can simply add commands to the command line of shell that is already running a command.

Either type carefully, or type it elsewhere, verify, and cut-and-paste. Even if the current command is spewing lines of output, you can still type something new, hit enter, and it will run when all commands running or entered before it complete.

Example follows. You can cut-and-paste the whole thing, or enter the subsequent commands while the 1st is running (if you type really really fast), or more likely while the 2nd is running.:

echo "foo"
sleep 10; echo "bar"
sleep 3
echo "baz"
0

the hackiest way i can think of is to maintain the queue "state" with simple lock-files in /tmp. when file1.sh is doing its cp commands, it will keep a lock file (or hardlink) in /tmp. when it is done, erase the file. every other script will have to look in /tmp for lock files with the naming scheme you choose. naturally this is subject to all sorts of failures, but it is trivial to set up and is doable completely within bash.

1
  • Difficult to use in practice, since it requires that every job you want to run needs to be modified to keep lock files, and needs to know (or accept as parameters) which lock files to use. Whenever possible, it's best to have the queueing external to the job. Dec 17, 2012 at 16:25

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .