I wrote this ansible task to run a process on a remote vagrant box. (Well, actually the ansible file itself is much longer, but this is a reproducer that only runs the start script.)

- hosts: myappname_server
    - install_myappname_vars.yaml
  gather_facts: false
  sudo: true
  sudo_user: "{{ project_name }}"

  - name: Restart application
    command: "{{ project_target_dir_env }}/run"
      chdir: "{{ project_target_dir_env }}"

It works with these vars in the included vars file:

project_name: myappname
project_source_dir_files: files/myappname
project_source_dir_env: "{{ project_source_dir_files }}/environment_files"
project_target_root: /home/myappname
project_target_dir_env: "{{ project_target_root }}/bin"

The idea is to use user "myappname" on the remote box (aliased correctly by "myappname_server", other plays I run against is work just fine) to run "/home/myappname/bin/run" after changing the directory to "/home/myappname/bin". If I do that manually, everything works fine, i.e. the dirs exist, the files are readable, the script works etc., all great. But if I execute the script, something seems to be wrong with the generation of the ansible execution code. Is it me and my config hope so)? Is it ansible?

I ran it with -vvvv to get a lot of information:

monsterkill@monsterkill-ub-dt:~/playbooks$ ansible-playbook install_myappname_restart.yaml -vvvv

PLAY [myappname_server] ********************************************************** 

TASK: [Restart application] *************************************************** 
<vagrant1> REMOTE_MODULE command chdir=/home/myappname/bin /home/myappname/bin/run
<vagrant1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/monsterkill/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'IdentityFile=/home/monsterkill/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', 'vagrant1', "/bin/sh -c 'mkdir -p /tmp/ansible-tmp-1422343063.07-259463565013754 && chmod a+rx /tmp/ansible-tmp-1422343063.07-259463565013754 && echo /tmp/ansible-tmp-1422343063.07-259463565013754'"]
<vagrant1> PUT /tmp/tmpBduhE7 TO /tmp/ansible-tmp-1422343063.07-259463565013754/command
<vagrant1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/monsterkill/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'IdentityFile=/home/monsterkill/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', 'vagrant1', "/bin/sh -c 'chmod a+r /tmp/ansible-tmp-1422343063.07-259463565013754/command'"]
<vagrant1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/monsterkill/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'IdentityFile=/home/monsterkill/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', 'vagrant1', u'/bin/sh -c \'sudo -k && sudo -H -S -p "[sudo via ansible, key=ucmsbsauynfzeeyxwdmgfduwovdneeqg] password: " -u myappname /bin/sh -c \'"\'"\'echo SUDO-SUCCESS-ucmsbsauynfzeeyxwdmgfduwovdneeqg; /usr/bin/python /tmp/ansible-tmp-1422343063.07-259463565013754/command\'"\'"\'\'']
<vagrant1> EXEC ['ssh', '-C', '-tt', '-vvv', '-o', 'ControlMaster=auto', '-o', 'ControlPersist=60s', '-o', 'ControlPath=/home/monsterkill/.ansible/cp/ansible-ssh-%h-%p-%r', '-o', 'Port=22', '-o', 'IdentityFile=/home/monsterkill/insecure_private_key', '-o', 'KbdInteractiveAuthentication=no', '-o', 'PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey', '-o', 'PasswordAuthentication=no', '-o', 'User=vagrant', '-o', 'ConnectTimeout=10', 'vagrant1', "/bin/sh -c 'rm -rf /tmp/ansible-tmp-1422343063.07-259463565013754/ >/dev/null 2>&1'"]
failed: [vagrant1] => {"cmd": ["/home/myappname/bin/run"], "failed": true, "rc": 8}
msg: [Errno 8] Exec format error

FATAL: all hosts have already failed -- aborting

PLAY RECAP ******************************************************************** 
           to retry, use: --limit @/home/monsterkill/install_myappname_restart.yaml.retry

vagrant1                   : ok=0    changed=0    unreachable=0    failed=1   

I tried things like:

  • playing around with slashes after the directory
  • using with relative and absolute paths on the remote machine
  • work with and without sudo and sudo_user in my tasks

I know that all other ansible modules I use with the same bunch of vars from some neighbouring playbooks work just fine. Also built-in stuff, like group, user, file, apt, unarchive, copy. Note that a number of these also require the group/user stuff to be correct, so I know that's all fine as well.

/edit: I also know that the path to the run script is correct, because if I rename the run script and run the playbook, I get another error ("msg: [Errno 2] No such file or directory", as expected). So it's actually trying to run the existing run script, but failing.

But nothing seems to work. What is going on, what is wrong with that last piece of generated EXEC stuff? Thanks for your time.


If the thing you're trying to run is a shell script, check:

  • That it isn't missing a shebang line at the top like:

    #!/usr/bin/env bash
  • That the user ansible will run as has execute permissions for it (e.g. mode 0755)

In general, 'exec format error' in ansible can mean:

  • a program you gave ansible to execute is, literally, not an executable.
  • ansible has found a file marked as an executable, which is not really executable, and tried to execute it.

In other words: It almost always means permissions are incorrect, but it can happen in both directions (underprivileged or overprivileged files can cause exec errors in different ways).

Personally, I have found that I get such an error when I do things like "chmod 777 /etc/ansible/facts/.." on certain directories and so on.


"Exec format error" means, simply, that you have tried to execute a file that the kernel doesn't recognize as valid program – it is in unsuitable format. In this case that seems to apply to rm on the target server.

Ssh directly into the server and verify that rm works; use file $(which rm) to check its format (compare against other tools like mkdir). Do the same with /usr/bin/python just in case. Perhaps it was copied from a different architecture system, or from a different OS, or outright filled with garbage.

  • Thanks! Don't think so, though :'( rm works fine (works fine on the target box, also with the -rf switch). The target box is a default Ubuntu 14.04 Vagrant setup right out of the box. I checked the output of other playbooks and they can rm their tempfiles just fine, so that can't be it. Also, to me, the error seems to indicate that it encountered an invalid generated "EXEC" structure. Like, the parser forgot to escape a special char or something. The second in the block of three has a weird "u'/bin/sh" in it's line, looks strange... Last thing before removing tempfiles. How did it get there?
    – SadBunny
    Jan 27 '15 at 12:21
  • @SadBunny: Does it? Programs don't use 'errno' for higher level errors; it almost always comes from low-level libc functions. At that point, the only remaining structure is a plain C array given to execvp(); the only errors come from the kernel; and ENOEXEC "Exec format error" has a rather specific meaning. I'd say it's just coincidence that Ansible's log messages also say "EXEC" (since, well, they talk about executing a command).
    – user1686
    Jan 27 '15 at 14:11
  • ok, that's slightly above my paygrade, so to speak :) I'm just trying to do some layman's Sherlocking in this respect... Thanks for the explanation. Still, apparently it isn't a broken 'rm' since other playbooks executed from/to the same systems under the same users. Considering your insights and the "evidence", there may be something wrong with the actual run script or the shell that executes it or something... I'll get back to it tomorrow at work. Thanks for thinking along!
    – SadBunny
    Jan 27 '15 at 14:14
  • By the way, the file $(which rm) gives: vagrant@vagrant-ubuntu-trusty-64:~$ file $(which rm) /bin/rm: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, BuildID[sha1]=da0387e892c7a6dc57acf6618891cdb97f7f605c, stripped
    – SadBunny
    Jan 27 '15 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.