Can't make Ansible to wait for a server to reboot and continue playbook to work

Question

I am creating a playbook to install a software and I need to restart the server int he middle of the playbook.

I need my Ansible host to wait till the server is back and execute the rest of my tasks but this is not happenning.

These are my Ansible tasks related to reboot and wait:

...
- name: restart server
  shell: shutdown -r now
  async: 1
  poll: 0
  become: yes
  become_method: sudo
  ignore_errors: true
- name: waiting for server to come back after reboot
  local_action: wait_for host={{ ansible_ssh_host }} state=started port=22 delay=30 timeout=300 connect_timeout=15
....

This is the output of running my playbook:

...

TASK [restart server] **********************************************************
fatal: [X.X.X.X]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true}
    to retry, use: --limit @ansible_pb.retry

PLAY RECAP *********************************************************************
X.X.X.X               : ok=2    changed=0    unreachable=1    failed=0

Any idea what is wrong and how can I fix it?

Answer 1

This is a commonly known problem. See Reboot a server and wait for it to come back. Since Ansible 1.9.4 SSH loses connection before proceeding to the next task.

You need to add a delay (sleep) before the shutdown command:

- name: restart server
  shell: sleep 2 && shutdown -r now
  async: 1
  poll: 0
  become: yes
  become_method: sudo
  ignore_errors: true
- name: waiting for server to come back after reboot
  wait_for_connection:
....

Answer 2

On Ansible 2.7, the reboot module was introduced, which solves this problem.

You can use:

- name: restart server
  reboot:

Answer 3

I found that this worked for me for EX-407 (which is based on Ansible 2.3):

- name: restart the server
  shell: (sleep 2 && shutdown -r now) &
  async: 1
  poll: 0
  ignore_errors: true

- name: wait for the server to come back
  wait_for:
    host: "{{ inventory_hostname }}"
    port: 22
    delay: 15
  delegate_to: localhost

What was a surprise to me was that "state: started" option in the wait_for module was initially causing the whole thing to fail; wait_for was not able to discover that port 22 is opened again. Can't explain that behavior though. Once "state" option was removed, the playbook started to work as expected.