0

I am using a HPC composed by 4 machines (master, slave1, slave2 and slave3) I try to run a script on a HPC structure :

mpirun -report-uri - -host master,slave1,slave2,slave3 -np 8 --machinefile machines hellompi

but I'm facing this error message :

657129472.0;tcp://102.102.0.1,101.101.0.1,100.100.0.2,103.103.0.1:54761
[charlotte-ProLiant-DL380-Gen10-slave1:07172] [[10027,0],1] tcp_peer_send_blocking: send() to socket 9 failed: Broken pipe (32)

ORTE was unable to reliably start one or more daemons.
This usually is caused by:

* not finding the required libraries and/or binaries on
  one or more nodes. Please check your PATH and LD_LIBRARY_PATH
  settings, or configure OMPI with --enable-orterun-prefix-by-default

* lack of authority to execute on one or more specified nodes.
  Please verify your allocation and authorities.

* the inability to write startup files into /tmp (--tmpdir/orte_tmpdir_base).
  Please check with your sys admin to determine the correct location to use.

*  compilation of the orted with dynamic libraries when static are required
  (e.g., on Cray). Please check your configure cmd line and consider using
  one of the contrib/platform definitions for your system type.

* an inability to create a connection back to mpirun due to a
  lack of common network interfaces and/or no route found between
  them. Please check network connectivity (including firewalls
  and network routing requirements).

I am working on Ubuntu. Firewalls (ufw) are disabled on each machine. ssh login is okay, even in passwordless mode. Mpirun version is the same on each machine.

The script I am trying to run is a simple fortran code :

program hello
include 'mpif.h'
integer rank, size, ierror, nl
character(len=MPI_MAX_PROCESSOR_NAME) :: hostname

call MPI_INIT(ierror)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
call MPI_GET_PROCESSOR_NAME(hostname, nl, ierror)
print*, 'node', rank, ' of', size, ' on ', hostname(1:nl), ': Hello world'
call MPI_FINALIZE(ierror)
end

Only master and slave2 are okay :

mpirun -report-uri - -host master,slave2 -np 4 --machinefile machines hellompi

669843456.0;tcp://102.102.0.1,101.101.0.1,100.100.0.2,103.103.0.1:35387
node           0  of           4  on charlotte-ProLiant-DL380-Gen10-master: Hello world
node           2  of           4  on charlotte-ProLiant-DL380-Gen10-master: Hello world
node           1  of           4  on charlotte-ProLiant-DL380-Gen10-slave2: Hello world
node           3  of           4  on charlotte-ProLiant-DL380-Gen10-slave2: Hello world

The machine file contains :

master slots=2 max_slots=2
slave1 slots=2 max_slots=2
slave2 slots=2 max_slots=2
slave3 slots=2 max_slots=2

On master, ifconfig gives :

eno1      Link encap:Ethernet  HWaddr 54:80:28:57:0f:7e  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B)
          Interruption:16 

eno2      Link encap:Ethernet  HWaddr 54:80:28:57:0f:7f  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B)
          Interruption:17 

eno3      Link encap:Ethernet  HWaddr 54:80:28:57:0f:80  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B)
          Interruption:16 

eno4      Link encap:Ethernet  HWaddr 54:80:28:57:0f:81  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B)
          Interruption:17 

eno5      Link encap:Ethernet  HWaddr 80:30:e0:31:b1:68  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B)
          Interruption:32 Mémoire:e7800000-e7ffffff 

eno6      Link encap:Ethernet  HWaddr 80:30:e0:31:b1:6c  
          inet adr:100.100.0.2  Bcast:100.100.0.255  Masque:255.255.255.0
          adr inet6: fe80::9451:8431:7010:46/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Packets reçus:6511 erreurs:0 :0 overruns:0 frame:0
          TX packets:7854 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:1126379 (1.1 MB) Octets transmis:5446489 (5.4 MB)
          Interruption:144 Mémoire:e8800000-e8ffffff 

ens2f0    Link encap:Ethernet  HWaddr 20:67:7c:06:5f:a8  
          inet adr:102.102.0.1  Bcast:102.102.0.255  Masque:255.255.255.0
          adr inet6: fe80::67eb:b74d:b6a4:c4ea/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Packets reçus:2324 erreurs:0 :0 overruns:0 frame:0
          TX packets:1455 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:1517185 (1.5 MB) Octets transmis:214236 (214.2 KB)
          Interruption:28 Mémoire:e3000000-e37fffff 

ens2f1    Link encap:Ethernet  HWaddr 20:67:7c:06:5f:ac  
          inet adr:101.101.0.1  Bcast:101.101.0.255  Masque:255.255.255.0
          adr inet6: fe80::df77:4d6c:742f:a248/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Packets reçus:1035 erreurs:0 :0 overruns:0 frame:0
          TX packets:1034 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:154354 (154.3 KB) Octets transmis:138630 (138.6 KB)
          Interruption:123 Mémoire:e4000000-e47fffff 

ens5f0    Link encap:Ethernet  HWaddr 20:67:7c:06:5f:a0  
          UP BROADCAST MULTICAST  MTU:1500  Metric:1
          Packets reçus:0 erreurs:0 :0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:0 (0.0 B) Octets transmis:0 (0.0 B)
          Interruption:34 Mémoire:f0000000-f07fffff 

ens5f1    Link encap:Ethernet  HWaddr 20:67:7c:06:5f:a4  
          inet adr:103.103.0.1  Bcast:103.103.0.255  Masque:255.255.255.0
          adr inet6: fe80::d4de:7007:c103:83b9/64 Scope:Lien
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          Packets reçus:542 erreurs:0 :0 overruns:0 frame:0
          TX packets:487 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:49945 (49.9 KB) Octets transmis:36934 (36.9 KB)
          Interruption:165 Mémoire:f1000000-f17fffff 

lo        Link encap:Boucle locale  
          inet adr:127.0.0.1  Masque:255.0.0.0
          adr inet6: ::1/128 Scope:Hôte
          UP LOOPBACK RUNNING  MTU:65536  Metric:1
          Packets reçus:30126 erreurs:0 :0 overruns:0 frame:0
          TX packets:30126 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 lg file transmission:1000 
          Octets reçus:7903814 (7.9 MB) Octets transmis:7903814 (7.9 MB)

Any idea what's going on ?

New contributor
Charlotte Rieux is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

Charlotte Rieux is a new contributor. Be nice, and check out our Code of Conduct.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.