0

I have set up Hadoop to run as a linux service in an AWS EC2 instance (deployed via an auto-scaling group) and have it run as a spark user. However, when I start it, I get the following message in the systemctl status of the Hadoop service complaining that it can't verify against the master (I'm using Consul for auto-discovery of master and workers):

Jan 11 23:40:23 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: Stopping namenodes on [spark-master.service.consul]
Jan 11 23:40:23 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: spark-master.service.consul: Host key verification failed.
Jan 11 23:40:23 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: spark-worker.service.consul: Host key verification failed.
Jan 11 23:40:24 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: Stopping secondary namenodes [0.0.0.0]
Jan 11 23:40:24 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: 0.0.0.0: Host key verification failed.
Jan 11 23:40:25 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: stopping yarn daemons
Jan 11 23:40:25 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: no resourcemanager to stop
Jan 11 23:40:25 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: spark-worker.service.consul: Host key verification failed.
Jan 11 23:40:25 ip-172-21-1-19.us-west-2.compute.internal hadoop[12095]: no proxyserver to stop
Jan 11 23:40:25 ip-172-21-1-19.us-west-2.compute.internal systemd[1]: Started Hadoop.

I have tried adding a config file into my spark .ssh directory to ignore host verification:

Host *.service.consul
  StrictHostKeyChecking=no
  UserKnownHostsFile=/dev/null

Which I know works because if I run /opt/hadoop/sbin/start-all.sh as the spark user, it authorizes immediately:

spark-master.service.consul: Warning: Permanently added 'spark-master.service.consul,172.21.3.106' (ECDSA) to the list of known hosts.
spark-master.service.consul: starting namenode, logging to /var/log/hadoop/hadoop-spark-namenode-ip-172-21-3-106.us-west-2.compute.internal.out
spark-worker.service.consul: Warning: Permanently added 'spark-worker.service.consul,172.21.1.19' (ECDSA) to the list of known hosts.
spark-worker.service.consul: datanode running as process 7173. Stop it first.
Starting secondary namenodes [0.0.0.0]
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /var/log/hadoop/hadoop-spark-secondarynamenode-ip-172-21-1-19.us-west-2.compute.internal.out
starting yarn daemons
starting resourcemanager, logging to /opt/hadoop/logs/yarn-spark-resourcemanager-ip-172-21-1-19.us-west-2.compute.internal.out
spark-worker.service.consul: Warning: Permanently added 'spark-worker.service.consul,172.21.1.19' (ECDSA) to the list of known hosts.
spark-worker.service.consul: starting nodemanager, logging to /opt/hadoop/logs/yarn-spark-nodemanager-ip-172-21-1-19.us-west-2.compute.internal.out

And I know for a fact that when I start the service it is running as my spark user:

spark     13987      1  0 23:44 ?        00:00:00 bash /opt/hadoop/sbin/start-all.sh
spark     14000  13987  0 23:44 ?        00:00:00 bash /opt/hadoop/sbin/start-dfs.sh --config /opt/hadoop/etc/hadoop
spark     14074  14000  0 23:44 ?        00:00:00 bash /opt/hadoop/sbin/slaves.sh --config /opt/hadoop/etc/hadoop cd /opt/hadoop ; /opt/hadoop/sbin/hadoop-daemon.sh --config /opt/hadoop/etc/hadoop --script /opt/hadoop/sbin/hdfs start namenode
spark     14099  14074  0 23:44 ?        00:00:00 ssh spark-master.service.consul cd /opt/hadoop ; /opt/hadoop/sbin/hadoop-daemon.sh --config /opt/hadoop/etc/hadoop --script /opt/hadoop/sbin/hdfs start namenode

I need auto-verification for all instances deployed in my Auto-scaling group rather than me having to log into each and every one. So does anyone know how to do this? Is there some setting in my Hadoop service that I'm missing?

This is the service:

[root@ip-172-21-1-19 ~]# cat /usr/lib/systemd/system/hadoop.service
[Unit]
Description=Hadoop
After=syslog.target network.target remote-fs.target nss-lookup.target network-online.target spark-worker.service
Requires=network-online.target spark-worker.service

[Service]
User=spark
Group=spark
Type=forking
PermissionsStartOnly=true
ExecStartPre=/usr/bin/install -o spark -g spark -d /var/run/hadoop
ExecStart=/opt/hadoop/sbin/start-all.sh
ExecStop=/opt/hadoop/sbin/stop-all.sh
WorkingDirectory=/opt/hadoop
TimeoutStartSec=2min
Restart=on-failure
SyslogIdentifier=hadoop
StandardOutput=journal
StandardError=journal
LimitNOFILE=infinity
LimitMEMLOCK=infinity
LimitNPROC=infinity
LimitAS=infinity
SuccessExitStatus=143
RestartSec=20

[Install]
WantedBy=multi-user.target

Please let me know. Thanks.

New contributor
Ethan Stein is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.

Your Answer

Ethan Stein is a new contributor. Be nice, and check out our Code of Conduct.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.