Questions tagged [hadoop]

Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware

Filter by
Sorted by
Tagged with
0
votes
0answers
24 views

pyspark - exception while installing in windows

I installed Anaconda, Java, Spark, pyspark and set the environment variables too, but whenever I run pyspark in Anaconda shell, the exception I get is the following: Exception: Java gateway process ...
0
votes
2answers
142 views

ssh between two ubuntu machines via WAN

We are trying to set-up a two-nodes hadoop clusters on two different machines. SSH to localhost without password have been setted up on both machines. And hadoop can run well as single node on each ...
0
votes
0answers
35 views

HDFS on docker containers - is it possible?

we are using Hadoop cluster based on HDP version - 2.6.5 with ambari platform we want to know if we can use the following HDFS components on docker containers as: journal nodes ZK fail controller ...
-3
votes
1answer
90 views

localhost : ERROR: ssh: connect to host localhost port 22: Address not available

Newbie to docker, trying to set up Hadoop container for which I first create ssh-server container for easy debug as my first container is working fine am able to login ssh localhost within the ...
0
votes
0answers
38 views

how to use eclipse-hadoop-plugin to connect between host machine and guest machine?

I want to use the eclipse which located in the host machine (windows) to operate the hadoop on the guest machines(Centos7). And I use the eclipse-hadoop-plugin to connect. However, there is an error I ...
0
votes
0answers
188 views

Can't create spark session using yarn inside kubernetes pod

I have a kubernetes pod with spark client installed. bash-4.2# spark-shell --version Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/...
0
votes
0answers
46 views

throwable caught when submitting- druid coordinator

I have upgraded my druid cluster from 0.12 to 0.16 recently. coordinator-overload log is flooded with bellow exception now. 2019-10-25T16:57:12,257 ERROR [Master-PeonExec--0] org.apache.druid.server....
2
votes
0answers
22 views

How to avoid (skip) full datanodes on write or replica process

I started a small hadoop cluster for experiment purpose, in my own hardware, with three datanodes with 30GB disk space. Later I added two more nodes with 200GB and now my cluster has approximately ...
0
votes
0answers
40 views

hadoop daemons not starting

I have followed this tutorial to install hadoop on an Ubuntu 18.04 Virtual Machine (4 cores and 10 gb of ram) with learning purposes. I have successfully followed all the steps without any errors. ...
3
votes
2answers
2k views

I can't run hive in command lie

Hive 3.1.2 Hadoop 3.2.1 When I run hive in command line, it tell me the error message below: which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) SLF4J: Class path ...
1
vote
2answers
576 views

Hadoop 3.2.1 namenode formatting not working

I am having [what sounds like] the same issue to Hadoop 3.2.1 namenode formatting Per request to ask this as a new question, and also provide more details: This happens on a Desktop, Windows 10, ...
0
votes
0answers
235 views

datanode and nodemanager shuts down after start-all.cmd

STARTUP_MSG: java = 1.8.0_221 ************************************************************/ 19/10/12 17:03:36 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties 19/10/12 17:...
0
votes
0answers
16 views

spark left join which caused cartesian product

I have two dataframes,one has more than 100 rows,and the other one has only one row.The two dataframes have a same column names "period",and the value of period in two dataframes are all the same,as ...
0
votes
0answers
9 views

Couldn't install Hadoop

I have tried to install Hadoop but each time in command when writing "haddop version" this message appears So what is the problem and how can I solve it?
0
votes
0answers
20 views

Spark not allocating expected number of executors

I have an HDP 3.1 cluster (essentially Spark 2.3 running on Yarn). The cluster has 5 data nodes with 119GB and 32 cores per node. However when I try to run spark jobs, it seems to restrict me to ...
0
votes
0answers
491 views

Hadoop 3.2.1 cannot start yarn, name node

in hadoop yarn i got this error ERROR util.SysInfoWindows: ExitCodeException exitCode=1: PdhAddCounter \Network Interface(*)\Bytes Received/Sec failed with 0xc0000bb8. Error in GetDiskAndNetwork. Err:...
0
votes
0answers
43 views

Inquiry about Hbase Query failure

Good morning I have Hadoop and Hbase configured with HDP. The following error occurred in operating Hbase. =================== Error Log ============================================= 2019-10-01 17:...
0
votes
0answers
31 views

Hive Metastore Standalone Usage

I installed and ran the Metastore Server standalone, without installing Hive. However, I cannot find any documentation about the thrift network API for communicating with the server. I need to be able ...
0
votes
0answers
10 views

Not able to do Backup in cloudera enterprise edition

Backing Up and Restoring HDFS Data Backing Up HDFS Files Establish a Peer Relationship to the Production Cluster Configure the Replication Schedule for the Backup Verify Successful ...
0
votes
0answers
321 views

mkdir: `/user': Input/output error

I just install hadoop I'm trying to to test this exemple : https://tecadmin.net/hadoop-running-a-wordcount-mapreduce-example/ when I tape hdfs dfs -mkdir /user I have this error mkdir: `/user': Input/...
0
votes
0answers
149 views

Error while running yarn for hadoop

While running start-yarn.cmd for hadoop-3.1.2(first time run) I got following error. yarn run v1.17.3 error Couldn't find a package.json file in "C:\hadoop-3.1.2\sbin" info Visit https://yarnpkg.com/...
0
votes
0answers
15 views

Sqoop create dublice values importing from MSSQL

I am noticing an issue with a Sqoop import from MSSQL table, in an Oozie workflow. We have a prod cluster Ambari HDP-2.6.3.0, Oozie 4.2.0. and Sqoop 1.4.6. I am importing with the following comment:...
0
votes
0answers
5 views

can i access hdfs in my local windows sytem from tableau desktop which is also on my local windows

I am trying to access hdfs files which are on my local windows hdfs setup, I want tableau to access these csv files and show some analysis in tableau. I had tried tableau is providing connections to ...
0
votes
0answers
200 views

hadoop failing Start HDFS Daemons

I am installing hadoop as written in documentation here under windows 8. I decided for the hadoop version 2.9.2 I downloaded the winutil from here I relaced the bin folder with the files in the ...
0
votes
0answers
24 views

how do i assign master and slaves node in hadoop?

i'm new to hadoop and ubuntu. I just installed hadoop on three hosts. Now my three hosts can ping each other, but i don't know how to assign the master and slaves. does anyone know how to determine ...
0
votes
0answers
282 views

Write YARN aggregated log to S3 on AWS EMR

I'm using emr-5.26.0 and I followed the documentation here (in the 'To aggregate logs in Amazon S3 using the AWS CLI' section) and I was not able to have the aggregated log file from the my spark app. ...
0
votes
0answers
93 views

Getting 0KB output file for Mapreduce Program given with Apache Hadoop

I have recently setup Hadoop in Pseudo Mode. When I run the mapreduce program given in $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.9.2.jar using the command hadoop jar share/hadoop/...
0
votes
0answers
18 views

How does ranger clean up the space?

We used ranger and now found that the ranger's /opt/ambari_infra_solr/data directory is large and the space is almost full. How to clean up, is there a configuration in the ranger, such as retaining ...
0
votes
0answers
393 views

While setting up the Hadoop 3.1.2, Nodemanager, resourcemanager and datanode keep on shutting down

I have jdk12.0.2 and jre1.8.0_221. java.io.IOException: All specified directories have failed to load. at org.apache.hadoop.hdfs.server.datanode.DataStorage.recoverTransitionRead(DataStorage....
0
votes
0answers
128 views

Hive on Tez: Wrong queue for multitasks script

I am actually running Hive on Tez and when I try to run a multitask Hive script when setting a queue (e.g.: set tez.queue.name=q; select count(*) from t0; select count(*) from t1;), only the first ...
0
votes
0answers
20 views

Ambari 2.5.1. Service Check for HDFS Files View fails during “Transition to undefined”

I'm checking an undermantained multinode lab that has already been having many issues with network configuration due to proxy usage. The view I'm trying to open is the automatically created HDFS ...
0
votes
0answers
21 views

cloudera manager cluster installation step5 (Installing Selected Parcels ) stuck

I use the vmware provided by the official website to add a new cluster action and try to use two nodes for deploy. Repeating it several times is always stuck at this stage and i can't find the problem....
0
votes
0answers
105 views

Installation issue with Hadoop/Snappy on MacOS

I try to install hadoop on one machine with snappy support but meet with the issue below. How to solve it? hadoop checknative -a 19/07/29 16:47:31 DEBUG util.NativeCodeLoader: Trying to load the ...
0
votes
0answers
35 views

Ambari-server always is “pending”

Why is the ambari-server's operation always "pending"?  How can I change it to "in-progress"?
0
votes
0answers
84 views

Hadoop 3.1.2: MapReduce/Yarn AM Port Range?

I'm in an environment with firewall restrictions and to control the port range yarn's application manager uses I'm setting this in mapred-site.xml: <property> <name>yarn.app.mapreduce....
1
vote
0answers
748 views

Hive Installation - Error Executing SQL Query “select ”DB_ID“ from ”DBS“”

I'm trying to install Apache Hive (3.1.1) on a Hadoop (3.2.0) multi-node cluster with 1 namenode and 3 data nodes. I have followed the getting started tutorial step-by-step on the apache website but ...
0
votes
0answers
56 views

apache Ranger Installation issue on centos 6

I am trying to install apache ranger without ambari and getting error while mvn -DskipTests=false clean compile package install assembly:assembly getting error like [ERROR] Failed to execute goal ...
-2
votes
1answer
59 views

kerberos error while creating database

kerberos error while creating database Loading random data Initializing database '/var/lib/krb5kadc/principal' for realm 'HADOOP.COM', master key name 'K/M@HADOOP.COM' You will be prompted for the ...
0
votes
0answers
59 views

/home/subham/Desktop/confluent-5.2.2/bin/confluent start is not working

I have installed confluent perfectly without any error but when I run the command /home/subham/Desktop/confluent-5.2.2/bin/confluent start to start confluent and other services it is showing this. ...
0
votes
0answers
26 views

Deploying Geomesa on OpenShift Origin

I've been trying to replicate some of these manipulations in order to get to deploy Geomesa on openshift. The project is using Hadoop, Accumulo, Zookeeper and Geomesa. The first thing I tried is to ...
1
vote
0answers
41 views

Cannot build cube with apache kylin

I have connected Apache Kylin to our Microsoft SQL Data Warehouse, when trying to build a cube the process is failing with the below error: 19/06/25 15:35:54 INFO sqoop.Sqoop: Running Sqoop version: ...
0
votes
0answers
9 views

Is it possible to use a file inside hadoop file system without copying it locally?

I put a python script inside an hadoop file system running in a cluster of nodes, is it possible for me run the script without make a copy of it locally? for instance run directly in this way “python ...
0
votes
1answer
76 views

HBase & Snappy: CompressionTest Succeeds but Table Create Fails

I'm working with a fresh install of HBase 2.1.4 w/ Hadoop 3.1.2 on RHEL 7.6. In HBase shell, I can create a non-compressed table no problem but this command: create 'testtable', {NAME => 'cf', ...
0
votes
0answers
83 views

No contribution on Spoop2 development / stable release

Why was Sqoop2 deprecated? I can see this alert message in Cloudera: Note: Sqoop 2 is deprecated. Cloudera recommends you use Sqoop 1. Also, they mentioned that the 1.99.x version is not stable ...
0
votes
0answers
119 views

Hadoop Mapreduce job doesn't start properly

in the last few days I've been playing with Hadoop, and I've been trying to implement the k-meters count (find the absolute frequencies of different words lengths) Everything is implemented following ...
0
votes
1answer
116 views

Why are read operations slower than write operations in hadoop?

In cluster mode. We're running different I/O tests in Hadoop. And we noticed that writing data is faster than reading them. How to boost reading throughput to be as fast as the writing throughput. Is ...
0
votes
0answers
206 views

Hadoop file system from non-hadoop machine

I'm having a difficult time finding information on this issue because so many of the results in my searches turn up basic information about copying files from a machine that's part of a cluster. ...
1
vote
0answers
32 views

Sqoop import data from mysql, job does not start

I am using using sqoop 1.4.7 with Hadoop multi node cluster. I have a problem with sqoop import (sqoop list-tables and list-databases work fine). I have a Name Node and 2 data nodes in separate VM (...
1
vote
0answers
62 views

How to run MapReduce jobs without HDFS?

So I'm using Hadoop with 8 nodes, all of them are configured as workers (I'm not interested in the namenode/datanodes architecture, I just need them as compute nodes that run on yarn without HDFS as I'...
0
votes
3answers
5k views

Port binding error in PySpark

I have been trying to get PySpark to work. I use the PyCharm IDE on a Windows 10 machine. For the setup I took these steps: installed PySpark installed Java 8u211 downloaded and pasted the winutils....