Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [hadoop]

Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware

0
votes
0answers
9 views

Error initializing SparkContext [on hold]

When I run this command : ./bin/pyspark, while show the error ERROR spark.SparkContext: Error initializing SparkContext. I think it must be cause of that: java.io.FileNotFoundException: File ...
0
votes
0answers
15 views

Disable Host Verification for Hadoop

I have set up Hadoop to run as a linux service in an AWS EC2 instance (deployed via an auto-scaling group) and have it run as a spark user. However, when I start it, I get the following message in ...
0
votes
0answers
9 views

In general,which is faster, Spark or RDMS?

Spark can load data from Oracle by using JDBC,and then process data by RDD or Spark-SQL.Oracle can also process data by using its native SQL. I wonder which is faster and why when doing same task.
0
votes
0answers
5 views

Hadoop webdfs generates wrong links

So I am hosting a hadoop hdfs with spark on top on a remote server. Currently the hdfs works correctly. The problem I face has to with the webdfs. When I try to download a file with the file browser. ...
0
votes
0answers
16 views

Install hadoop before installing Spark

I have a fresh install of Ubuntu Server on virtualbox. My goal is to achieve the following. Install Hadoop and Spark; Run Spark on top of Hadoop using its hdfs as storage; Install Jupyter and ...
0
votes
0answers
14 views

Zookeeper best practice - using only SSD disks [closed]

we have HDP clusters version 2.6.0 / 2.6.1 ( hortonworks ) , and maybe in the future the version 3.0 I searched a lot in HORTONWORKS documentation , but not found the details about using SSD disks ...
0
votes
0answers
8 views

HDFS “-appendToFile” corrupts last line of file

In a nutshell, here's my example code. file1 = CSV delimited file, say, 1000 rows file2 = CSV delimited file, say, 3000 rows file3 = CSV delimited file, say, 10000 rows hdfs dfs -put file1 /home/...
0
votes
0answers
17 views

Nodemanager and Resourcemanager not showing up in jps

I installed hadoop 2.9.2 and while running start_all.sh command there is no issue creeping up. But when I execute the jps command only 4 daemons show namely Namenode, Datanode, SecondaryNamenode, ...
0
votes
0answers
20 views

Failed to start Hadoop services: localhost: nice: cannot set niceness: Permission denied

I'm trying to install Hadoop on Ubuntu as Windows subsystem, the installation is finished, but I'm getting error while starting Hadoop services. $ start-dfs.sh Starting namenodes on [localhost] ...
1
vote
0answers
10 views

Apache Spark 2.4 is not working with yarn cluster manager?

I have installed Hadoop version 2.8.3 in my windows 10 environment and its working fine. Now when i try to install Apache Spark(version 2.4.0) with yarn as cluster its not working. When i try to ...
1
vote
1answer
29 views

YARN 3 and Spark: allocate a GPU to executors only

I can't find the Spark option to require executors with a GPU. I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark. So far I'm trying out my ...
0
votes
1answer
27 views

Running Hadoop 2.8.0 on Windows 10, YarnRuntimeException Permissions incorrectly set, should be rwxr-xr-x, actual value = rw-rw-rw-

I recently followed article https://www.quora.com/How-do-I-install-Hadoop-2-8-2-on-Windows-10-for-practice, and tried to run Hadoop on my Windows 10 machine (i7, 16GB RAM). It started correctly - no ...
0
votes
0answers
12 views

Using sqoop to connect from one server to another

We're using: sqoop.Sqoop: Running Sqoop version: 1.4.6.2.6.5.19-1 on a Linux system in concordance with Hadoop. and Linux VERSION="7.4 I'm trying to connect bwtween two servers automatically (...
0
votes
1answer
177 views

ssh: connect to host localhost port 22: Connection refused on Windows 10

thanks in advance for the responses ! I've been trying to make a multi node cluster with hadoop, but my mates can't ssh to my server. I have hadoop working, and daemons launched, but when I try to ...
0
votes
0answers
6 views

Multiple kerberos config groups is Ambari

I'm trying to create different configuration group for Kerberos Service in Ambari. I was able to create new config group, but i cannot save any changes in new config group. Button save is simple ...
0
votes
0answers
31 views

HortonWorks HDP setup using a bridged network on VirtualBox

Sorry if I am posting in a wrong place. I am trying to access Hortonworks HDP from another system using SSH in the same network. For that I have configured the virtualBox network using NAT and Bridged ...
0
votes
1answer
25 views

Yum Error in Cent OS 7 Server

I am running into the following error when using yum. yum repolist Loaded plugins: fastestmirror File contains no section headers. file: file:///etc/yum.repos.d/ambari.repo, line: 1 '<!DOCTYPE ...
0
votes
1answer
16 views

Can I make two node Hadoop 2 Cluster on two different VirtualBoxes on two different lap-tops on the same network?

So I have two lap-tops with VirtualBox both running Ubuntu 18.4 and I am trying to set up Hadoop 2 Cluster. One VB to be the master and one VB to be the slave is it possible and how? Because all the ...
2
votes
0answers
241 views

Can you connect two GNOME-boxes via ssh?

I’m trying to set up 2 node Hadoop cluster using two lap-tops with GNOME-boxes, both running Ubuntu 17.10. The both laptops are connectes to the same wifi but the GNOME-boxes alone are not connected ...
0
votes
0answers
10 views

Recover hdfs metadata

I have a single node HDFS server. I accidentally ran the following command: hadoop namenode -format, which deleted the namenode contents. The data of the datanode is still in hdfs (the size on disk ...
0
votes
0answers
138 views

Apache nifi - System Error : “The request contained an invalid host header [localhost:8080]”

I am running Apache nifi in background on my system. While accessing it in my browser via this URL : http://localhost:8080/nifi I am getting this error : System Error : The request contained an ...
0
votes
0answers
17 views

Reg-exp to read weblogic log file

MY weblogic log file sample record is as below.I am looking for a regular expression that can read this file and i can then create a hive external table sing SerDe with regex patern ,please can ...
2
votes
0answers
320 views

What is the difference between headless, service and user Kerberos keytabs

I often work on Kerberised Hadoop clusters, and have to use a variety of keytabs, such as: hbase.headless.keytab hbase.service.keytab devuser.keytab I'm not entirely clear on exactly what the ...
0
votes
0answers
28 views

Hadoop on windows connecting to Hadoop on VirtualBox

I'm running Apache BigTop on VirtualBox and I'd like to develop and connect from windows 10. I've tried several configurations but not getting "100%" of what I'd like to do. Is there a proper ...
0
votes
0answers
144 views

How to edit HDFS group?

I have a standalone cluster for practice purpose. I tried to add the root user to hdfs group by logging in as a HDFS user. The command I used is : hdfs dfs -chown -R root:hdfs / Now, I am able ...
0
votes
0answers
67 views

Difference between Root User on SSH client and Root User when logging in the HDP Sandbox

So I am using Hortonworks Sandbox 2.5 via virtual box and here is my simple query. I can login to my account using an SSH software called putty and using the username maria_dev. I then type su - to ...
0
votes
0answers
10 views

How to run simple web pages from a Hortonworks Sandbox?

I'd like to run a simple web page off a Hortonworks 2.6 sandbox running on VmWare. Nothing really fancy, just a prototype page to test some basics. I found this recipe, but I get a connection ...
-1
votes
1answer
77 views

Putty Loading Previous instance Even though i load right .ppk file

I am trying to connect to Google cloud centos terminal using External IP Address and .ppk file but putty is taking previous loaded IP Address from AWS centos terminal and I loaded right .ppk file ...
0
votes
0answers
35 views

spark.dynamicAllocation + setting the spark parameters according to ambari cluster

we want to find the values for the following spark parameters according to inputs as memory on datanode machine , CPU CORE on data node machine , numbers of datanode machine etc ,, spark....
1
vote
0answers
224 views

Best Practices for Transferring Data Between Hadoop Clusters

Situation: I need to transfer around 100TB from an old Hadoop Cluster to a new Hadoop cluster. The two clusters have a direct 1GbE connection from the namenode on each cluster. The datanodes are not ...
0
votes
1answer
62 views

HDFS datanode use all available space

I have a 1TB (931GB usable) harddrive that I've dedicated to HDFS. I mount it at /hdfs. Here is my hdfs-site.xml <configuration> <property> <name>dfs.replication</name> ...
-1
votes
1answer
34 views

Elliptical Curve Cryptography in Hadoop

Does Hadoop distributed file system use Elliptical curve cryptography anywhere for securing its data? If not, which method is used by it to secure data access?
3
votes
4answers
592 views

Why is Hadoop not a Data Warehouse?

What are functional reasons why Hadoop cannot be a Data Warehouse On several sites one can see statements that a Hadoop cluster is not a replacement of a traditional data warehouse. However, I can't ...
13
votes
2answers
18k views

7Zip Cannot create symbolic link, access is denied to libhdfs.so and libhadoop.so

I am working on Windows 10 and trying to install Hadoop I downloaded it from here. When trying to extract Hadoop for files (libhdfs.so and libhadoop.so) I am getting the error. Cannot create ...
0
votes
1answer
2k views

SSH permission denied from Linux to Windows using keys

I am a beginner in hadoop, and I am trying to set-up hadoop cluster in fully distribution mode. My master is windows and my slave machine is Linux running on VM. I have created SSH keys and in both ...
0
votes
1answer
119 views

Copy files mentioned in List one by one only after completion

I have a list of file names in a text file files.txt which are listed one by one in next line ie. file1.txt file2.txt file3.txt file4.txt ... ... ... I have to execute the following command for each ...
0
votes
2answers
56 views

Will Hadoop benefit from just one disk?

If I use a single node setup with Hadoop where it has only one disk, I can easily use it to play around with Hadoop and stuff, but will there actually be any benefit to using Hadoop in this case over ...
2
votes
2answers
4k views

Open eclipse with a linux user that doesn't have graphical environment (created from console)

I created an user from console in my Ubuntu Destkop 14.04 LTS, doing this: sudo addgroup hadoop sudo adduser --ingroup hadoop hduser I used that user for doing all sort of stuff, because i'm using ...
1
vote
0answers
660 views

How do I find which disk (volume) is failing?

I am using stock Apache Hadoop 1.1.1 and I can't get a datanode to start due to: 2015-04-23 09:12:48,138 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2....
0
votes
1answer
54 views

Hadoop output during run — What is the number right after the timestamp

When Hadoop runs, it puts out the messages we've all seen many times. E.g.: 2015-03-30 10:09:08,750 [main] WARN org.apache.hadoop.conf.Configuration - dfs.data.dir is deprecated. Instead, use dfs....
0
votes
1answer
6k views

ERROR: Can't get Master address from zookeeper; znode data == null

I have two linux machine one is master machine(192.168.8.174) and another one is slave1 machine(192.168.8.173). I have installed and configured Hadoop 2.6.0 in fully distributed mode successfully. ...
0
votes
1answer
267 views

Copy data from local disk to hdfs in a federation of HDFS

I am trying to configure a Federation of HDFS. Therefore, I configured the $ cat etc/hadoop/hdfs-site.xml in both hosts (hadoop-coc-1, and hadoop-coc-2). The problem is that with this configuration, ...
0
votes
1answer
27 views

What does “streaming-off-disk” mean?

I am taking a course offered by www.statistics.com—Advanced Analytics and Machine Learning with Hadoop—and I reading through course material. I stumbled upon the following statement (emphasis mine): ...
0
votes
1answer
944 views

Is it normal behavior for the Reduce tasks to wait until the Map tasks finish?

I have a 50 GB MapReduce job which is (basically) a WordCount app and it has the following Map/Reduce job percentages (listed at the end of the question). It seems like the reducers want to wait until ...
1
vote
0answers
25 views

View all CURRENT parameters for a hadoop installation?

I have inherited a hadoop installation and I would like to query all usage parameters and configuration options. I am sifting through the conf directory and it is helping, but this is an old ...
1
vote
1answer
402 views

Hadoop Uninstall : Shall I remove linux-image packages?

I removed hadoop from my machine via apt-get remove --purge hadoop In the output I see the following packages were not removed : The following packages were automatically installed and are no longer ...
0
votes
1answer
925 views

Permission denied (publickey) error when using scp

I had created 3 linux instances on the Amazon EC2 cloud without a passphrase for the private key. I had stored the private key in a file id_rsa in one of my instances(the master) to allow it to ssh ...
-2
votes
2answers
120 views

Can't download hadoop

Am I taking crazy pills or is it impossible to download hadoop right now? I am looking at http://www.apache.org/dyn/closer.cgi/hadoop/common/ to find mirrors for it and absolutely none of them work ...
1
vote
0answers
339 views

adding nodes to hadoop cluster virtualbox

I am trying to setup a mini hadoop cluster with virtualbox: hosts file (hosts file properly configured): hadoop@rui-VirtualBox:/home/rui$ cat /etc/hosts | grep 10. 10.0.13.96 hnode1 10.0.13.97 ...
1
vote
2answers
1k views

Unable to SSH, cannot create /dev/null/.ssh

I am trying to deploy hadoop with a master and slave node. But the master fails to start tasttracker and datanode processes in slaves, as it is not able to ssh into slaves. The error message is 'Could ...