Questions tagged [hadoop]

Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware

Filter by
Sorted by
Tagged with
30 votes
2 answers
71k views

7Zip Cannot create symbolic link, access is denied to libhdfs.so and libhadoop.so

I am working on Windows 10 and trying to install Hadoop I downloaded it from here. When trying to extract Hadoop for files (libhdfs.so and libhadoop.so) I am getting the error. Cannot create ...
Matas Vaitkevicius's user avatar
4 votes
4 answers
1k views

Why is Hadoop not a Data Warehouse?

What are functional reasons why Hadoop cannot be a Data Warehouse On several sites one can see statements that a Hadoop cluster is not a replacement of a traditional data warehouse. However, I can't ...
Dennis Jaheruddin's user avatar
4 votes
2 answers
7k views

Open eclipse with a linux user that doesn't have graphical environment (created from console)

I created an user from console in my Ubuntu Destkop 14.04 LTS, doing this: sudo addgroup hadoop sudo adduser --ingroup hadoop hduser I used that user for doing all sort of stuff, because i'm using ...
chomp's user avatar
  • 143
4 votes
2 answers
4k views

I can't run hive in command line

Hive 3.1.2 Hadoop 3.2.1 When I run hive in command line, it tell me the error message below: which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin) SLF4J: Class path ...
Smith Cruise's user avatar
4 votes
1 answer
4k views

Hive Installation - Error Executing SQL Query "select "DB_ID" from "DBS""

I'm trying to install Apache Hive (3.1.1) on a Hadoop (3.2.0) multi-node cluster with 1 namenode and 3 data nodes. I have followed the getting started tutorial step-by-step on the apache website but ...
A Tol's user avatar
  • 41
4 votes
0 answers
1k views

Can you connect two GNOME-boxes via ssh?

I’m trying to set up 2 node Hadoop cluster using two lap-tops with GNOME-boxes, both running Ubuntu 17.10. The both laptops are connectes to the same wifi but the GNOME-boxes alone are not connected ...
anagardastro's user avatar
4 votes
0 answers
3k views

Hadoop - java.io.IOException: Connection reset by peer when creating when creating a new directory

I have installed hadoop 2.4.0 as a single node for learning purposes but after I start hadoop and create a directory using the command: hadoop fs -mkdir /tmp I get the following error: ls: Failed ...
Qurashi's user avatar
  • 141
3 votes
3 answers
14k views

Port binding error in PySpark

I have been trying to get PySpark to work. I use the PyCharm IDE on a Windows 10 machine. For the setup I took these steps: installed PySpark installed Java 8u211 downloaded and pasted the winutils....
Moritz's user avatar
  • 111
3 votes
4 answers
4k views

Unable to send files via SCP

I tried using the scp command with the -i option to transfer the file from local machine to remote EC2 instance: Akhis-Macbook-Pro:~ aswinakhilesh$ sudo scp -i Mykey.pem FileA ec2-user@ec2-23-20-46-...
aswin akhilesh's user avatar
3 votes
2 answers
2k views

Setting Hostname as IP on Linux for Hadoop VM

How can I have an Ubuntu server VM set /etc/hostname/ to the value fo the VM's assigned IP address automatically on startup? I'm creating an Ubuntu server VM image to run Hadoop. When a client ...
EngineerBetter_DJ's user avatar
3 votes
2 answers
30k views

Slave: ssh: connect to host slave port 22: Connection timed out

i have done with single node cluster on two different machine,I have made one as master(192.168.1.1) and other m/c as slave(192.168.1.2), I am successfully able to ping between two machine,I have made ...
Aashu's user avatar
  • 131
3 votes
1 answer
4k views

Error when Starting CYGWin SSH daemon on windows 7 home edition

I am using Windows 7 Home edition, trying to install CYGWin SSH daemon. Successfully done the same, but when I run the same, I see error in C:\cygwin\var\log\sshd.log file Privilege separation user ...
user avatar
2 votes
2 answers
2k views

What is the difference between headless, service and user Kerberos keytabs

I often work on Kerberised Hadoop clusters, and have to use a variety of keytabs, such as: hbase.headless.keytab hbase.service.keytab devuser.keytab I'm not entirely clear on exactly what the ...
Ben Watson's user avatar
2 votes
1 answer
3k views

How can I access the ganglia web interface with ssh tunneling to monitor my EMR job?

Ive been using the standard hadoop monitoring tools with: ssh -L 9100:localhost:9100 -L 9101:localhost:9101 -o ServerAliveInterval=10 -o StrictHostKeyChecking=no -i key.pem hadoop@ec2-blah-blah-....
kelorek's user avatar
  • 210
2 votes
1 answer
2k views

JAVA_HOME not set properly

I am trying to configure hadoop in cygwin. I have set JAVA_HOME as /cygdrive/c/work/java/jdk1.6.0_30 If I echo $JAVA_HOME, it displays correctly. If I run the command bin/hadoop version it is giving ...
Rashmi's user avatar
  • 31
2 votes
0 answers
74 views

How to avoid (skip) full datanodes on write or replica process

I started a small hadoop cluster for experiment purpose, in my own hardware, with three datanodes with 30GB disk space. Later I added two more nodes with 200GB and now my cluster has approximately ...
José Luis Capote Fernández's user avatar
2 votes
0 answers
406 views

Best Practices for Transferring Data Between Hadoop Clusters

Situation: I need to transfer around 100TB from an old Hadoop Cluster to a new Hadoop cluster. The two clusters have a direct 1GbE connection from the namenode on each cluster. The datanodes are not ...
tbenz9's user avatar
  • 7,095
1 vote
2 answers
4k views

Unable to SSH, cannot create /dev/null/.ssh

I am trying to deploy hadoop with a master and slave node. But the master fails to start tasttracker and datanode processes in slaves, as it is not able to ssh into slaves. The error message is 'Could ...
krackoder's user avatar
  • 125
1 vote
1 answer
1k views

Check Oozie status in UNIX

Hadoop Admin Utility Created a basic hadoop admin utility which checks dfs health and oozie status. Checking oozie status In my above utility, I am using the below command : bin/oozie admin -...
Debaditya's user avatar
  • 115
1 vote
1 answer
1k views

Running Hadoop on Windows 8

Running Hadoop framework on Windows is always a pain using Cygwin. Is the process similar for Windows 8 just like Windows 7 as mentioned here or is it different ? http://alans.se/blog/2010/hadoop-...
CodeMonkey's user avatar
1 vote
1 answer
9k views

Disable RAID Controller at BOOT and Partition Each Disk

I have some decent HP Proliants server that come with "HP Smart Array P410i Controller" enabled, I am using these boxes to set up a Hadoop cluster and I know, RAID is for sure a no-no for Hadoop since ...
B.Mr.W.'s user avatar
  • 127
1 vote
1 answer
2k views

Configuring hadoop results in "/bin/java : no such file or directory"

I am trying to install hadoop on Cygwin. I have set JAVA_HOME as follows: \cygdrive\c\work\java\jdk1.6.0_30 in the file conf/hadoop-env.sh. Performing a echo $JAVA_HOME also displays correctly ...
Rashmi's user avatar
  • 31
1 vote
1 answer
981 views

YARN 3 and Spark: allocate a GPU

I can't find the working Spark option to require executors with a GPU. I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark. So far I'm trying ...
Kévin Azoulay's user avatar
1 vote
1 answer
5k views

SSH permission denied from Linux to Windows using keys

I am a beginner in hadoop, and I am trying to set-up hadoop cluster in fully distribution mode. My master is windows and my slave machine is Linux running on VM. I have created SSH keys and in both ...
Sriram Chandramouli's user avatar
1 vote
1 answer
2k views

Sqoop JDBC connection timeout after import but before hive import

We are using Sqoop v1.4.4 14/05/27 13:49:14 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.0 Sqoop 1.4.4-cdh5.0.0 git commit id 8e266e052e423af592871e2dfe09d54c03f6a0e8 When I'm importing a ...
Andrey Dmitriev's user avatar
1 vote
5 answers
2k views

Mac OS X 10.9 Mavericks breaks passwordless SSH to localhost (for hadoop)

Sorry if this sounds like a repeat, but I'm fairly sure it's specifically Mac OS X 10.9 related, it works fine for me on my 10.8 machine. I've done everything identically, as usual: $:~ ssh-keygen -t ...
EdB's user avatar
  • 11
1 vote
2 answers
383 views

How to start after installing OpenStack

I have installed OpenStack following the tutorial from here I want to know that how can I deploy simple PHP application using Apache Hadoop on OpenStack? Any reference?
Umair Ayub's user avatar
1 vote
1 answer
2k views

How to remove large number of files/folders in linux

We are using hadoop to split a table into smaller files to feed to mahout, but in the process, we created a huge amount of _temporary logs. We have an NFS mount for the hadoop volume so we can use ...
user1745713's user avatar
1 vote
2 answers
2k views

wants to remove cdh4(hadoop) from ubuntu 12.04

i have installed cdh3 pseudo mode in my ubuntu 12.04(precise) system using the following web reference cdh3-install-ubuntu12.04 now i tried to install cdh4 to replace cdh3 but,unfortunately found ...
Ferdinand Rosario's user avatar
1 vote
1 answer
2k views

Pig tutorial in local mode returns OutOfMemoryError

I am new with pig and I try to learn from pig tutorial on Apache website. I am working with hadoop 1.0.1 and PigVersion 0.11.1. In the tutorial, it is proposed to try pig on two examples (called ...
MarAja's user avatar
  • 321
1 vote
1 answer
1k views

Modify LD_LIBRARY_PATH and CLASSPATH on cluster node before running a hadoop job

I need to set LD_LIBRARY_PATH and CLASSPATH before running a job a cluster. In LD_LIBRARY_PATH i need to add location of some jars which are required while running the job, As these jars are ...
Ashish Sharma's user avatar
1 vote
0 answers
326 views

SSH to localhost keeps asking for password on MacBook Air M1

I am trying to install hadoop on my MacBook Air M1 and when I write start-all.sh in terminal it prompts fatix@localhost: Permission denied (publickey,password,keyboard-interactive). I did following ...
user1785339's user avatar
1 vote
2 answers
3k views

Hadoop 3.2.1 namenode formatting not working

I am having [what sounds like] the same issue to Hadoop 3.2.1 namenode formatting Per request to ask this as a new question, and also provide more details: This happens on a Desktop, Windows 10, ...
SteveT's user avatar
  • 11
1 vote
0 answers
71 views

Cannot build cube with apache kylin

I have connected Apache Kylin to our Microsoft SQL Data Warehouse, when trying to build a cube the process is failing with the below error: 19/06/25 15:35:54 INFO sqoop.Sqoop: Running Sqoop version: ...
Bjorn Demicoli's user avatar
1 vote
1 answer
875 views

Why are read operations slower than write operations in hadoop?

In cluster mode. We're running different I/O tests in Hadoop. And we noticed that writing data is faster than reading them. How to boost reading throughput to be as fast as the writing throughput. Is ...
user9332151's user avatar
1 vote
0 answers
56 views

Sqoop import data from mysql, job does not start

I am using using sqoop 1.4.7 with Hadoop multi node cluster. I have a problem with sqoop import (sqoop list-tables and list-databases work fine). I have a Name Node and 2 data nodes in separate VM (...
abdou's user avatar
  • 11
1 vote
0 answers
281 views

How to run MapReduce jobs without HDFS?

So I'm using Hadoop with 8 nodes, all of them are configured as workers (I'm not interested in the namenode/datanodes architecture, I just need them as compute nodes that run on yarn without HDFS as I'...
user9332151's user avatar
1 vote
1 answer
2k views

How to copy files from Windows to Linux HDFS directly

I am trying to copy a large amount files to Hadoop HIVE. Right now I am doing this via a two-step scripts. Step 1: Powershell scripts copying files from Windows to Linux using Putty scp tool. Step ...
Seaport's user avatar
  • 23
1 vote
0 answers
337 views

Sqoop import sql server table process shows no progress

This is my first-try using Apache Sqoop to import a SQL Server table (6 columns, 4 records) to Hive. Below is the code. sqoop import --connect "jdbc:sqlserver://192.168.10.101:1433;database=Testdb" --...
Seaport's user avatar
  • 23
1 vote
0 answers
4k views

Nodemanager and Resourcemanager not showing up in jps

I installed hadoop 2.9.2 and while running start_all.sh command there is no issue creeping up. But when I execute the jps command only 4 daemons show namely Namenode, Datanode, SecondaryNamenode, ...
Saran V S's user avatar
1 vote
0 answers
69 views

Apache Spark 2.4 is not working with yarn cluster manager?

I have installed Hadoop version 2.8.3 in my windows 10 environment and its working fine. Now when i try to install Apache Spark(version 2.4.0) with yarn as cluster its not working. When i try to ...
Anubhav Jain's user avatar
1 vote
0 answers
1k views

How do I find which disk (volume) is failing?

I am using stock Apache Hadoop 1.1.1 and I can't get a datanode to start due to: 2015-04-23 09:12:48,138 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2....
drjrm3's user avatar
  • 1,496
1 vote
0 answers
31 views

View all CURRENT parameters for a hadoop installation?

I have inherited a hadoop installation and I would like to query all usage parameters and configuration options. I am sifting through the conf directory and it is helping, but this is an old ...
drjrm3's user avatar
  • 1,496
1 vote
1 answer
561 views

Hadoop Uninstall : Shall I remove linux-image packages?

I removed hadoop from my machine via apt-get remove --purge hadoop In the output I see the following packages were not removed : The following packages were automatically installed and are no longer ...
Soumitra's user avatar
  • 113
1 vote
0 answers
448 views

adding nodes to hadoop cluster virtualbox

I am trying to setup a mini hadoop cluster with virtualbox: hosts file (hosts file properly configured): hadoop@rui-VirtualBox:/home/rui$ cat /etc/hosts | grep 10. 10.0.13.96 hnode1 10.0.13.97 ...
SQL.injection's user avatar
1 vote
0 answers
253 views

hadoop cluster high availability error couldn't find or load main class

[hduser@mn1 bin]$ ./hdfs zkfc -formatZK Error: Could not find or load main class ”-Djava.library.path=”.home.hduser.2.3.0″.lib” I did every step just like in this tutorial, TWICE. The error still the ...
Tarek's user avatar
  • 141
1 vote
0 answers
1k views

Why does eclipse report an error when I use it under a hadoop account?

I created two accounts, one is a common user, the other is just for hadoop, which is a superuser as root. I can load eclipse normally under my common user account, but I could not use it under hadoop ...
city's user avatar
  • 393
1 vote
0 answers
911 views

Cygwin ssh-host-config on windows for hadoop

I'm new to cygwin and hadoop, please anyone could explain which mirror link is the proper to install cygwin what version of hadoop is bug free ( .sh works good ) is it must to run below commands ...
Josh's user avatar
  • 111
1 vote
1 answer
2k views

Hadoop installation on ubuntu

I installed Hadoop on ubuntu from the .deb package when I run start-all.sh I get this error I configured the core-site.xml for hdfs:localhost:9001 but i gives me the error. I think the problem in the ...
Mohammad Alkahtani's user avatar
0 votes
1 answer
9k views

kernel parameter - net.core.netdev_max_backlog , what happened when its too low

we want to understand what are the aspects when net.core.netdev_max_backlog kernel value is very low and not as recommended on our Linux RHEL machines , the value for this parameter is 1000 since our ...
King David's user avatar