Questions tagged [hadoop]
Hadoop is an open-source software framework for distributed storage and distributed processing of very large data sets on computer clusters built from commodity hardware
97
questions
30
votes
2
answers
71k
views
7Zip Cannot create symbolic link, access is denied to libhdfs.so and libhadoop.so
I am working on Windows 10 and trying to install Hadoop
I downloaded it from here.
When trying to extract Hadoop for files (libhdfs.so and libhadoop.so) I am getting the error.
Cannot create ...
4
votes
4
answers
1k
views
Why is Hadoop not a Data Warehouse?
What are functional reasons why Hadoop cannot be a Data Warehouse
On several sites one can see statements that a Hadoop cluster is not a replacement of a traditional data warehouse. However, I can't ...
4
votes
2
answers
7k
views
Open eclipse with a linux user that doesn't have graphical environment (created from console)
I created an user from console in my Ubuntu Destkop 14.04 LTS, doing this:
sudo addgroup hadoop
sudo adduser --ingroup hadoop hduser
I used that user for doing all sort of stuff, because i'm using ...
4
votes
2
answers
4k
views
I can't run hive in command line
Hive 3.1.2
Hadoop 3.2.1
When I run hive in command line, it tell me the error message below:
which: no hbase in (/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin)
SLF4J: Class path ...
4
votes
1
answer
4k
views
Hive Installation - Error Executing SQL Query "select "DB_ID" from "DBS""
I'm trying to install Apache Hive (3.1.1) on a Hadoop (3.2.0) multi-node cluster with 1 namenode and 3 data nodes.
I have followed the getting started tutorial step-by-step on the apache website but ...
4
votes
0
answers
1k
views
Can you connect two GNOME-boxes via ssh?
I’m trying to set up 2 node Hadoop cluster using two lap-tops with GNOME-boxes, both running Ubuntu 17.10. The both laptops are connectes to the same wifi but the GNOME-boxes alone are not connected ...
4
votes
0
answers
3k
views
Hadoop - java.io.IOException: Connection reset by peer when creating when creating a new directory
I have installed hadoop 2.4.0 as a single node for learning purposes but after I start hadoop and create a directory using the command:
hadoop fs -mkdir /tmp
I get the following error:
ls: Failed ...
3
votes
3
answers
14k
views
Port binding error in PySpark
I have been trying to get PySpark to work. I use the PyCharm IDE on a Windows 10 machine. For the setup I took these steps:
installed PySpark
installed Java 8u211
downloaded and pasted the winutils....
3
votes
4
answers
4k
views
Unable to send files via SCP
I tried using the scp command with the -i option to transfer the file from local machine to remote EC2 instance:
Akhis-Macbook-Pro:~ aswinakhilesh$ sudo scp -i Mykey.pem FileA ec2-user@ec2-23-20-46-...
3
votes
2
answers
2k
views
Setting Hostname as IP on Linux for Hadoop VM
How can I have an Ubuntu server VM set /etc/hostname/ to the value fo the VM's assigned IP address automatically on startup?
I'm creating an Ubuntu server VM image to run Hadoop. When a client ...
3
votes
2
answers
30k
views
Slave: ssh: connect to host slave port 22: Connection timed out
i have done with single node cluster on two different machine,I have made one as master(192.168.1.1) and other m/c as slave(192.168.1.2), I am successfully able to ping between two machine,I have made ...
3
votes
1
answer
4k
views
Error when Starting CYGWin SSH daemon on windows 7 home edition
I am using Windows 7 Home edition, trying to install CYGWin SSH daemon.
Successfully done the same, but when I run the same, I see error in C:\cygwin\var\log\sshd.log file
Privilege separation user ...
2
votes
2
answers
2k
views
What is the difference between headless, service and user Kerberos keytabs
I often work on Kerberised Hadoop clusters, and have to use a variety of keytabs, such as:
hbase.headless.keytab
hbase.service.keytab
devuser.keytab
I'm not entirely clear on exactly what the ...
2
votes
1
answer
3k
views
How can I access the ganglia web interface with ssh tunneling to monitor my EMR job?
Ive been using the standard hadoop monitoring tools with:
ssh -L 9100:localhost:9100 -L 9101:localhost:9101 -o ServerAliveInterval=10 -o StrictHostKeyChecking=no -i key.pem hadoop@ec2-blah-blah-....
2
votes
1
answer
2k
views
JAVA_HOME not set properly
I am trying to configure hadoop in cygwin.
I have set JAVA_HOME as /cygdrive/c/work/java/jdk1.6.0_30
If I echo $JAVA_HOME, it displays correctly.
If I run the command bin/hadoop version it is giving ...
2
votes
0
answers
74
views
How to avoid (skip) full datanodes on write or replica process
I started a small hadoop cluster for experiment purpose, in my own hardware, with three datanodes with 30GB disk space. Later I added two more nodes with 200GB and now my cluster has approximately ...
2
votes
0
answers
406
views
Best Practices for Transferring Data Between Hadoop Clusters
Situation:
I need to transfer around 100TB from an old Hadoop Cluster to a new Hadoop cluster. The two clusters have a direct 1GbE connection from the namenode on each cluster. The datanodes are not ...
1
vote
2
answers
4k
views
Unable to SSH, cannot create /dev/null/.ssh
I am trying to deploy hadoop with a master and slave node. But the master fails to start tasttracker and datanode processes in slaves, as it is not able to ssh into slaves. The error message is 'Could ...
1
vote
1
answer
1k
views
Check Oozie status in UNIX
Hadoop Admin Utility
Created a basic hadoop admin utility which checks dfs health and oozie status.
Checking oozie status
In my above utility, I am using the below command :
bin/oozie admin -...
1
vote
1
answer
1k
views
Running Hadoop on Windows 8
Running Hadoop framework on Windows is always a pain using Cygwin. Is the process similar for Windows 8 just like Windows 7 as mentioned here or is it different ?
http://alans.se/blog/2010/hadoop-...
1
vote
1
answer
9k
views
Disable RAID Controller at BOOT and Partition Each Disk
I have some decent HP Proliants server that come with "HP Smart Array P410i Controller" enabled, I am using these boxes to set up a Hadoop cluster and I know, RAID is for sure a no-no for Hadoop since ...
1
vote
1
answer
2k
views
Configuring hadoop results in "/bin/java : no such file or directory"
I am trying to install hadoop on Cygwin.
I have set JAVA_HOME as follows:
\cygdrive\c\work\java\jdk1.6.0_30
in the file conf/hadoop-env.sh.
Performing a echo $JAVA_HOME also displays correctly
...
1
vote
1
answer
981
views
YARN 3 and Spark: allocate a GPU
I can't find the working Spark option to require executors with a GPU.
I'm trying to setup a HADOOP cluster in order to run Machine Learning algorithms on available GPUs via Spark.
So far I'm trying ...
1
vote
1
answer
5k
views
SSH permission denied from Linux to Windows using keys
I am a beginner in hadoop, and I am trying to set-up hadoop cluster in fully distribution mode. My master is windows and my slave machine is Linux running on VM. I have created SSH keys and in both ...
1
vote
1
answer
2k
views
Sqoop JDBC connection timeout after import but before hive import
We are using Sqoop v1.4.4
14/05/27 13:49:14 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4-cdh5.0.0
Sqoop 1.4.4-cdh5.0.0
git commit id 8e266e052e423af592871e2dfe09d54c03f6a0e8
When I'm importing a ...
1
vote
5
answers
2k
views
Mac OS X 10.9 Mavericks breaks passwordless SSH to localhost (for hadoop)
Sorry if this sounds like a repeat, but I'm fairly sure it's specifically Mac OS X 10.9 related, it works fine for me on my 10.8 machine. I've done everything identically, as usual:
$:~ ssh-keygen -t ...
1
vote
2
answers
383
views
How to start after installing OpenStack
I have installed OpenStack following the tutorial from here I want to know that how can I deploy simple PHP application using Apache Hadoop on OpenStack?
Any reference?
1
vote
1
answer
2k
views
How to remove large number of files/folders in linux
We are using hadoop to split a table into smaller files to feed to mahout, but in the process, we created a huge amount of _temporary logs.
We have an NFS mount for the hadoop volume so we can use ...
1
vote
2
answers
2k
views
wants to remove cdh4(hadoop) from ubuntu 12.04
i have installed cdh3 pseudo mode in my ubuntu 12.04(precise) system using the following web reference
cdh3-install-ubuntu12.04
now i tried to install cdh4 to replace cdh3 but,unfortunately found ...
1
vote
1
answer
2k
views
Pig tutorial in local mode returns OutOfMemoryError
I am new with pig and I try to learn from pig tutorial on Apache website. I am working with hadoop 1.0.1 and PigVersion 0.11.1. In the tutorial, it is proposed to try pig on two examples (called ...
1
vote
1
answer
1k
views
Modify LD_LIBRARY_PATH and CLASSPATH on cluster node before running a hadoop job
I need to set LD_LIBRARY_PATH and CLASSPATH before running a job a cluster. In LD_LIBRARY_PATH i need to add location of some jars which are required while running the job, As these jars are ...
1
vote
0
answers
326
views
SSH to localhost keeps asking for password on MacBook Air M1
I am trying to install hadoop on my MacBook Air M1 and when I write start-all.sh in terminal it prompts
fatix@localhost: Permission denied (publickey,password,keyboard-interactive).
I did following ...
1
vote
2
answers
3k
views
Hadoop 3.2.1 namenode formatting not working
I am having [what sounds like] the same issue to Hadoop 3.2.1 namenode formatting
Per request to ask this as a new question, and also provide more details:
This happens on a Desktop, Windows 10, ...
1
vote
0
answers
71
views
Cannot build cube with apache kylin
I have connected Apache Kylin to our Microsoft SQL Data Warehouse, when trying to build a cube the process is failing with the below error:
19/06/25 15:35:54 INFO sqoop.Sqoop: Running Sqoop version: ...
1
vote
1
answer
875
views
Why are read operations slower than write operations in hadoop?
In cluster mode. We're running different I/O tests in Hadoop. And we noticed that writing data is faster than reading them. How to boost reading throughput to be as fast as the writing throughput. Is ...
1
vote
0
answers
56
views
Sqoop import data from mysql, job does not start
I am using using sqoop 1.4.7 with Hadoop multi node cluster.
I have a problem with sqoop import (sqoop list-tables and list-databases work fine). I have a Name Node and 2 data nodes in separate VM (...
1
vote
0
answers
281
views
How to run MapReduce jobs without HDFS?
So I'm using Hadoop with 8 nodes, all of them are configured as workers (I'm not interested in the namenode/datanodes architecture, I just need them as compute nodes that run on yarn without HDFS as I'...
1
vote
1
answer
2k
views
How to copy files from Windows to Linux HDFS directly
I am trying to copy a large amount files to Hadoop HIVE. Right now I am doing this via a two-step scripts.
Step 1: Powershell scripts copying files from Windows to Linux using
Putty scp tool.
Step ...
1
vote
0
answers
337
views
Sqoop import sql server table process shows no progress
This is my first-try using Apache Sqoop to import a SQL Server table (6 columns, 4 records) to Hive. Below is the code.
sqoop import --connect "jdbc:sqlserver://192.168.10.101:1433;database=Testdb" --...
1
vote
0
answers
4k
views
Nodemanager and Resourcemanager not showing up in jps
I installed hadoop 2.9.2 and while running start_all.sh command there is no issue creeping up. But when I execute the jps command only 4 daemons show namely Namenode, Datanode, SecondaryNamenode, ...
1
vote
0
answers
69
views
Apache Spark 2.4 is not working with yarn cluster manager?
I have installed Hadoop version 2.8.3 in my windows 10 environment and its working fine. Now when i try to install Apache Spark(version 2.4.0) with yarn as cluster its not working. When i try to ...
1
vote
0
answers
1k
views
How do I find which disk (volume) is failing?
I am using stock Apache Hadoop 1.1.1 and I can't get a datanode to start due to:
2015-04-23 09:12:48,138 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from hadoop-metrics2....
1
vote
0
answers
31
views
View all CURRENT parameters for a hadoop installation?
I have inherited a hadoop installation and I would like to query all usage parameters and configuration options. I am sifting through the conf directory and it is helping, but this is an old ...
1
vote
1
answer
561
views
Hadoop Uninstall : Shall I remove linux-image packages?
I removed hadoop from my machine via apt-get remove --purge hadoop
In the output I see the following packages were not removed :
The following packages were automatically installed and are no longer ...
1
vote
0
answers
448
views
adding nodes to hadoop cluster virtualbox
I am trying to setup a mini hadoop cluster with virtualbox:
hosts file (hosts file properly configured):
hadoop@rui-VirtualBox:/home/rui$ cat /etc/hosts | grep 10.
10.0.13.96 hnode1
10.0.13.97 ...
1
vote
0
answers
253
views
hadoop cluster high availability error couldn't find or load main class
[hduser@mn1 bin]$ ./hdfs zkfc -formatZK
Error: Could not find or load main class ”-Djava.library.path=”.home.hduser.2.3.0″.lib”
I did every step just like in this tutorial, TWICE. The error still the ...
1
vote
0
answers
1k
views
Why does eclipse report an error when I use it under a hadoop account?
I created two accounts, one is a common user, the other is just for hadoop, which is a superuser as root. I can load eclipse normally under my common user account, but I could not use it under hadoop ...
1
vote
0
answers
911
views
Cygwin ssh-host-config on windows for hadoop
I'm new to cygwin and hadoop, please anyone could explain
which mirror link is the proper to install cygwin
what version of hadoop is bug free ( .sh works good )
is it must to run below commands ...
1
vote
1
answer
2k
views
Hadoop installation on ubuntu
I installed Hadoop on ubuntu from the .deb package when I run start-all.sh I get this error
I configured the core-site.xml for hdfs:localhost:9001 but i gives me the error. I think the problem in the ...
0
votes
1
answer
9k
views
kernel parameter - net.core.netdev_max_backlog , what happened when its too low
we want to understand what are the aspects when net.core.netdev_max_backlog kernel value is very low and not as recommended
on our Linux RHEL machines , the value for this parameter is 1000
since our ...