1

This is my first-try using Apache Sqoop to import a SQL Server table (6 columns, 4 records) to Hive. Below is the code.

sqoop import --connect "jdbc:sqlserver://192.168.10.101:1433;database=Testdb" --username abc--password abc --table "DimEmployee" --create-hive-table --hive-import --hive-table DboDimEmployee

The execution ran OK but stopped at this output

19/01/25 13:19:36 INFO mapreduce.Job: Running job: job_1548438714494_0003

I checked the Hadoop UI web page. This particular application has no resource allocated to it and the progress is 0%. I am not sure what I did wrong.

Below is additional information.

  1. The sql db connection parameters are correct and I have tested the connection from Hadoop side.
  2. The Hive works fine. I was able to create a database or table in the hive.
  3. The entire Hadoop system is in VirtualBox on my laptop. The master node has 4G memory and the data node has 1G memory.
  4. Below is the only memory-related configurations I did on Hadoop. I am not sure the issue is related to memory and I posted it just in case.

    vi mapred-site.xml
      <property>
        <name>mapreduce.framework.name</name>
              <value>yarn</value>
      </property>
      <property>
            <name>yarn.app.mapreduce.am.resource.mb</name>
            <value>256</value>
      </property>
      <property>
              <name>mapreduce.map.memory.mb</name>
              <value>128</value>
      </property>
      <property>
              <name>mapreduce.reduce.memory.mb</name>
              <value>128</value>
      </property>
    
  5. I never saw the progress of map reduce, like % of map and % of reduce.

  6. My installation did not have HBASE, HCatalog, Accumulo, or zookeeper. I do not think I need them but I might be wrong.

  7. Below is all execution messages I got from Sqoop.

Warning: /home/admin1/sqoop/../hbase does not exist! HBase imports will fail. Please set $HBASE_HOME to the root of your HBase installation. Warning: /home/admin1/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /home/admin1/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. Warning: /home/admin1/sqoop/../zookeeper does not exist! Accumulo imports will fail. Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation. 19/01/28 09:56:12 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7 19/01/28 09:56:12 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 19/01/28 09:56:12 INFO tool.BaseSqoopTool: Using Hive-specific delimiters for output. You can override 19/01/28 09:56:12 INFO tool.BaseSqoopTool: delimiters with --fields-terminated-by, etc. 19/01/28 09:56:12 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manage r). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time. 19/01/28 09:56:12 INFO manager.SqlManager: Using default fetchSize of 1000 19/01/28 09:56:12 INFO tool.CodeGenTool: Beginning code generation 19/01/28 09:56:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DimEmployee AS t WHERE 1=0 19/01/28 09:56:12 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DimEmployee AS t WHERE 1=0 19/01/28 09:56:12 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/admin1/hadoop Note: /tmp/sqoop-admin1/compile/e8e0b042e5ecc16c39484556762dae8a/DimEmployee.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. 19/01/28 09:56:17 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-admin1/compile/e8e0b042e5ecc16c39484556762dae8a/DimEmployee.jar 19/01/28 09:56:18 INFO mapreduce.ImportJobBase: Beginning import of DimEmployee 19/01/28 09:56:18 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 19/01/28 09:56:18 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM DimEmployee AS t WHERE 1=0 19/01/28 09:56:19 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 19/01/28 09:56:19 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 19/01/28 09:56:29 INFO db.DBInputFormat: Using read commited transaction isolation 19/01/28 09:56:29 INFO db.DataDrivenDBInputFormat: BoundingValsQuery: SELECT MIN(EmployeeKey), MAX(EmployeeKey) FROM DimEmployee 19/01/28 09:56:29 INFO db.IntegerSplitter: Split size: 73; Num splits: 4 from: 1 to: 296 19/01/28 09:56:29 INFO mapreduce.JobSubmitter: number of splits:4 19/01/28 09:56:29 INFO Configuration.deprecation: yarn.resourcemanager.system-metrics-publisher.enabled is deprecated. Instead, use yarn.system-metrics-publisher.enabled 19/01/28 09:56:30 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1548697949348_0001 19/01/28 09:56:31 INFO impl.YarnClientImpl: Submitted application application_1548697949348_0001 19/01/28 09:56:31 INFO mapreduce.Job: The url to track the job: http://name1:8088/proxy/application_1548697949348_0001/ 19/01/28 09:56:31 INFO mapreduce.Job: Running job: job_1548697949348_0001

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.