Hive 3.1.2를 설치해보자! + HADOOP 3.2.1 연동

Hive 다운로드

wget archive.apache.org/dist/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz

$ wget archive.apache.org/dist/hive/hive-3.1.2/apache-hive-3.1.2-bin.tar.gz
$ tar -zxvf apache-hive-3.1.2-bin.tar.gz
$ cp -r apache-hive-3.1.2-bin /usr/local/hive

 

Hive 환경변수 설정

$ cat ~/.bashrc | grep HIVE
export HIVE_HOME=/usr/local/hive
export HIVE_CONF_DIR=/usr/local/hive/conf
export PATH=$PATH:$JAVA_HOME/bin:$HADOOP_HOME/sbin:$SPARK_HOME/bin:$SQOOP_HOME/bin:$HIVE_HOME/bin

 

Hive 설정

//hive디렉토리 권한 설정
$ chown -R hadoopuser:hadoopuser hive

//hive-env.sh 설정
$ cd $HIVE_HOME/conf                 
$ cp hive-env.sh.template hive-env.sh
$ cat hive-env.sh | grep HADOOP_HOME
# Set HADOOP_HOME to point to a specific hadoop install directory
 HADOOP_HOME=/usr/local/hadoop

//hive-site.xml 설정(hive-site.xml 파일 생성)
$ cat hive-site.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
<property>
   <name>hive.metastore.local</name>
   <value>false</value>
</property>
<property>
   <name>hive.metastore.warehouse.dir</name>
   <value>hdfs://하둡마스터호스트:9000/user/hive</value>
</property>

 

Hive를 위한 HDFS 디렉토리 생성

$ hadoop fs -mkdir /user/hive
$ hadoop fs -ls /user/
Found 3 items
drwxr-xr-x   - hadoopuser supergroup          0 2020-12-09 22:44 /user/hadoopuser
drwxr-xr-x   - hadoopuser supergroup          0 2020-12-13 23:57 /user/hive
drwxr-xr-x   - hadoopuser supergroup          0 2020-12-09 23:03 /user/songhee

 

Hive실행

메타스토어 초기화 중 에러 발생 

이 경우, hive/lib/guava-19.0.jar를 삭제하고, hadoop lib중 guava-27.0-jre.jar를 hive/lib로 복사한다.

//메타스토어 초기화
$ schematool -dbType derby -initSchema

//에러
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/usr/local/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/local/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2020-12-14 00:02:07,226 INFO  [main] conf.HiveConf (HiveConf.java:findConfigFile(187)) - Found configuration file file:/usr/local/hive/conf/hive-site.xml
Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;Ljava/lang/Object;)V
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1357)
        at org.apache.hadoop.conf.Configuration.set(Configuration.java:1338)
        at org.apache.hadoop.mapred.JobConf.setJar(JobConf.java:536)
        at org.apache.hadoop.mapred.JobConf.setJarByClass(JobConf.java:554)
        at org.apache.hadoop.mapred.JobConf.<init>(JobConf.java:448)
        at org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:5141)
        at org.apache.hadoop.hive.conf.HiveConf.<init>(HiveConf.java:5104)
        at org.apache.hive.beeline.HiveSchemaTool.<init>(HiveSchemaTool.java:96)
        at org.apache.hive.beeline.HiveSchemaTool.main(HiveSchemaTool.java:1473)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
        
$ rm $HIVE_HOME/lib/guava-19.0.jar
$ cp /usr/local/hadoop/share/hadoop/hdfs/lib/guava-27.0-jre.jar /usr/local/hive/lib/

 

재시도

$ cp $HIVE_HOME/bin
$ ./schematool -dbType derby -initSchema --verbose
schemaTool completed

$ hive

설치 완료!

 

Hive 기본 명령어

$ hive

//DB생성
hive> create database test;
hive> show databases;

//테이블 생성
hive> create table test.tbl (
       >   col1 integer,
       >   col2 string);

//insert
hive> insert into table test.tbl
    > select 1 as col1, 'song' as col2;
    
//select
hive > select * from test.tbl;

 

 

HDFS 확인

$ hadoop fs -ls /user/hive/warehouse/test.db/tbl
Found 1 items
-rw-r--r--   2 hadoopuser supergroup          7 2020-12-15 02:16 /user/hive/warehouse/test.db/tbl/000000_0

$ hadoop fs -cat /user/hive/warehouse/test.db/tbl/000000_0
1song