Hand in hand teaching you to build hive environment

Time:2021-6-7

Preparation environment: Hadoop basic environment (3.1.3)   Hive version 3.1.2

Download address:https://hive.apache.org/downloads.html

Hive3. X supports Hadoop 3. X, hive2. 0 supports Hadoop 2. X

1. Upload the installation package and unzip the file

tar -zxvf     Corresponding package name

2. Create a soft connection

ln -s  apache-hive-1.2.2-bin hive

3. Modify environment variables

vim ~/.bash_profile

Hand in hand teaching you to build hive environment

After modification, the environment variable will take effect immediately

source ~/.bash_profile

4. Modify profile name

Hand in hand teaching you to build hive environment

mv hive-env.sh.template hive-env.sh

5. Enter the file and configure Hadoop path and hive_ Conf path

Hand in hand teaching you to build hive environment

6. Create relevant directory on HDFS and attach relevant permissions

hdfs dfs -mkdir -p  /user/hive/warehouse

hdfs dfs -mkdir /tmp

hdfs dfs -chmod g+w /tmp  /user/hive/warehouse

7. Start hive (HDFS and yarn need to be started before this)

Hand in hand teaching you to build hive environment

Here we need to pay attention

1) If the jar package in Hadoop lib conflicts with the jar package in hive, the startup will fail: see the link for related problems

http://www.bubuko.com/infodetail-3286965.html

2) When the view database HQL statement is executed, the initialization of the database is not executed at the beginning, which leads to the start failure. The default database for hive to store metadata is Derby.

https://blog.csdn.net/lz6363/article/details/95805446

Hand in hand teaching you to build hive environment

Initialization related instructions

schematool -dbType derby -initSchema

So far, the preliminary hive environment has been built.

8. Change hive metadata database to MySQL. First, this paper introduces the installation steps of MySQL database

1) First, download and install the official MySQL Yum repository

wget -i -c http://dev.mysql.com/get/mysql57-community-release-el7-10.noarch.rpm

Hand in hand teaching you to build hive environment

2) Install MySQL server

yum -y install mysql57-community-release-el7-10.noarch.rpm

Hand in hand teaching you to build hive environment

3) Start MySQL and view the status of MySQL

systemctl start  mysqld.service

systemctl status mysqld.service

Hand in hand teaching you to build hive environment

Display active, start successfully.

4) View initial password

grep “password” /var/log/mysqld.log

Hand in hand teaching you to build hive environment

5) Log in to MySQL database and change the password

mysql -uroot -p

It should be noted that if the password setting is too simple, an error will be reported

Hand in hand teaching you to build hive environment

The reason is that MySQL has a password setting specification, specifically with validate_ password_ The value of policy is related to:

Hand in hand teaching you to build hive environment

Change the password rules

set global validate_password_policy=0;

set global validate_password_length=1;

At this point, you can change the password

ALTER USER ‘root’@’localhost’ IDENTIFIED BY ‘new password’;

6) View database

show databases;

Hand in hand teaching you to build hive environment

9. Create the file hive-site.xml in hive / conf directory

<configuration> <property> <name>javax.jdo.option.ConnectionURL</name> <value>jdbc:mysql://172.16.25.27:3306/metastore?createDatabaseIfNotExist=true</value> <description>JDBC connect string for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionDriverName</name> <value>com.mysql.jdbc.Driver</value> <description>Driver class name for a JDBC metastore</description> </property> <property> <name>javax.jdo.option.ConnectionUserName</name> <value>root</value> <description>username to use against metastore database</description> </property> <property> <name>javax.jdo.option.ConnectionPassword</name> <value>root</value> <description>password to use against metastore database</description> </property> </configuration>

10. Download the MySQL driver package and import the dependent jar package into the / hive / lib folder

Download address:https://dev.mysql.com/downloads/file/?id=496585

MySQL 5.7 can be driven by version 8.0 or version 5.1

Hand in hand teaching you to build hive environment

After configuration, start hive and finish.