[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

Time:2021-7-21

Brief introduction: take the test cluster version as an example (emr-4.4.1) — Flink SQL client integrated hive use document
Author: Lin Zhicheng, technical support of Alibaba cloud EMR product team, has many years of open source big data experience

1. Take the test cluster version as an example (emr-4.4.1)

[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

2. The execution order is as follows

cp /opt/apps/ecm/service/flink/1.10-vvr-1.0.2-hadoop3.1/package/flink-1.10-vvr-1.0.2-hadoop3.1/conf/sql-client-defaults.yaml /etc/ecm/flink-conf/

And make the following modifications
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

3. Distribute the configuration to each node

(if it needs to be used in other clusters, perform the following steps, and all steps of copying jars are performed on all machines.)

scp /etc/ecm/flink-conf/sql-client-defaults.yaml [email protected]:/etc/ecm/flink-conf/
scp /etc/ecm/flink-conf/sql-client-defaults.yaml [email protected]:/etc/ecm/flink-conf/

… (refer to the following here, because there are several jars to copy.)

4. Copy jar package

start-cluster.sh
sql-client.sh embedded
You will find the following errors:
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

The reason is the lack of jar package. Perform the following operation, otherwise there will be a series of errors.
cd /usr/lib/flink-current/lib
sudo cp /lib/hive-current/lib/hive-exec-3.1.2.jar .
sudo wget https://repo1.maven.org/maven…
sudo wget https://repo1.maven.org/maven…
sudo wget https://repo1.maven.org/maven…

5. Start up

start-cluster.sh
sql-client.sh embedded
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

The following is empty because there is no data in the new cluster. Go to hive to create point data
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

Re execute sql-client.sh embedded to enter SQL client and find that you can see it.
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

The query found that the report was wrong.
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

6. Troubleshooting

Check the next look 8081 ports are not up
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

Finally, it was found that the compatibility problem of community hive connector. The previous conflict was caused by 1.10.2. Theoretically, vvr-1.10 can use community hive-connector-1.10. X. this problem has been fixed in 1.11.
So let’s change the jar package.
mv flink-connector-hive_2.11-1.10.2.jar /tmp/
sudo wget https://repo1.maven.org/maven…

7. Reexecution

start-cluster.sh;sql-client.sh embedded
[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

[Alibaba cloud EMR actual combat] take EMR test cluster version as an example to explain the steps of using Flink SQL client to integrate hive

It’s successful as shown in the figure above.

Original link

This article is the original content of Alibaba cloud and cannot be reproduced without permission.