To solve the problem of starting Azkaban error reporting: java.lang.NoSuchMethodError : com.google.common . collect.ImmutableMap.toImmutableMap

Time:2020-6-28

Problem Description:

Start Azkaban error:
java.lang.NoSuchMethodError:com.google.common.collect.ImmutableMap.toImmutableMap

resolvent:

From the error information, we can’t find the toimmutable map method. First, find the jar package corresponding to immutable map as guava, and then find the jar package on the server:

find / -name “guava*.jar”

It is found that in addition to the Azkaban installation directory, there are guava packages in other program directories. The version of the guava package in Azkaban is guava-21.0.jar, and the versions of other programs are 11, 14, 17, 18, 19, 21, 25. It is suspected that there are multiple guava packages, but the lower version package is used, and there is no toimmutable map method in it.

Next, let’s see what the classpath looks like when Azkaban is running. View web server startup filestart-web.sh

more /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/bin/start-web.sh

The content is:


#!/bin/bash

script_dir=$(dirname $0)

${script_dir}/internal/internal-start-web.sh >webServerLog_`date +%F+%T`.out 2>&1 &

Internal start is called- web.sh , then look at:

more /u01/app/azkaban-3.50.0/azkaban-web-server-0.1.0-SNAPSHOT/bin/internal/internal-start-web.sh

The content is:


#!/bin/bash

azkaban_dir=$(dirname $0)/../..

# Specifies location of azkaban.properties, log4j.properties files
# Change if necessary
conf=$azkaban_dir/conf

if [[ -z "$tmpdir" ]]; then
tmpdir=/tmp
fi

for file in $azkaban_dir/lib/*.jar;
do
 CLASSPATH=$CLASSPATH:$file
done

for file in $azkaban_dir/extlib/*.jar;
do
 CLASSPATH=$CLASSPATH:$file
done

for file in $azkaban_dir/plugins/*/*.jar;
do
 CLASSPATH=$CLASSPATH:$file
done

if [ "$HADOOP_HOME" != "" ]; then
    echo "Using Hadoop from $HADOOP_HOME"
    CLASSPATH=$CLASSPATH:$HADOOP_HOME/conf:$HADOOP_HOME/*
    JAVA_LIB_PATH="-Djava.library.path=$HADOOP_HOME/lib/native/Linux-amd64-64"
else
    echo "Error: HADOOP_HOME is not set. Hadoop job types will not run properly."
fi

if [ "$HIVE_HOME" != "" ]; then
    echo "Using Hive from $HIVE_HOME"
    CLASSPATH=$CLASSPATH:$HIVE_HOME/conf:$HIVE_HOME/lib/*
fi

echo $azkaban_dir;
echo $CLASSPATH;

executorport=`cat $conf/azkaban.properties | grep executor.port | cut -d = -f 2`
serverpath=`pwd`

if [[ -z "$AZKABAN_OPTS" ]]; then
 AZKABAN_OPTS="-Xmx4G"
fi
# Set the log4j configuration file
if [ -f $conf/log4j.properties ]; then
 AZKABAN_OPTS="$AZKABAN_OPTS -Dlog4j.configuration=file:$conf/log4j.properties -Dlog4j.log.dir=$azkaban_dir/logs"
else
 echo "Exit with error: $conf/log4j.properties file doesn't exist."
 exit 1;
fi
AZKABAN_OPTS="$AZKABAN_OPTS -server -Dcom.sun.management.jmxremote -Djava.io.tmpdir=$tmpdir -Dexecutorport=$executorport -Dserverpat
h=$serverpath"

java $AZKABAN_OPTS $JAVA_LIB_PATH -cp $CLASSPATH azkaban.webapp.AzkabanWebServer -conf $conf [email protected] &

echo $! > $azkaban_dir/currentpid

As you can see, the jar package of the Azkaban installation directory is after the original $classpath, as follows:


for file in $azkaban_dir/lib/*.jar;
do
 CLASSPATH=$CLASSPATH:$file
done

In this way, guava packages in other directories will be found and used first. Well, when the cause of the problem is found, the solution is very simpleCLASSPATH=$CLASSPATH:$fileChange toCLASSPATH=$file:$CLASSPATH, let the guava package under the Azkaban installation directory be found and used first. In addition, the startup file of executor server / u01 / APP / azkaban-3.50.0/azkaban-exec-server-0.1.0-snapshot / bin / internal / internal start- executor.sh The same changes need to be made.

After the modification, it is normal to start the service again. (both executor server and web server need to be restarted)

complete.

Here is an error report about starting Azkaban: java.lang.NoSuchMethodError : com.google.common . collect.ImmutableMap.toImmutableMap This is the introduction of the article of. For more information about launching Azkaban, please search the previous articles of developpaer or continue to browse the related articles below. I hope you can support developpaer in the future!