EBay’s practice of building a unified serverless spark gateway based on Apache Kyuubi


This article is shared by Wang Fei, a software engineer of eBay and a PPMC member of Apache Kyuubi, in the joint meetup of Apache seatunnel & Kyuubi. It introduces the basic architecture and usage scenarios of Apache Kyuubi (incubating), the enhancements made by eBay to Kyuubi based on its own needs, and how to build a unified & serverless spark gateway based on Kyuubi.

What is Kyuubi

First, let’s introduce Kyuubi. Kyuubi is a distributed thrift JDBC / ODBC server, which supports multi tenant and distributed features, and can meet the applications of big data scenarios such as ETL and Bi reports in enterprises. The project was initiated by Netease Shufan and has been incubated by Apache foundation. At present, the main direction is to build a serverless SQL on Lakehouse service around various mainstream computing engines based on its own architecture design. At present, the supported engines include spark, Flink and Trino (that is, PRESTO). My theme today is around Kyuubi and spark. I won’t expand on other computing engines here.

For spark, Kyuubi has the API of hiveserver2, supports spark multi tenancy, and then runs in serverless mode. Hiveserver2 is a classic JDBC service. Spark community also has a similar service called spark thrift server. Here is a comparison between spark thrift server and Kyuubi.

Spark thrift server can be understood as an independent spark app, which is responsible for receiving users’ SQL requests. SQL compilation and execution will run in this app. When the scale of users reaches a certain level, there may be a single point bottleneck.

For Kyuubi, we can see the figure on the right. There is a red user and a blue user. They have a red spark app and a blue spark app respectively. After their SQL requests come in, the SQL is compiled and executed on the corresponding app, that is to say, Kyuubi server only transfers the SQL requests once and sends the SQL directly to the spark app behind them.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

For spark thrift server, it needs to save results and status information. It is stateful, so it cannot support HA and lb. Kyuubi does not save the results and is almost a stateless service. Therefore, Kyuubi supports HA and lb. we can increase the number of Kyuubi servers to meet the needs of enterprises. So Kyuubi is a better spark SQL gateway.

Kyuubi’s architecture is divided into two layers, one is the server layer and the other is the engine layer. Both the server layer and the engine layer have a service discovery layer. The service discovery layer of the Kyuubi server layer is used to randomly select a Kyuubi server, which is shared by all users. The service discovery layer of the Kyuubi engine layer is invisible to users. It is used by the Kyuubi server to select the spark engine of the corresponding user. When a user’s request comes in, it will randomly select a Kyuubi server. The Kyuubi server will select an engine from the service discovery layer of the engine. If the engine does not exist, it will create a spark engine, After the engine is started, it will register with the service discovery layer of the engine, and then an internal connection will be made between Kyuubi server and engine. Therefore, Kyuubi server is shared by all users, and Kyuubi engine is resource isolation between users.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

Kyuubi supports the sharing level of some engines, which is based on the balance between isolation and resources. On eBay, we mainly use the user and connection levels. Firstly, for the connection level, a new app will be created for each user’s connection, that is, a Kyuubi engine, which is suitable for ETL scenarios. ETL has a high workload and needs an independent app to execute; For the user level, we can see that there are two users, one is Tom and the other is Jerry. Two clients of Tom connect to the Kyuubi server and will connect to the same Kyuubi engine belonging to Tom. The user level is applicable to the ad-hoc scenario, that is, for the same user, all connections will be executed to the same Kyuubi engine, and all requests for Jerry will be executed to Jerry’s Kyuubi engine.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

The user sharing level Kyuubi has been strengthened by introducing the concept of an engine pool. Just like the thread pool in programming, we can create an engine pool with a number. For example, Tom created two pools called pool-a and pool-b, numbered pool-a-0 and pool-a-1. If the name of the pool is directly specified when the client requests it, Kyuubi server will randomly select an engine from this pool to execute; If Tom specifies not only the name of the pool, but also the index of the engine in the pool. For example, if pool-b-0 is specified here, Kyuubi server will select the engine with number 0 from the pool-b for calculation. The corresponding parameter iskyuubi.engine.share.level.subdomain.

This provides great convenience for Bi tool integration on eBay. On eBay, each analyst team may use the same account to perform data analysis. Bi tool will create a Kyuubi engine according to the user’s IP, because the parameter configuration required by each analyst may be different. For example, their memory configuration is different. Bi tool can create such an engine pool, Then save the user’s IP and a mapping of the created engine index. When the user’s request comes, find the engine created by the user according to the IP mapping relationship saved by the Bi tool. In other words, many people in a team use a pool to pre create many spark apps, so that people in this group can randomly select an engine for execution, which can increase the degree of concurrency. At the same time, it can also be used as a label under the user sharing level to mark the purpose of the engine. For example, we can create different engine pools under the user sharing level for beeline scenarios and Java JDBC application usage scenarios, and use different engine pools under different usage scenarios to isolate each other.

As mentioned earlier, there are different engine sharing levels. Some are to create a spark app for each connection, and some are to create one or more Kyuubi engines for a user. You may be worried about resource waste. Here is the dynamic management of resources by Kyuubi engine. First, a Kyuubi engine, that is, a spark app, is divided into spark driver and spark executor. For the executor, spark itself has an executor dynamic allocation mechanism, which will decide whether to apply for more resources from the cluster or return the currently applied resources to the cluster according to the load of the current spark job. So we put some restrictions on the Kyuubi server layer, such as forcing the dynamic allocation of the executor to be turned on, and setting the minimum number of executors to 0 when idle, that is to say, when an app is very idle, only one driver runs with it, so as to avoid wasting resources. In addition to the dynamic recycling mechanism of the executor layer, Kyuubi also adds a resource recycling mechanism to the driver layer. For the connection sharing level, Kyuubi engine can only be used in the current connection. When the connection is closed, spark driver will be recycled directly. Kyuubi has a parameter for user level sharingkyuubi.session.engine.idle.timeoutTo control the maximum idle time of engine. For example, we set the idle time to 12 hours. If there is no request to connect to the spark app within 12 hours, the spark app will end automatically to avoid resource waste.

Usage scenario of Kyuubi

Let’s talk about use case. At present, Kyuubi supports the execution of SQL language and Scala language, and can also write SQL and scala together to run. Because SQL is a very user-friendly language, it allows you to use simple SQL statements to query data without understanding the internal principles of spark, but it also has some limitations; Scala language needs a certain threshold, but it is very flexible. We can write code or manipulate some spark dataframe APIs.

For example, you can mix programming in an SQL file or a notebook. First, create a training data table with SQL statement. After creating the table, set the language mode to scala through set statement, and then start writing code in scala. Here, use a kmeans to process the training data. After processing, save the output in a table, and then switch the language mode to SQL to continue processing with SQL. This is very convenient. We can combine the advantages of SQL and scala to basically solve most of the cases in data analysis. We also provide a very friendly interface in Kyuubi JDBC, which can be called directlyKyuubiStatement::ExecuteScalaTo execute Scala statements.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

Kyuubi’s practice on eBay

EBay demand background

Our Hadoop team manages many Hadoop clusters. These clusters are distributed in different data centers and have different purposes. There is a unified set of permission verification based on KDC and LDAP.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

When we first introduced Kyuubi, did we want to deploy a Kyuubi service for each cluster? If so, we may need to deploy three or four Kyuubi services, which need to be repeated during the upgrade, which is very inconvenient to manage, so we wonder whether we can use a set of Kyuubi to serve multiple Hadoop clusters.

EBay’s enhancements to Kyuubi

The following figure shows some enhancements we have made to this requirement. First of all, because we support KDC and LDAP authentication, we let Kyuubi support Kerberos and plain type permission authentication at the same time, and optimize the startup of Kyuubi engine and Kyuubi beeline. Then we extended some of Kyuubi’s thrift APIs to support uploading and downloading data. As mentioned earlier, to access multiple Hadoop clusters with one Kyuubi, we added the concept of cluster selector, which can specify parameters during connection to route requests to the corresponding cluster. In addition, we are also improving the restfull API, and have supported SPNEGO and basic restfull API permission authentication for Kyuubi. In addition, we are also doing some work of restfull API to run SQL query and batch job. Numbered in the figure are some PR that have been fed back to the community.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

Let’s talk about 2, 3 and 4, some optimizations of Kyuubi’s thrift RPC. First, because thrift RPC itself is designed for hiveserver2, it is very fast to establish a connection in hiveserver2 / spark thriftserver2. To establish a connection in Kyuubi, you must first connect to the Kyuubi server. The Kyuubi server cannot return the results to the client until the connection with the remote Kyuubi engine is completed.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

If the Kyuubi engine does not exist at first, and there is a resource problem when starting the Kyuubi engine, or some parameters are set incorrectly, for example, it sets invalid parametersspark.yarn.queu, if an error occurs, there may be a delay of one minute or tens of seconds. The client has to wait all the time, and no log is returned to the client during the waiting process. We have made some asynchronous opensessions for this. Opensession is divided into two parts. The first step is to connect to Kyuubi server. Kyuubi server starts a launchengine operation asynchronously, and then immediately connects Kyuubi server to the client, so that the client can connect to Kyuubi server in one second. However, his next statement and after entering will not start running until the engine is initialized. In fact, this is also a requirement of our PM. Even if the first statement runs for a long time, it doesn’t matter, but the connection must be fast, so we did this work.

Because hive thrift RPC is a widely used and user-friendly RPC, we have made some extensions based on thrift RPC without damaging its compatibility. Firstly, for the request and return result of executestatement, it will return an operationhandle in the API, and then obtain the status and log of the current operation according to the operationhandle. Because we have previously disassembled opensession into opensession plus a launchengine operation, we want to return some information of launchengine operation through the configuration map of opensession request. We divide an operationhandler into two parts, one is guid and the other is secret, which are put into the configuration map of opensessionresp.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

Then, after getting opensessionresp, you can spell out the operationhandler corresponding to the launch engine operation according to the configuration, and then get the log and status of the launch engine according to it.

The following is an effect. When establishing a Kyuubi connection, we can know in real time what happened in the process of spark submit. For example, users willspark.yarn.queueIf the settings are wrong, or if you have been waiting for resources, you can clearly know what happened in the process. You don’t need to find platform maintenance personnel to read the log, which not only makes users feel very friendly, but also reduces the efforts of platform maintenance personnel.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

Building a unified & serverless spark gateway

As mentioned earlier, we need to use one Kyuubi service to serve multiple clusters, so we built a unified & serverless spark gateway based on Kyuubi. Unified means that we only have one endpoint. We are deployed on kubernetes. We use kubernetes LB as the service discovery of Kyuubi server. The form of endpoint is a kubernetes LB plus a port, for examplekyuubi.k8s-lb.ebay.com:10009, to serve multiple clusters, we only need to add a parameter to the JDBC URLkyuubi.session.cluster, specify cluster name, and his request can be executed in the specified cluster. We also use unified for permission verification, and support Kerberos and LDAP permission verification at the same time. The functions are also unified and support the submission of spark SQL, spark Scala and ETL spark jobs.

As for serverless, Kyuubi server is deployed on kubernetes and is cloud native. Moreover, Kyuubi server supports HA and LB, and Kyuubi engine supports multi tenancy, so the cost for platform maintenance personnel is very low.

This is our general deployment. For multiple clusters, we have introduced the concept of cluster selector. Each cluster is equipped with a kubernetes configmap file. In this file, there are some unique configurations of the cluster, such as the configuration of the zookeeper of the cluster and the environment variables of the cluster, which will be injected into the startup process when starting Kyuubi engine.

EBay's practice of building a unified serverless spark gateway based on Apache Kyuubi

The super user configuration of each cluster is also different, so we also support super user verification for each cluster. At present, Kyuubi supports the refresh of Hadoop fsdelegation token and hivedelegation token, which can make spark engine run for a long time without KeyTab, without worrying about the expiration of token. We also make this function support multiple clusters.

The process of a user’s request is as follows: first, he needs to specify a cluster selector. Kyuubi server (on kubernetes) will find the corresponding cluster according to this selector, connect the zookeeper of the cluster, and then find out whether there is a corresponding spark app in the zookeeper. If not, submit an app to yarn (spark on yarn), and the engine will register with zookeeper after startup, Kyuubi server and Kyuubi engine find the host and port through zookeeper and create a connection.

Kyuubi has supported thrift / JDBC / ODBC API from the beginning. At present, the community is also improving restful API EBay is also doing some work to improve the restful API. We have added permission verification support to the restful API, and support SPNEGO (Kerberos) and basic (password based) permission verification at the same time. We plan to add more restful APIs to Kyuubi. At present, there is an API for sessions. For example, you can turn off sessions, which is mainly used to manage Kyuubi’s sessions. We are going to add some APIs for SQL and batch job. About SQL, you can submit an SQL query directly through the restful API, and then you can get its results and log. About batch job, you can submit a common spark app running with jar through restful API, and then get the applicationid of spark app and spark submit log, which makes it easier for users to use Kyuubi to complete various common spark operations.

EBay revenue

For users, they can easily use spark services. They can use thrift, JDBC, ODBC and restful interfaces. It is also very lightweight. They don’t need to install Hadoop / spark binary or manage the conf of Hadoop and spark. They just need to connect in the form of restful or beeline / JDBC.

For our platform development team, we have a centralized spark service, which can provide SQL, Scala services, and spark submit services. We can easily manage the spark version without distributing the spark installation package to users. We can better complete the grayscale upgrade, make the spark version transparent to users, make the whole cluster use the latest spark, and save cluster resources, Save the cost of the company. In addition, the maintenance of the platform is also relatively convenient and the cost is very low, because we only need to maintain one Kyuubi service to serve multiple Hadoop clusters.

Author: Wang Fei, eBay software engineer, Apache Kyuubi PPMC member

Attached video playback and PPT download:

Apache Kyuubi’s practice on eBay – Wang Fei

Extended reading:

Depth of travel in Apache T3

Who is using Apache Kyuubi (Incubating)?

Kyuubi project home page

Kyuubi code warehouse