Kettle on maxcompute User Guide

Time:2021-12-2

Introduction: kettle is an open source ETL tool, which is implemented in pure Java. It can run on windows, UNIX and Linux. It provides a graphical operation interface. It can easily define the topology of data transmission by dragging and dropping controls. Kettle supports rich data input and output sources. The database supports Oracle, mysql, DB2, etc. it also supports various open-source big data systems in the industry, such as HDFS, HBase, Cassandra, mongodb, etc. This article will introduce how to use maxcompute’s plug-in to seamlessly connect to Alibaba cloud’s big data computing platform – maxcompute.

Tools and driver versions used in this example

Kettle on maxcompute User Guide

Setup

  1. Download and install kettle
  2. Download maxcompute JDBC Driver
  3. Place the maxcompute jdbc driver in the Lib subdirectory (data integration / LIB) under the kettle installation directory
  4. Start spoon

Job

We can organize and execute tasks in maxcompute through kettle + maxcompute jdbc driver.

First, you need to do the following:

  1. New job
  2. New database connection

The format of JDBC connection string is JDBC: ODPs: < maxcompute_ endpoint>? project=<maxcompute_ project_ name>
Jdbc driver class: com.aliyun.odps.jdbc.odpsdriver
Username is alicloud accessKey ID
The password is Alibaba cloud accessKey secret
For more JDBC configurations, see:https://help.aliyun.com/docum…

Kettle on maxcompute User Guide

After that, you can access maxcompute through the SQL node according to business needs. Let’s take a simple ETL process as an example:

Kettle on maxcompute User Guide

The create table node is configured as follows:

Note:

Here, the connection needs to be configured
Do not check send SQL as single statement

The load from OSS node is configured as follows:
Kettle on maxcompute User Guide

The points needing attention are the same as the create table node. For more usage of load, see:https://help.aliyun.com/docum…

The processing node is configured as follows:

Kettle on maxcompute User Guide

The points needing attention are the same as the create table node.

Original link
This article is the original content of Alibaba cloud and cannot be reproduced without permission.