MapReduce development under IDE

Time:2019-12-18

MapReduce development under IDE

In the Hadoop class at school, group project has to write some MapReduce. The school has given the cluster, but it’s not very used to directly open VIM to write java on the cluster. So I tossed and stepped on some pits and recorded them here.

Git deployment code

The most convenient way to think of first is to use GIT. I usually use git to manage when I write scripts like python, and then a clone repo on the server synchronizes.
But if I find this, I have to write a Makefile, manage compilation and other issues.
So I thought of using ide to debug MapReduce remotely

IntelliJ idea developing MapReduce

Eclipse has MapReduce plug-in, which is very convenient for debugging after installation. But I usually write Java is more used to IntelliJ, so I found a solution under IntelliJ.

Maven to develop MapReduce

Install maven

brew install maven

In IntelliJ, right-click the item, ‘add framework support…’

Select maven

Content of pom.xml:

?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>4480Group16</groupId>
    <artifactId>stochastic</artifactId>
    <version>0.1</version>


    <dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.6.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-jobclient</artifactId>
            <version>2.6.0</version>
        </dependency>
        <dependency>
            <groupId>commons-cli</groupId>
            <artifactId>commons-cli</artifactId>
            <version>1.2</version>
        </dependency>
    </dependencies>

    <build>
        <finalName>${project.artifactId}</finalName>
    </build>


</project>

You will be asked if you want to import. Select auto import bulabula

In edit configuration, select application, and fill in the input and output path of HDFS with program argument.

Then write code in main / Java and package jar deployment. Basically done

Recommended Today

Analysis of new data type instances of ES6 learning notes

An example of this paper describes the new data types of ES6 learning notes. To share with you for your reference, as follows: 1. Data deconstruction assignment 1. Deconstruction and assignment of arrays Basic usage: let [key1, key2…] = [value1, Value2…] Let [name, age, sex] = [‘xiaoming ‘, 24,’ male ‘]; console.log(name); console.log(age); console.log(sex); Note […]