Idea remote connection Hadoop (MacOS)

Time:2020-9-30

Windows system can be referenced directly

Because the icon position in the idea of MAC system is not the same, and the Hadoop installation location is also different, so here is to add my configuration method on Mac

The premise is that Hadoop is installed in the computer

First download and install jdk8Download jdk8 from official website
There was jdk11 on my computer, but the configuration of Hadoop for jdk9 or above is too unfriendly for me, and the remote virtual machine is installed with Hadoop 2.7, so I choose the very safe jdk8

After installation, enter idea to create a new Maven project
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)

Next, open finder and find the project directory you just created
Idea remote connection Hadoop (MacOS)
If there is no lib folder, create one manually

Then go to the Hadoop installation path and copy it to the Lib folder you just created
Since I installed with homebrew, the path is hidden by default
Open finder, press Shift + Command + G, and enter / usr / local
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)
My Hadoop path is / usr / local / cell / Hadoop / 3.2.1
And keep looking
Idea remote connection Hadoop (MacOS)
After copying the above figure to the Lib folder, open idea
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)
There is a choice to make when importing the tools folder
Idea remote connection Hadoop (MacOS)
Select jar directory
After importing the above six folders, import the Lib folder under common
Idea remote connection Hadoop (MacOS)
This is the result of the import
Idea remote connection Hadoop (MacOS)
The next step is to test whether the remote virtual machine can be connected normally

If all of them have been imported, the directory results will appear on the left
Create a new main file in the SRC main Java folder
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)

Copy the following code into the file

import org.apache.hadoop.conf.Configuration;  
import org.apache.hadoop.fs.FileStatus;  
import org.apache.hadoop.fs.FileSystem;  
import org.apache.hadoop.fs.Path;  
  
import java.net.URI;  
  
public class Main {  
        public static void main(String [] args) throws Exception {  
            System.out.println("Compile Over");  
 FileSystem fs =  FileSystem.get (New URI ("HDFS: // change to your own IP: change to your own port"), new configuration());  
  
 FileStatus[] files = fs.listStatus(new Path("/"));  
 for (FileStatus f : files) {  
                System.out.println(f);  
 }  
        }  
}

Open the projectstructer again, and follow the steps below
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)

Then you can run the code in main just now!
The output is the HDFS folder on the virtual machine
At this point, the process is successful!
Idea remote connection Hadoop (MacOS)
Idea remote connection Hadoop (MacOS)