The Hadoop command line interface runs its own classes


Transferred from:
Because I also encountered this problem, so I reprinted it directly…

Recently, when learning Hadoop, you can use the command line to manage Hadoop simply. For example, you can use:

View the HDFS root node of Hadoop

hadoop fs -ls /

Create folder lgG

hadoop fs mkdir /user/root/LGG
Operate through the command line interface.
Of course, the Hadoop command line interface also supports the parameters of classes written by itself. For example, an example in the Hadoop authoritative guide is as follows:

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;

 * @author LGG
public class URLCat {
        URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
    public static void main(String[] args) throws Exception{
        InputStream in = null;
            in = new URL(args[0]).openStream();
            IOUtils.copyBytes(in, System.out, 4096,false);
take After compiling, the class file is generated, and then this class can be used as a parameter of Hadoop command line interface, as follows:

hadoop URLCat hdfs://localhost/user/tom/readme.txt
The function is to store the data in the HDFS file readme.txt Output to System.out Standard output is the standard output. However, the following errors will be reported during execution:

Exception in thread "main" java.lang.NoClassDefFoundError: URLCat
Caused by: java.lang.ClassNotFoundException: URLCat
    at Method)
    at java.lang.ClassLoader.loadClass(
    at sun.misc.Launcher$AppClassLoader.loadClass(
    at java.lang.ClassLoader.loadClass(
Could not find the main class: URLCat. Program will exit.

This exception is a typical exception of class loading problem. The exception indicates that the definition of urlcat is not found during class loading. So LGG looked up the relevant information and found that when the Hadoop script is executed, the search path of classpath will be in the current system variable $Hadoop by default_ Classpath to find, so echo view, found that it was empty!
The problem has been found. Because the classpath is specified by the system, you can go back to this environment variable to find it, but you can’t find the class urlcat in the current path. You can solve the problem by using the export command

. represents the current directory, that is, the directory of the class. Only in this way can Hadoop find this class


Then execute:
hadoop URLCat hdfs://master:9000/user/grid/test/test1.txt
It’s done~~