The Hadoop command line interface runs its own classes

Time:2021-1-19

Transferred from:http://blog.chinaunix.net/uid-11767650-id-1673086.html
Because I also encountered this problem, so I reprinted it directly…

Recently, when learning Hadoop, you can use the command line to manage Hadoop simply. For example, you can use:

View the HDFS root node of Hadoop

hadoop fs -ls /

Create folder lgG

hadoop fs mkdir /user/root/LGG
Operate through the command line interface.
Of course, the Hadoop command line interface also supports the parameters of classes written by itself. For example, an example in the Hadoop authoritative guide is as follows:

import java.io.InputStream;
import java.net.URL;
import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;
import org.apache.hadoop.io.IOUtils;

/**
 * 
 * @author LGG
 *
 */
public class URLCat {
    static{
        URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory());
    }
    public static void main(String[] args) throws Exception{
        InputStream in = null;
        try{
            in = new URL(args[0]).openStream();
            IOUtils.copyBytes(in, System.out, 4096,false);
        }finally{
            IOUtils.closeStream(in);
        }
    }
}
take URLCat.java After compiling, the class file is generated, and then this class can be used as a parameter of Hadoop command line interface, as follows:

hadoop URLCat hdfs://localhost/user/tom/readme.txt
The function is to store the data in the HDFS file readme.txt Output to System.out Standard output is the standard output. However, the following errors will be reported during execution:

Exception in thread "main" java.lang.NoClassDefFoundError: URLCat
Caused by: java.lang.ClassNotFoundException: URLCat
    at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: URLCat. Program will exit.

This exception is a typical exception of class loading problem. The exception indicates that the definition of urlcat is not found during class loading. So LGG looked up the relevant information and found that when the Hadoop script is executed, the search path of classpath will be in the current system variable $Hadoop by default_ Classpath to find, so echo view, found that it was empty!
The problem has been found. Because the classpath is specified by the system, you can go back to this environment variable to find it, but you can’t find the class urlcat in the current path. You can solve the problem by using the export command

. represents the current directory, that is, the directory of the class. Only in this way can Hadoop find this class

export HADOOP_CLASSPATH=.

Then execute:
hadoop URLCat hdfs://master:9000/user/grid/test/test1.txt
It’s done~~