Hadoop – operation and maintenance command

Time:2020-10-16

HDFS dfsadmin – Report / / print info and working status

HDFS dfsadmin – safemode leave / / exit safe mode

HDFS DFS – put file / / transfer file from host to HDFS

HDFS DFS – LS / / list HDFS / directory

hadoop distcp -i hftp://192.168.40.181 :50070/data/hbase/tmp/hbase/data/default/t1 hdfs://192.168.40.116 : 8020 / HBase / data / default / T1 / / distcp data transmission

HDFS DFS – RM – R / HBase / data / default / T1 / / delete

HDFS DFS – chown – R HBase / HBase / data / default / T3 / / modify permissions recursively

HDFS dfsadmin – refreshnodes / / refresh namenode

HDFS zkfc – formatzk / / format zkfc

HDFS namenode – format / / format namenode

start- dfs.sh / / start DFS, namenode, journal node, datanode and zkfc

start- yarn.sh / / start yarn, nodemanager and ResourceManager

hadoop- daemon.sh Start journalnode / / start journalnode, or start namenode and datanode separately

yarn- daemon.sh Start ResourceManager / / start ResourceManager or nodemanager separately

start-secure- dns.sh / / start datanode separately
./sbin/yarn- daemon.sh Start nodemanager / / start nodemanager separately

HDFS DFS – MKDIR / put / / create the put directory under the root

Command line operation:

-help

Function: output the command parameter manual

-ls

Function: display directory information

Example: Hadoop FS – LS hdfs://hadoop-server01 :9000/

Note: in these parameters, all HDFS paths can be abbreviated

–>Hadoop FS – LS / equivalent to the effect of the previous command

-mkdir

Function: create directory on HDFS

Example: Hadoop FS – MKDIR – P / AAA / BBB / cc / DD

-moveFromLocal

Function: cut and paste from local to HDFS

Example: Hadoop FS – movefromlocal / home / Hadoop / a.txt / AAA / BBB / cc / DD

-moveToLocal

Function: cut and paste from HDFS to local

Example: Hadoop FS – movetolocal / AAA / BBB / cc / DD / home / Hadoop / a.txt

–appendToFile

Function: append a file to the end of an existing file

Example: Hadoop FS – appendtofile/ hello.txt hdfs://hadoop-server01 :9000/ hello.txt

It can be abbreviated as:

Hadoop  fs  -appendToFile  ./hello.txt  /hello.txt

-cat

Function: display file content

Example: Hadoop FS – Cat/ hello.txt

-tail

Function: display the end of a file

Example: Hadoop FS – tail / weblog / access_ log.1

-text

Function: print the content of a file in character form

Example: Hadoop FS – text / weblog / access_ log.1

-chgrp

-chmod

-chown

Function: the usage of Linux file system is the same as that in Linux file system

Example:

hadoop  fs  -chmod  666  /hello.txt

hadoop  fs  -chown  someuser:somegrp   /hello.txt

-copyFromLocal

Function: copy files from local file system to HDFS path

Example: Hadoop FS – copyfromlocal/ jdk.tar.gz  /aaa/

-copyToLocal

Function: copy from HDFS to local

Example: Hadoop FS – copytolocal / AAA/ jdk.tar.gz

-cp

Function: copy another path of HDFS from one path of HDFS

Example: Hadoop FS – CP / AAA/ jdk.tar.gz  /bbb/ jdk.tar.gz .2

-mv

Function: move files in HDFS directory

Example: Hadoop FS – MV / AAA/ jdk.tar.gz /

-get

Function: equivalent to copytolocal, which downloads files from HDFS to local

Example: Hadoop FS – get / AAA/ jdk.tar.gz

Function: merge and download multiple files

Example: there are more than one file under the directory / AAA / of getmerge, such as HDFS: log. 1, log. 2, log. 3

hadoop fs -getmerge /aaa/log.* ./log.sum

-put

Function: equivalent to copyfromlocal

Example: Hadoop FS – put / AAA/ jdk.tar.gz  /bbb/ jdk.tar.gz .2

-rm

Function: delete file or folder

Example: Hadoop FS – RM – R / AAA / BBB/

-rmdir

Function: delete empty directory

Example: Hadoop FS – rmdir / AAA / BBB / CCC

-df

Function: statistics the free space information of file system

Example: Hadoop FS – DF – H/

-du

Function: Statistics folder size information

Example:

hadoop  fs  -du  -s  -h /aaa/*

-count

Function: count the number of file nodes in a specified directory

Example: Hadoop FS – count / AAA/

-setrep

Function: set the number of copies of files in HDFS

Example: Hadoop FS – setrep 3 / AAA/ jdk.tar.gz

Supplement: command to view the working status of DFS cluster

HDFS dfsadmin – reporthdfs dfsadmin – Report / / print info and working status

HDFS dfsadmin – safemode leave / / exit safe mode

HDFS DFS – put file / / transfer file from host to HDFS

HDFS DFS – LS / / list HDFS / directory

hadoop distcp -i hftp://192.168.40.181 :50070/data/hbase/tmp/hbase/data/default/t1 hdfs://192.168.40.116 : 8020 / HBase / data / default / T1 / / distcp data transmission

HDFS DFS – RM – R / HBase / data / default / T1 / / delete

HDFS DFS – chown – R HBase / HBase / data / default / T3 / / modify permissions recursively

HDFS dfsadmin – refreshnodes / / refresh namenode

HDFS zkfc – formatzk / / format zkfc

HDFS namenode – format / / format namenode

start- dfs.sh / / start DFS, namenode, journal node, datanode and zkfc

start- yarn.sh / / start yarn, nodemanager and ResourceManager

hadoop- daemon.sh Start journalnode / / start journalnode, or start namenode and datanode separately

yarn- daemon.sh Start ResourceManager / / start ResourceManager or nodemanager separately

start-secure- dns.sh / / start datanode separately
./sbin/yarn- daemon.sh Start nodemanager / / start nodemanager separately

HDFS DFS – MKDIR / put / / create the put directory under the root

Command line operation:

-help

Function: output the command parameter manual

-ls

Function: display directory information

Example: Hadoop FS – LS hdfs://hadoop-server01 :9000/

Note: in these parameters, all HDFS paths can be abbreviated

–>Hadoop FS – LS / equivalent to the effect of the previous command

-mkdir

Function: create directory on HDFS

Example: Hadoop FS – MKDIR – P / AAA / BBB / cc / DD

-moveFromLocal

Function: cut and paste from local to HDFS

Example: Hadoop FS – movefromlocal / home / Hadoop / a.txt / AAA / BBB / cc / DD

-moveToLocal

Function: cut and paste from HDFS to local

Example: Hadoop FS – movetolocal / AAA / BBB / cc / DD / home / Hadoop / a.txt

–appendToFile

Function: append a file to the end of an existing file

Example: Hadoop FS – appendtofile/ hello.txt hdfs://hadoop-server01 :9000/ hello.txt

It can be abbreviated as:

Hadoop  fs  -appendToFile  ./hello.txt  /hello.txt

-cat

Function: display file content

Example: Hadoop FS – Cat/ hello.txt

-tail

Function: display the end of a file

Example: Hadoop FS – tail / weblog / access_ log.1

-text

Function: print the content of a file in character form

Example: Hadoop FS – text / weblog / access_ log.1

-chgrp

-chmod

-chown

Function: the usage of Linux file system is the same as that in Linux file system

Example:

hadoop  fs  -chmod  666  /hello.txt

hadoop  fs  -chown  someuser:somegrp   /hello.txt

-copyFromLocal

Function: copy files from local file system to HDFS path

Example: Hadoop FS – copyfromlocal/ jdk.tar.gz  /aaa/

-copyToLocal

Function: copy from HDFS to local

Example: Hadoop FS – copytolocal / AAA/ jdk.tar.gz

-cp

Function: copy another path of HDFS from one path of HDFS

Example: Hadoop FS – CP / AAA/ jdk.tar.gz  /bbb/ jdk.tar.gz .2

-mv

Function: move files in HDFS directory

Example: Hadoop FS – MV / AAA/ jdk.tar.gz /

-get

Function: equivalent to copytolocal, which downloads files from HDFS to local

Example: Hadoop FS – get / AAA/ jdk.tar.gz

Function: merge and download multiple files

Example: there are more than one file under the directory / AAA / of getmerge, such as HDFS: log. 1, log. 2, log. 3

hadoop fs -getmerge /aaa/log.* ./log.sum

-put

Function: equivalent to copyfromlocal

Example: Hadoop FS – put / AAA/ jdk.tar.gz  /bbb/ jdk.tar.gz .2

-rm

Function: delete file or folder

Example: Hadoop FS – RM – R / AAA / BBB/

-rmdir

Function: delete empty directory

Example: Hadoop FS – rmdir / AAA / BBB / CCC

-df

Function: statistics the free space information of file system

Example: Hadoop FS – DF – H/

-du

Function: Statistics folder size information

Example:

hadoop  fs  -du  -s  -h /aaa/*

-count

Function: count the number of file nodes in a specified directory

Example: Hadoop FS – count / AAA/

-setrep

Function: set the number of copies of files in HDFS

Example: Hadoop FS – setrep 3 / AAA/ jdk.tar.gz

Supplement: command to view the working status of DFS cluster

hdfs dfsadmin -report