[Hadoop chapter 04] Hadoop configuration log aggregation

Time:2020-8-1

The way to grow up is to know how to shut up and work hard, know how to keep a low profile and be humble, and learn to be strong. In every worthy day, I try my best to be what I want to be

Hadoop configuration log aggregation

Application scenarios

In order to let the application run after the completion of the application log information uploaded to the HDFS system, after the log can view the program error information, so as to debug the program

Configuration steps

1. Configure yarn- site.xml

<! -- enable log aggregation function -- >
<property>
    <name>yarn.log-aggregation-enable</name>
    <value>true</value>
</property>

<! -- set log retention time for 7 days -- >
<property>
    <name>yarn.log-aggregation.retain-seconds</name>
    <value>604800</value>
</property>

[Hadoop chapter 04] Hadoop configuration log aggregation

2. Restart the cluster

sbin/stop-yarn.sh
sbin/stop-dfs.sh
sbin/mr-jobhistory-daemon.sh stop historyserver

sbin/start-dfs.sh
sbin/start-yarn.sh
sbin/mr-jobhistory-daemon.sh start historyserver

3. Operation procedure

Upload data to HDFS

[Hadoop chapter 04] Hadoop configuration log aggregation

Assign file permissions

[Hadoop chapter 04] Hadoop configuration log aggregation

Run the program

[Hadoop chapter 04] Hadoop configuration log aggregation

View run results

[Hadoop chapter 04] Hadoop configuration log aggregation

4. View log

[Hadoop chapter 04] Hadoop configuration log aggregation

[Hadoop chapter 04] Hadoop configuration log aggregation

[Hadoop chapter 04] Hadoop configuration log aggregation

Related information

[Hadoop chapter 04] Hadoop configuration log aggregation

This paper is a complete setGitHub:https://github.com/zhutiansam…

Supporting official account: FocusBigData

Reply to big data interview experience big data learning Roadmap