Hadoop MapReduce Spark Configuration Item

Time:2019-9-20

Scope of application

The configuration items covered in this article are mainly for Hadoop 2.x and Spark 2.x.

MapReduce

Official documents

https://hadoop.apache.org/doc…
Lower left corner: mapred-default.xml

Hadoop MapReduce Spark Configuration Item

Examples of configuration items

name value description
mapreduce.job.reduce.slowstart.completedmaps 0.05 Resource requests for Reduce Task will not be made until the percentage of Map Task completed reaches that value.
mapreduce.output.fileoutputformat.compress false MapReduce Job output requires compression

Spark

Official documents

https://spark.apache.org/docs…
Finally, it was mentioned that if you want to configure Hadoop items directly, you can add spark. Hadoop before Hadoop items.
Custom Hadoop/Hive Configuration

Hadoop MapReduce Spark Configuration Item

Examples of configuration items

Property Name Default Meaning
spark.dynamicAllocation.enabled false Whether memory is allocated dynamically (Spark native configuration item)
spark.hadoop.mapreduce.output.fileoutputformat.compress false Whether Job’s result output uses compression (Hadoop MR configuration item)

This article is from: Walker snapshot

Recommended Today

MySQL creates and authorizes users

I. create users 1. Root user (administrator) logs in and enters mysql database mysql> use mysql Database changed 2. Create users 1. Create users: # specify Sroxi user login with IP: 192.168.43.144 create user ‘Srox’@’192.168.43.144’ identified by ‘123’; # specify IP: 192.162.43. Sroxi user at the beginning of login create user ‘Sroxi’@’192.118.1.%’ identified by ‘123’; […]