Big data cluster building – 2. CDH cluster installation – NTP, mysql, nginx installation


5、 Install ntpd time server

According to cloudera's official recommendation, all CDH nodes and cloudea manager nodes need to start the ntpd service. Otherwise, the following errors will be reported: 
1. The health of the host for this role is poor. The following health tests are poor: clock skew 
2. The host's NTP service is not synchronized to any remote server.

Configure an NTP server with all CDH nodes as clients of the NTP server.
1. An NTP server, This machine is responsible for synchronizing standard time with NTP service of external network and acting as NTP server in LAN.
2. Two NTP clients, These machines synchronize with the server.
1. Check whether NTP time server is installed
rpm -qa | grep ntp
2. Install NTP service
yum install -y ntp ntpdate
3. Modify NTP configuration file
vi /etc/ntp.conf

What the NTP server needs to change is
#The most active time server in China:

# allow update time by the upper server
#Allow the upper level time server to actively modify the local time
restrict nomodify notrap noquery
restrict nomodify notrap noquery
restrict nomodify notrap noquery

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
#When the external time server is not available, use local time as the time service
server     # local clock
fudge stratum 10

4. Start NTP service
Service ntpd start start NTP service
service ntpd status 

5. Set startup
chkconfig ntpd on

Check whether the service has been opened
systemctl list-unit-files | grep ntp

6. View service connection and monitoring
netstat -tlunp | grep ntp

7. View the NTP servers in the network, and display the relationship between the client and each server
ntpq -p

8. Check the time synchronization status, which generally takes 5-10 minutes to connect and synchronize successfully. So, wait a moment after the server starts

Client configuration

1. Configure the / etc / ntp.conf file, and add the following configuration in this file: 
server prefer 
2. Execute chkconfig ntpd on to set the startup.
3. Execute service ntpd start to start the ntpd service.
4. Synchronize NTP server time 
ntpdate -u
5. Execute hwlock – systohc to synchronize the system time to the hardware BIOS.

After NTP is installed, it is set to start automatically after power on, but NTP does not start after restart.
This is because CentOS 7 has already installed a similar tool, chronyd.
You need to disable chronyd first.

systemctl enable ntpd
systemctl disable chronyd

6、 Install MySQL

1. Check whether MySQL server has been installed
rpm -qa | grep -i mysql

2. Download 7.5 rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update

3. Install MySQL server
sudo yum install mysql-server

4. Start mysqld
sudo systemctl start mysqld

5. Set startup
sudo systemctl enable mysqld

6. Configure MSYQL
Stop MySQL service first
sudo systemctl stop mysqld

Install JDBC Driver
1. Download JDBC Driver (in .tar.gz format).
2. Decompress the driver package
tar zxvf mysql-connector-java-5.1.46.tar.gz
3. Copy the jdbc driver, rename it, and put it in / usr / share / Java /
If the destination directory does not exist, create it.
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

7、 Install nginx, Mount local repo


  • Install nginx and open directory browse
  • Download repo as tarball package of CM and CDH
Install nginx and open directory browse
Install the source corresponding to nginx
rpm -Uvh
yum install nginx

Open directory permissions
vi /etc/nginx/nginx.conf

Add the following parameters to the configuration file
autoindex on;
autoindex_exact_size on; 
autoindex_localtime on;

autoindex_exact_size on;
Shows the exact size of the file in bytes.
After changing to off, the approximate size of the file will be displayed, in KB or MB or GB

autoindex_localtime on;
The default is off, and the displayed file time is GMT.
After changing to on, the displayed file time is the server time of the file
Download repo as tarball package of CM and CDH
Download the tar package corresponding to the OS release version

Cloudera Manager 5:

Check MD5 value of CentOS
md5sum -c cm5.14.3-centos7.tar.gz.md5

Copy the extracted file to the HTML directory
cd /usr/share/nginx/html

We can visit our warehouse at http://dev-hd001/cm/

Guidance address
Define repo file
vi /etc/yum.repos.d/cloudera-cm.repo
Add the following
# Packages for Cloudera's Distribution for cm, Version 5, on RedHat    or CentOS 7 x86_64
name=Cloudera's Distribution for cm, Version 5
gpgkey = http://dev-hd001/cm/RPM-GPG-KEY-cloudera    
gpgcheck = 1
Install CM
sudo yum install cloudera-manager-daemons cloudera-manager-server

Copy drive to CMF folder
cp mysql-connector-java.jar /usr/share/cmf/lib/

Execute MySQL configuration statement
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep scm scm keep

After executing this statement, we find that the file has changed

[[email protected] soft]# cat /etc/cloudera-scm-server/
# Auto-generated by on Sun Oct  7 09:45:15 CST 2018
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."

sudo /usr/share/cmf/schema/ mysql -uroot -pkeep amon amon keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep rman rman keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep hue hue keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep metastore metastore keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep sentry sentry keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep nav nav keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep navms navms keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep oozie oozie keep
sudo /usr/share/cmf/schema/ mysql -uroot -pkeep activity_monitor activity_monitor keep

Start CM
sudo systemctl start cloudera-scm-server

View cm log
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

Visit CM

Install CDH agent on each machine

sudo yum install -y cloudera-manager-agent

Recommended Today

Oracle scheduled tasks

Timing task query To query Oracle scheduled tasks, you can use: –Scheduled tasks for all users SELECT * FROM dba_jobs; –Timing task of the user select * from user_jobs; In the query results, the what field generally stores the name of the stored procedure (or the specific stored procedure content). Broken = n indicates that […]