Big data cluster building – 2. CDH cluster installation – NTP, mysql, nginx installation

Time:2020-2-13

5、 Install ntpd time server

According to cloudera's official recommendation, all CDH nodes and cloudea manager nodes need to start the ntpd service. Otherwise, the following errors will be reported: 
1. The health of the host for this role is poor. The following health tests are poor: clock skew 
2. The host's NTP service is not synchronized to any remote server.

resolvent:
Configure an NTP server with all CDH nodes as clients of the NTP server.
1. An NTP server, 192.168.168.81. This machine is responsible for synchronizing standard time with NTP service of external network and acting as NTP server in LAN.
2. Two NTP clients, 192.168.168.82192.168.168.83. These machines synchronize with the 192.168.168.81 server.
1. Check whether NTP time server is installed
rpm -qa | grep ntp
2. Install NTP service
yum install -y ntp ntpdate
3. Modify NTP configuration file
vi /etc/ntp.conf

-----------------------------------------------------
What the NTP server needs to change is
-----------------------------------------------------
#The most active time server in China: http://www.pool.ntp.org/zone/cn
server 0.cn.pool.ntp.org
server 0.asia.pool.ntp.org
server 3.asia.pool.ntp.org

# allow update time by the upper server
#Allow the upper level time server to actively modify the local time
restrict 0.cn.pool.ntp.org nomodify notrap noquery
restrict 0.asia.pool.ntp.org nomodify notrap noquery
restrict 3.asia.pool.ntp.org nomodify notrap noquery

# Undisciplined Local Clock. This is a fake driver intended for backup
# and when no outside source of synchronized time is available.
#When the external time server is not available, use local time as the time service
server  127.127.1.0     # local clock
fudge   127.127.1.0 stratum 10


4. Start NTP service
Service ntpd start start NTP service
service ntpd status 

5. Set startup
chkconfig ntpd on

Check whether the service has been opened
systemctl list-unit-files | grep ntp

6. View service connection and monitoring
netstat -tlunp | grep ntp

7. View the NTP servers in the network, and display the relationship between the client and each server
ntpq -p

8. Check the time synchronization status, which generally takes 5-10 minutes to connect and synchronize successfully. So, wait a moment after the server starts
ntpstat

Client configuration

1. Configure the / etc / ntp.conf file, and add the following configuration in this file: 
server 192.168.168.81 prefer 
2. Execute chkconfig ntpd on to set the startup.
3. Execute service ntpd start to start the ntpd service.
4. Synchronize NTP server time 
ntpdate -u 192.168.168.81
5. Execute hwlock – systohc to synchronize the system time to the hardware BIOS.

After NTP is installed, it is set to start automatically after power on, but NTP does not start after restart.
This is because CentOS 7 has already installed a similar tool, chronyd.
You need to disable chronyd first.

systemctl enable ntpd
systemctl disable chronyd

6、 Install MySQL

1. Check whether MySQL server has been installed
rpm -qa | grep -i mysql

2. Download 7.5 rpm
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm
sudo yum update

3. Install MySQL server
sudo yum install mysql-server

4. Start mysqld
sudo systemctl start mysqld

5. Set startup
sudo systemctl enable mysqld

6. Configure MSYQL
Stop MySQL service first
sudo systemctl stop mysqld



Install JDBC Driver
1. Download JDBC Driver
http://www.mysql.com/downloads/connector/j/5.1.html (in .tar.gz format).
2. Decompress the driver package
tar zxvf mysql-connector-java-5.1.46.tar.gz
3. Copy the jdbc driver, rename it, and put it in / usr / share / Java /
If the destination directory does not exist, create it.
sudo cp mysql-connector-java-5.1.46-bin.jar /usr/share/java/mysql-connector-java.jar

7、 Install nginx, Mount local repo

Steps:

  • Install nginx and open directory browse
  • Download repo as tarball package of CM and CDH
Install nginx and open directory browse
Install the source corresponding to nginx
rpm -Uvh http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm
yum install nginx

Open directory permissions
vi /etc/nginx/nginx.conf

Add the following parameters to the configuration file
autoindex on;
autoindex_exact_size on; 
autoindex_localtime on;

autoindex_exact_size on;
Shows the exact size of the file in bytes.
After changing to off, the approximate size of the file will be displayed, in KB or MB or GB

autoindex_localtime on;
The default is off, and the displayed file time is GMT.
After changing to on, the displayed file time is the server time of the file
Download repo as tarball package of CM and CDH
Download the tar package corresponding to the OS release version

Cloudera Manager 5: 
https://archive.cloudera.com/cm5/repo-as-tarball/


Check MD5 value of CentOS
md5sum -c cm5.14.3-centos7.tar.gz.md5

Copy the extracted file to the HTML directory
cd /usr/share/nginx/html

We can visit our warehouse at http://dev-hd001/cm/

Guidance address

https://www.cloudera.com/documentation/enterprise/latest/topics/cm_ig_create_local_package_repo.html#cmig_topic_21_3
Define repo file
vi /etc/yum.repos.d/cloudera-cm.repo
Add the following
# Packages for Cloudera's Distribution for cm, Version 5, on RedHat    or CentOS 7 x86_64
name=Cloudera's Distribution for cm, Version 5
baseurl=http://dev-hd001/cm/5.14.3/
gpgkey = http://dev-hd001/cm/RPM-GPG-KEY-cloudera    
gpgcheck = 1
Install CM
sudo yum install cloudera-manager-daemons cloudera-manager-server

Copy drive to CMF folder
cp mysql-connector-java.jar /usr/share/cmf/lib/

Execute MySQL configuration statement
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep scm scm keep

After executing this statement, we find that the db.properties file has changed

[[email protected] soft]# cat /etc/cloudera-scm-server/db.properties
# Auto-generated by scm_prepare_database.sh on Sun Oct  7 09:45:15 CST 2018
#
# For information describing how to configure the Cloudera Manager Server
# to connect to databases, see the "Cloudera Manager Installation Guide."
#
com.cloudera.cmf.db.type=mysql
com.cloudera.cmf.db.host=localhost
com.cloudera.cmf.db.name=scm
com.cloudera.cmf.db.user=scm
com.cloudera.cmf.db.setupType=EXTERNAL
com.cloudera.cmf.db.password=keep



sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep amon amon keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep rman rman keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep hue hue keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep metastore metastore keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep sentry sentry keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep nav nav keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep navms navms keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep oozie oozie keep
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql -uroot -pkeep activity_monitor activity_monitor keep


Start CM
sudo systemctl start cloudera-scm-server

View cm log
sudo tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log

Visit CM
http://dev-fhcb-09-hd001:7180/

Install CDH agent on each machine

sudo yum install -y cloudera-manager-agent