Good programmer big data training sharing: Hadoop cluster synchronization

Time:2022-6-20

Good programmers’ big data training and sharing: Hadoop cluster synchronization brings you the technology sharing – Hadoop cluster synchronization.

1、 Synchronization mode

Select a machine as the time server (hadoop01 here). All the machines are synchronized with the cluster time at regular intervals. For example, the time is synchronized every ten minutes.

2、 Required tools

There are two time synchronization servers: ntpd and ntpdatp. Although using both of them can achieve the purpose of time synchronization, one problem must be clarified before using them. What is the difference between ntpd and ntpdate when updating time. Ntpd is not only a time synchronization server, it can also synchronize time with the standard time server through the client. It is also a smooth synchronization, not an immediate synchronization of ntpdate. Use ntpdate with caution in the production environment, and the two cannot run at the same time.

3、 Specific operation steps

Time server configuration

(1) Check if NTP is installed. [ [email protected] Desktop]\rpm -qa|grep NTP if ntp-4.2.6p5-1 el6.centos. x86_ 64 . fontpackages-filesystem-1.41-1.1. el6.noarch

ntpdate-4.2.6p5-1. el6.centos. x86_ 64 indicates that NTP has been installed. Otherwise, install NTP service first. [ [email protected] Desktop]# yum install -y ntp.

(2) Modify NTP configuration file[ [email protected] Desktop]# vi /etc/ntp. Conf. first configure the authorization of the client, that is, set the permission to access the NTP server for the specified machine (client). This is achieved through the restrict configuration item. Release \restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap. Change to restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap If the cluster is in a closed LAN, the default server can be shielded.

Set server 0 centos. pool. ntp. org iburst. server 1. centos. pool. ntp. org iburst.

server 2. centos. pool. ntp. org iburst. server 3. centos. pool. ntp. Org iburst shielding changed to

#server 0. centos. pool. ntp. org iburst.# server 1. centos. pool. ntp. org iburst.# server 2. centos. pool. ntp. org iburst.# server 3. centos. pool. ntp. org iburst. Use the local clock of this computer as the time supply source, so that even if it loses network connection, it can continue to provide services for the network. Add server 127.127.1.0 fudge 127.127.1.0 stratum Modify /etc/sysconfig/ntpd file

[ [email protected] Desktop]\vim /etc/sysconfig/ntpd add sync_ Restart ntpd after hwlock=yes[ [email protected] Desktop]\\service ntpd restart, set ntpd startup default startup[ [email protected] Desktop]# chkconfig mtpd on。 After completing the above operations, use the date command to check whether the system time corresponds to the current time. If not, it should be the time zone. Modify time zone[ [email protected] Desktop]# tzselect。 Select the correct time zone according to the prompt, and then set CP /usr/share/zoneinfo/asia/shanghai as the link to /etc/localtime

  [[email protected] Desktop]# cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime

4、 Other machine configurations

Configure synchronization with the time server every ten minutes

You need to write time synchronization scripts in other machines in the cluster. Here, take hadoop02 as an example[ [email protected] Desktop]# crontab –e. */10 * * * * /usr/sbin/ntpdate hadoop01

Restart the crond service. [ [email protected] Desktop]# service crond restart。 If necessary, modify the time zone as above.

This is the end of sharing. Welcome to discuss and learn together!