Create an HBase cluster with a billion level platform – Notes



Distributed key value database for billions of datareal timeWarehousing andfastRandom access to. Millions of QPS and Pb level data need special learning.

It is rare for HBase to operate with MP, and it operates more with Java API.

Component composition
  • Hmaster: Cluster Management

  • Hregion server: specific data access

  • Zookeeper: cluster state management and metadata storage

HBase component composition
Data storage
  • Local file system

  • Or HDFS distributed file system

  • Or other object storage: S3 (AWS), OSS (aliyun), OBS (Huawei cloud)

100 million platform cluster

  • 1. Server selection

  • 2. Configuration optimization

  • 3. Daily operation and maintenance

1、 Server selection

The contents are as follows:

  • 1. Determine the load capacity of the cluster

  • 2. Determine required memory

  • 3. Determine CPU model and number of cores

  • 4. Determine disk type and capacity

  • 5. Determine network bearer

Determine the load capacity of the cluster

Determining the maximum bearing capacity is the most basic requirement of HBase. When the processing capacity is moderate, the processing capacity of HBase is expanded horizontally according to the regionserver: for example, the overall 10w/s read-write capacity of the cluster is expanded horizontally according to the regionserver. The bearing capacity of the cluster is designed as follows:

Load capacity design of cluster

With this configuration, the daily warehousing volume is 8.6 billion. Calculation rules:

1w/s * 3600 seconds /h * 24h * 10 hosts = 864million, about 8.6 billion
Determine required memory

HBase memory database is a co memory database. Data writing rules:

  1. First stored inMemostoreMedium,

  2. At the same time, the frequently queried hot data is cached in memoryRedCache, improve performance.

HBase memory database

So HBase is rightHigh memory requirementsIn order to ensure the stable operation of HBase, the memory configuration of the HBase server is required to be:16GB、32GB、64GB

Determine the number of CPU cores

In order to ensure fast processing of HBase data, we select 4 cores, 8 cores and 16 cores.

Different configurations are configured according to the requirements for real-time data. If the requirements for real-time data are not high, we choose 4-core 16g based on the principle of sufficiency. The ratio of CPU to memory is 1/4, as shown in the following figure:

Relationship between CPU cores and memory
Determine disk type and capacity

1. Disk selection:

  • Hdd: moderate data case and number of read requests

  • SSD: there are a large number of efficient read requests

Selection of the following requirementsSSDSpeed data read efficiency:

  • Large data, such as 10GB

  • Frequent data read requirements

Selection of the following requirementsHDSThat is:

  • High speed write, such as 500m/s

  • Sparsely written scenarios

2. Disk capacity

Determine the disk capacity and disk size according to the data volume. Refer to the following indicators:

  • data structure

  • compression algorithm

  • Number of copies

  • Data storage duration

Methods: determine the size of each data by writing some data in the test environment, and then determine the disk size according to the storage length and copy.

Determine the carrying capacity of the network

The replica mechanism of HBase needs to store multiple copies of one data on multiple HDFS in real time. If the data of one copy is lost, the data can be recovered from other copies.

The network requirements areAs long as it can ensure that the network can write data from multiple copies in real time

Replication mechanism of HBase

1. Replica count and network bandwidth algorithm

Estimation method:

  • Suppose a piece of data has a size of 10KB and a write speed of 100000 / s. The amount of data written per second is as follows:
10KB * 100000 /1024mb = 976mb
  • Save 3 copies of data, then the network bandwidth is at least
3 * 976mb/s = 2928mb/s approx. 2.9GB


A good network operation environment can also ensure that the time required for cluster rebalancing will not be too long.

Server selection summary

project main points
Quantity bearing capacity
Required memory Writes are cached to the memstore and hot data cache redcache first,16G、32G、64G
CPU model and number of cores 1/4 of the memory, low speed requirements, optional 4 cores
Disk selection and capacity SSD and HDD
Capacity consists of “data structure, compression algorithm, number of copies, and data storage time
Network capacity Impact of replica mechanism computing bandwidth and network environment on relief

Configuration optimization

Operating system and HBase cluster parameter tuning to achieve optimal performance.

1、 Operating system tuning

  • Number of file handles

  • Maximum virtual memory

  • Swap memory settings

It is also applicable to NoSQL databases such as mangodb, Cassandra and elasticsearch.

Number of file handles

LinuxdefaultThe number of handles to is 1024, which is not suitable for Linux as a server. Modify it as follows:

echo "* soft nofile 65535" >>  /etc/security/limits.conf

echo "* hard nofile 65535" >>  /etc/security/limits.conf

Maximum virtual memory

max_ map_ Count: defines the maximum memory area that a process can have. The recommended settings are as follows:

sysctl -w vm.max_map_count=102400

Swap memory settings

Swap is enabled for server throughput, but HBase needs an environment where memory operations can be performed quickly. Turning off swap will improve the reading efficiency. The settings are as follows

echo  "vm.swappiness = 0" >> /etc/sysctl.conf

2、 HBase configuration optimization

There are many configuration optimization items. Here are the main parameters

JVM memory configuration for HBase regionserver
  • 1. Heap memory

HBase real-time data is first written to the memory, then to the disk, and the hotspot data is cached at the same time.

Allocation quantity calculation rule reference to operating system32g total memoryFor example:

HBase JVM heap memory: 24g (3/4)

Operating system +jvm off heap memory: 8g (1/4) — save the system for stable operation

The configuration information is as follows

hbase_regionserfver_opts -Xmx24g -Xms24g -Xmn6g

#Xmx - sets the amount of heap memory used when the program starts

#XMS - sets the maximum amount of memory that can be used during program operation (exceeding OOM)

#XMN - younger generation size
  • 2. G1 configuration of garbage collector

G1 garbage collector can effectively reduce the number of JVM full CG.

-20: +useg1gc \\g1 garbage collection is configured, and JDK is upgraded to 11 (not supported by jdk8)

-XX: MaxGCPauseMillis = 500

-XX: +ParalleRefProcEnabled

-XX: -ResizePLAB

-XX: +ParalleGCThreads=8

-Xloggc: /data/log/hbase/gc/gc-%t.log

-XX: +PrintGCDetails

-XX: +PrintGCTimeStamps

-XX: +PrintGCCause

-XX: +PrintTenuringDistribution

-XX: +UseGCLogFileRotation

-XX: NumberOfGCLogFile=20

-XX: GCLogFileSize=5M
HBase thread parameter settings

Related to CPU computing resources

  • Hbase. regionserver. handler. Count: number of threads supported by regionserver at the same time
Number of threads supported by regionserver at the same time
  • compaction. Small and compact large

Load all the compaction requests. When the total size of the file compaction > (greater than) the throttlepoint, the comption is allocated to the largecomparison thread pool. Otherwise, it is processed by the smallcomparison thread pool.

Specific configuration

#Configure throttlepoint, which defaults to 2.5G

hbase.regionserver.thread.compaction.throttlePoint: 2.5G

#Both smallcomparison and largecomparison thread pools have only one thread by default

hbase.regionserver.thread.compaction.small: 4

hbase.regionserver.thread.compaction.large: 6
  • hbase.regionserver.max.filesize

The basic process of data writing, data merging and splitting.

Data writing process
  1. Write the memstore first. When the threshold value is exceeded, write the storefile (hfile). When the threshold value is exceeded, start the compilation and merge into a storefile. When the merged storefile is greater than hbase regionserver. When the parameter max.filesize is set, the split action will be triggered and split into two regions

  2. hbase. regionserver. The required size of max.filesize is moderate. When the filesize is too small, the probability of triggering splitting is greater, and the overall access service of the system will be unstable. When the filesize is too large, it will take a long time for a compaction and split, and even cause a sense of pause. Too large is not suitable for frequent compaction and split, which has a great impact on the application reading and writing.

  3. Practical experience: in the case of high concurrency, the optimal size is 5~10g

#Parameter settings are related to the size of a single piece of data and the number of regions

#Parameter tuning required

hbase.regionserver.max.filesize 10G
  1. Close the major of HBase table at peak time_ compact

Close major of HBase table_ Compact is to avoid major_ Impact of compact on system performance. In the off peak period, call major again_ Compact consolidates HBase tables to reduce split time and greatly improve cluster performance and throughput.

The statement is as follows:,

major_compact 'table_name'

  • hfile.block.cache.size

Size of regionserver cache,

#Represents 0.2% of the total heap memory of HBase

hfile.block.cache.size 0.2

The query performance will be improved by increasing the query size;If a large number of queries are dominant and the writing frequency is not high, set it to 0.5.

  • hfile.hregion.memstore.flush.size

The default size of a regionserver’s memstore is 64MB. Refer to the average number of regions managed by each regionserver. If the number of regions managed is small, it can be adjusted to a larger size, as follows:

hfile.hregion.memstore.flush.size 512MB



UPPERLIMIT and lowerlimit specify the percentage of total memory used by memstore. UPPERLIMIT starts to brush the disk and lowerlimit stops to brush the disk.

Relationship and function between UPPERLIMIT and lowerlimit

Read frequently and write infrequently: it should be properly reduced to make more memory available to the query cache.

hfile. regionserver. global. memstore. UPPERLIMIT 6 \

hfile. regionserver. global. memstore. Lowerlimit 4 \

HBase routine maintenance

  • Specify rowkey rule

  • Region pre partition

  • Turn on data compression

  • Monitoring alarm

Specify rowkey rule

Unreasonable rules will cause uneven data on the regionserver, skew data, and overheat a regionserver.

1. Try to save rowkey randomness

Specify rowkey rule

2. Pre create partition

create 't1','f1',SPLITS_FILE => 'splits.txt';

Multiple regions are divided in advance according to the possible rowkeys instead of one, so that multiple subsequent operations can be balanced to different regions to avoid hot spots.

Turn on data compression

The data stored in HBase is mainly at Pb level, which will cause high server disk costs.

create 'test',{NAME -> 'C', COMPRESSION => 'SNAPPY'};

The default compression algorithms of HBase are GZ, LZO and snappy. Snappy algorithm has the best compression rate and performance.

Monitoring alarm

Install cloudera monitoring to monitor the real-time status of HBase.