Original author: Sijie Guo
Translation: streamnnative Sijia
Apache bookkeeper is an enterprise class storage system designed to ensure high persistence, consistency and low latency. Pulsar, developed by Yahoo! Research, aims to achieve high availability of Hadoop distributed file system (HDFS) namenode. Before this, namenode did not have high availability and had a single point of failure. It has been successfully launched as a sub project of Apache book since January 2015. Over the past four years, companies such as twitter, Yahoo, and salesforce have used bookkeeper to store and service important data, and have supported many different scenarios. This article will briefly introduce the concept of bookkeeper and related terms.
The developers of bookkeeper (Benjamin reed, Flavio junqueira, Ivan Kelly) have designed a flexible system that can support multiple workloads based on their experience in building zookeeper. Initially, bookkeeper was a pre written logging (wal) mechanism for distributed systems. Now bookkeeper has developed into a basic building module supporting multiple enterprise systems, such as Twitter’s eventbus, Yahoo’s Apache pulsar, etc.
What is bookkeeper?
Bookkeeper is a storage service that optimizes real-time workload. It has the characteristics of scalability, high error tolerance and low latency. According to our years of work experience, the enterprise real-time storage platform should meet the following requirements:
- Read and write entry stream with very low latency (less than 5 ms)
- It can store data persistently, consistently and fault tolerance
- When writing data, can carry on the streaming transmission or the rear end transmission
- Effectively store and access historical data and real-time data
For example, it is designed to provide multiple copies of the data in a distributed system, such as multi server, multi server, etc Eventbus and Apache pulsar provide storage services; store immutable objects (such as snapshot of checkpoint data) for streaming work.
The concept and terminology of bookkeeper
Bookkeeper copies and persists the log stream. Log flow is a record stream that forms a good sequence.
Data is written to the Apache bookkeeper’s log in an indivisible sequence, rather than a single byte.recordIs the smallest I / O unit in bookkeeper, also known as the address unit. A single record contains a sequence number (such as an incremental length) associated with or assigned to the record. The client always reads from a specific record, or a trailing sequence. In other words, the client will listen to the sequence to find the next record to be added to the log. The client can receive a single record or a data block containing multiple records. Serial numbers can also be used to retrieve records at random.
Bookkeeper provides two nouns for log storage: one isledger(also known as log segment); the other isstream(also known as log flow).
Ledger is used to record or store a series of data records (logs). When the client actively shut down or the client acting as the writer goes down, the records that are writing to the ledger will be lost, while the data previously stored in the ledger will not be lost. Once a ledger is closed, it is immutable, that is, adding data records (logs) to a closed ledger is not allowed.
Figure 1 bookkeeper ledger: bounded data entries sequence
StreamLog flow is an unbounded and infinite sequence of data records. By default, stream is never lost. Stream is different from ledger. When appending records, ledger can only run once, while stream can run multiple times. A stream is composed of multiple ledgers, each of which cycles according to a rolling strategy based on time or space. Streams may exist for a relatively long time (days, months, or even years) before they are deleted. The main data retention mechanism of stream is truncation, which includes deleting the oldest ledger according to the time or space-based retention policy.
Figure 2 bookkeeper stream: unbounded data record stream
Ledger and stream provide a unified storage abstraction for historical data and real-time data. When writing data, the log stream transmission or rear end transmission of real-time data records. Real time data stored in ledger becomes historical data. The data accumulated in the stream is not limited by the single machine capacity.
In general, users classify and manage log streams in the namespace. A namespace is a mechanism used by tenants to create streams and is also a deployment or snap in. Users can configure data placement policies at the namespace level. All streams in the same namespace have the same namespace settings and store records in the storage node configured according to the data placement policy. This provides a strong support for the mechanism of managing multiple streams at the same time.
Books is the storage server. A bookie is a separate bookkeeper storage server for storing data records. Bookkeeper copies and stores data entries across books. For performance reasons, the ledger segment is stored on a single bookie instead of the entire ledger. So bookie is like part of the whole integration. For any given ledgerLIntegration refers to storageLA set of books in entries. When entries are written to ledger, entries cross integrationsubsection(write one group of books instead of all books).
Bookkeeper needs metadata storage service to store information about ledger and available bookie. At present, bookkeeper uses zookeeper to complete this work (in addition to data storage services, it also includes some coordination, configuration management tasks, etc.).
Interact with bookkeeper
When interacting with bookie, the bookkeeper application has two main functions: one is to create a ledger or stream for writing data; the other is to open a ledger or stream to read data. To interact with two different storage primitives in bookkeeper, bookkeeper provides two APIs.
|API | description|
|Ledger API | the lower level API allows users to interact directly with ledger, which is very flexible. Users can interact with bookie as needed. A kind of
|Stream API | a high-level, stream oriented API, implemented through Apache distributedlog. Users can interact with stream without managing the complexity of interaction with ledger. A kind of
Which API to use depends on the degree of granularity control the user has over the semantics of ledger. Users can also use both APIs in a single application.
Put them together
The following figure shows a typical installation example of bookkeeper.
Several notes in the above figure:
- A typical bookkeeper installation includes a metadata store (such as zookeeper), a bookie cluster, and multiple clients that interact with bookie through the provided client library.
- To facilitate client identification, bookie will broadcast itself to the metadata store.
- Bookie interacts with the metadata store to collect deleted data as a recycle bin.
The application interacts with bookkeeper through the provided client library (using the ledger API or distributedlog stream API)
- Application 1 needs to have granularity control on the ledger to use the ledger API directly.
- Application 2 does not require lower level ledger control, so it uses a simpler log flow API.
This paper gives a technical overview of bookkeeper. First, it introduces the concepts of entry, ledger, stream, namespace and bookie. Then it introduces typical bookkeeper deployment and how to process data.
If you are interested in bookkeeper or distributedlog, you can join our community in the following ways:
- BookKeeper Slack channel。 You can https://apachebookkeeper.herokuapp.com/ Registration.
- BookKeeper email list。
To learn more about the Apache bookkeeper project, visit the official website: http://bookkeeper.apache.org