EBay: building mission critical multi data center applications with mongodb

Timeļ¼š2020-9-29

As one of the world’s top 10 global retail brands, with 170 million active buyers and 1 billion online markets, eBay can’t afford the loss of system downtime. That’s why the company relies on mongodb as one of its core enterprise data platform standards for running ebay.com Provides support for multiple customer facing applications.

At this year’s mongodb global conference this year, Qu Feng, chief NoSQL DBA of eBay, provided a practical design pattern for elastic applications – a set of architectural blueprints developed by his team to support enterprise level mongodb deployment.

Mr. Qu began to discuss changes in the concept of usability over the years. In the past, websites could accept scheduled downtime for weekly maintenance. With the globalization of current services, users or businesses will not accept such frequent downtime! In addition, most companies now build their services on commercial hardware platforms, rather than the previous sun Solaris / SPARC servers. Although the cost of commercial hardware is much lower, it often fails. These two factors fundamentally change how the engineering team considers usability and guides eBay to create its “elastic design pattern” to establish database best practices that maximize mean time to failure (MTTF) and minimize mean time to recovery (MTTR).

To build applications, eBay developers can choose from five recognized database standards. In addition to mongodb, the team can choose to use Oracle or MySQL relational databases and two NoSQL databases. Mr. Qu’s DBA team provides guidance on the appropriate database selection based on the application’s data access mode, user load, data type, etc.

EBay currently runs more than 3000 non relational database instances to support a range of applications that can manage petabytes of data. In the past, oracle was a “record system,” rather than a relational database, which handled temporary data used in the participating system. However, the non relational database environment is mature. Through consistent point in time backup and recovery, mongodb now also serves the use cases of the recording system on eBay.

While all of eBay’s non relational database choices provide built-in fault recovery capabilities, they allow different designs to influence the behavior of the application. The DBA team evaluates these differences on six dimensions: availability, consistency, persistence, recoverability, scalability, and performance. For example, NoSQL databases with point-to-point, ownerless design have expensive data repair and rebalancing processes that must be started after a node failure. This rebalancing process affects application throughput and latency, and can cause connection stacking as clients wait for recovery, which can cause application downtime. In order to mitigate these impacts, eBay had to layer the application level products originally developed on oracle on these databases. This approach enables the DBA team to divide a larger cluster into a series of sub clusters, thus isolating the rebalancing overhead from a smaller set of nodes while affecting only a small number of queries. The eBay DBA team built its elastic design pattern for these different types of database behavior.

Mr. Qu introduced eBay’s “mongodb elastic design pattern”, as shown in Figure 1.

EBay: building mission critical multi data center applications with mongodb
Figure 1: the eBay design pattern of mongodb recovery architecture. (image courtesy of eBay’s mongodb World Congress demo).

In this design pattern, a 7-node mongodb replica set is distributed across eBay’s three U.S. data centers. This pattern ensures that in the event of a primary data center failure, the database cluster can maintain availability by establishing an arbitration between the remaining data centers. Mongodb’s replica set members can be assigned election priorities to control which slave members are considered candidates for promotion in the event of primary member failure. For example, if the primary member of the replica set fails, the node local to DC1 is preferred. Only when the whole DC1 is interrupted, the replica set members in DC2 are considered to be eligible for election. New primary members are selected according to which node has performed the latest write operation. This design pattern can be extended by using mongodb’s major write concern to enable persistent writes across data centers.

The standard mongodb design pattern is used as the basis for eBay’s “read intensive / highly available read mode,” which is used to support the eBay product catalog. For the directory load, mongodb replica set can be expanded to 50 members, providing read scalability and recovery capability for large amount of concurrent data distribution.

For more write intensive loads, eBay has developed “extremely high read / write mode”, which deploys a distributed mongodb cluster in its US data center.

EBay: building mission critical multi data center applications with mongodb
Figure 2: mongodb’s extremely high read / write mode of eBay design pattern. (image courtesy of eBay’s mongodb world demo).

Second, eBay developers can use specific mongodb write and read configurations to design patterns to adjust the level of persistence and consistency that best meets different application requirements.

Mr. Qu pointed out that with the recent increase in product functions, mongodb is increasingly meeting the needs of a wider range of applications:

  • Adding regional shards to mongodb 3.4 enables eBay to serve applications that need to provide distributed, persistent write availability across multiple data centers.
  • Rewritable writing for the upcoming mongodb version 3.6 will allow eBay to reduce application exception handling code.

Review Qu Feng’s speech at mongodb world to learn more about eBay’s design patterns.

Download the mongodb multi data center Deployment Guide to learn more about using mongodb to enable primary / primary data center deployment and global data distribution.

This article is translated from: https://www.mongodb.com/blog/post/ebay-building-mission-critical-multi-data-center-applications-with-mongodb


I am Shanghai xiaopang [miracle young], focusing on the Devops of mongodb, mysql, redis and other open source databases, embracing open source and accepting fees.

Original address of Shanghai xiaopang https://segmentfault.com/u/shanghaixiaopang/articles

You are welcome to comment.

Every Friday, please look forward to Shanghai xiaopang.

If Xia Yuhe is still waiting for me by Daming Lake, I will not change.