EBay: building mission critical multi data center applications using mongodb


As the world’s top 10 global retail brands, with 170 million active buyers and 1 billion online market, eBay can’t afford the loss caused by system downtime. This is why the company relies on mongodb as one of its core enterprise data platform standards to run eBay Com provides support for multiple customer-oriented applications.

At this year’s mongodb global conference, Qu Feng, eBay’s chief NoSQL DBA, providedPractical design patterns for elastic applications—His team developed a set of architecture blueprints to support enterprise mongodb deployment.

Mr. Qu began to discuss the changes in the concept of usability over the years. In the past, websites could accept scheduled downtime for weekly maintenance. With the globalization of current services, users or businesses will not accept such frequent downtime! In addition, most companies now build their services on commercial hardware platforms instead of the previous sun Solaris / SPARC servers. Although the cost of commercial hardware is much lower, it also often fails. These two factors fundamentally changed how the engineering team considered availability and guided eBay to create its “elastic design pattern” to establish database best practices to maximize mean time to failure (MTTF) and minimize mean time to recovery (MTTR).

To build applications, eBay developers can choose from five recognized database standards. In addition to mongodb, the team can also choose to use Oracle or MySQL relational database and two NoSQL databases. Mr. Qu’s DBA team provides guidance for appropriate database selection according to the application’s data access mode, user load, data type, etc.

EBay currently runs more than 3000 non relational database instances to support a range of applications that can manage several petabytes of data. In the past, oracle was a “recording system”, rather than a relational database, which handled temporary data used in the “participating system”. However, the non relational database environment has matured. Through consistent point in time backup and recovery, mongodb now also serves the use case of the recording system on eBay.

Although all of eBay’s non relational database choices provide built-in fault recovery capabilities, they can enable different designs to affect the behavior of applications. The DBA team evaluates these differences in six dimensions: availability, consistency, persistence, recoverability, scalability, and performance. For example, NoSQL databases using point-to-point, unowned designs have expensive data repair and rebalancing processes that must be started after a node failure. This rebalancing process affects application throughput and latency and may cause connection stacking because clients are waiting for recovery, which may cause application downtime. In order to mitigate these impacts, eBay had to tier the application level products originally developed on oracle on these databases. This approach enables the DBA team to divide a larger cluster into a series of sub clusters, isolating the rebalancing overhead from a smaller set of nodes while affecting only a small number of queries. The eBay DBA team built its elastic design pattern for these different types of database behavior.

Mr. Qu introduced eBay’s “mongodb elastic design pattern”, as shown in Figure 1.

EBay: building mission critical multi data center applications using mongodb
Figure 1: eBay design pattern of mongodb recovery architecture. (the picture is provided by eBay’s mongodb world conference presentation).

In this design pattern, a 7-nodeMongodb replica setThree U.S. data centers spread across eBay. This mode ensures that in the event of a failure in the primary data center, the database cluster can maintain availability by establishing an arbitration between the remaining data centers. The replica set members of mongodb can be assigned election priority to control which slave members are considered promotion candidates when the primary member fails. For example, if the replica set primary member fails, the node local to DC1 will be preferentially selected. Only when the entire DC1 is interrupted can the replica set members in DC2 be considered eligible for election. A new primary member is selected according to which node has performed the most recent write operation. You can use mongodb’smajority write concernTo extend this design pattern to enable persistent writes across data centers.

The standard mongodb design pattern is used as the basis for eBay’s “reading enhanced / highly available reading mode”, which is used to support eBay’s product catalog. For directory load, the mongodb replica set can be expanded to 50 members, providing read scalability and recovery for large concurrent data distribution.

For more write intensive loads, eBay has developed a “very high read / write mode”, which deploys a distributed mongodb cluster in its U.S. data center.

EBay: building mission critical multi data center applications using mongodb
Figure 2: eBay design pattern of mongodb in read / write mode. (the picture is provided by eBay’s mongodb world demo).

Secondly, eBay developers can use specific mongodb write and read configurations to design patterns to adjust the persistence and consistency levels that best meet the needs of different applications.

Mr. Qu pointed out that with the recent increase of product functions, mongodb is increasingly meeting a wider range of application requirements:

  • Adding regional sharding to mongodb 3.4 enables eBay to provide services for applications that need to provide distributed and permanent write availability across multiple data centers.
  • Rewritable writing for the upcoming mongodb version 3.6 will allow eBay to reduce application exception handling code.

reviewQu Feng’s speech at mongodb world, learn more about eBay design patterns.

downloadMongodb multi data center Deployment Guide, learn more about using mongodb to enable primary / primary data center deployment and global data distribution.

This article is translated from:https://www.mongodb.com/blog/post/ebay-building-mission-critical-multi-data-center-applications-with-mongodb

I am Shanghai xiaopang [miracleyong], Devops focusing on open source databases such as mongodb, MySQL and redis, embracing open source and accepting fees.

Original address of Shanghai xiaopang [miracleyong]:https://segmentfault.com/u/shanghaixiaopang/articles

Welcome to comment.

Every Friday, please look forward to Shanghai xiaopang’s independence.

If Xia Yuhe is still waiting for me by Daming Lake, I won’t be more.