2.5 billion messages per second! Flink is also officially nominated by Apache.


Abstract: Apache Flink is recognized as a new generation of open source big data computing engine, which can support stream processing, batch processing and machine learning, and is one of the most active projects of Apache Software Foundation and GitHub community.

On March 26, Apache’s official blog announced its 21st anniversary. In the blog, the way of Apache, the application of Apache large-scale projects and the milestone events of Apache foundation were reviewed. Among them, Apache Flink broke through the real-time computing message processing peak of 2.5 billion pieces / second in Alibaba double 11 scene in 2019, which was included in the official Apache blog.

With the release of Flink version 1.10 and the formal completion of blink merger, the overall performance and stability of Flink jobs have been significantly optimized. More and more enterprises begin to use Flink for production environment. I believe that Flink’s overall performance will continue to make new breakthroughs with the development of the community and the improvement of ecology.

The following is the translation of the original blog

On March 26, 2020, all volunteers and managers of the Apache Foundation (ASF) and more than 350 open source projects incubated announced its 21st anniversary.

In order to fulfill its mission of providing software to the public, ASF’s volunteer community has grown from 21 members responsible for the development of Apache HTTP server to 765 members, 206 Apache project management committees and more than 7600 contributors responsible for 300 projects, managing more than 200 million lines of Apache code (worth more than $20 billion).

Apache’s breakthrough technology involves all aspects of modern computing, which provides a lot of support for the Internet, manages EB level data, performs several megabytes of operations, and stores trillions of objects in almost every industry. Apache projects are 100% free and have no license fees.

“For the past two decades, the Apache Software Foundation has been a trusted, vendor centric, community led collaboration home,” says David Nalley, executive vice president of the Apache Software Foundation. “Today, ASF is the pioneer of open source, incubating large and small projects, and has a world-class innovation model.”
2.5 billion messages per second! Flink is also officially nominated by Apache.

The way of Apache

As a community led organization, ASF is strictly neutral. Its independence ensures that no organization, including ASF sponsors and organizations contributing to the Apache project, can control the direction of the project or have any special privileges.

The ASF community-centered development process, known as the Apache Road, guides existing projects and their communities and continues to inspire new generation innovation from around the world. Apache is embodied in:

  • Acquired power: all individuals have the opportunity to participate in the community, such as their contribution to the community.
  • Community of the same kind: the honor of an individual participating in the ASF is permanent and is not subject to employment or employer constraints.
  • Open communication: all communications related to code and decision making are publicly accessible to ensure that ASF can collaborate asynchronously within a globally distributed community.
  • Consensus decision: the Apache project is supervised by a team of volunteers who are actively involved in their projects.
  • Responsible supervision: ASF governance model is based on trust and delegated supervision.

The Apache way has always been a pioneer in collaborative computing, and has directly influenced the way source code and open development principles are applied to organizations. The Apache approach has been adopted by numerous organizations, including capital one, Comcast, Ericsson, HP, IBM, Google, Microsoft, PayPal, sap, T-Mobile and many others.

ASF’s attention to the community is closely related to the spirit of Apache, “community is higher than code” is the unswerving tenet. A vibrant and diverse community can keep code alive, but no matter how good the code is, code cannot thrive without community support. Members of the Apache community shared their views on “why Apache” in an upcoming ASF documentary.

ASF Documentary:

Apache large project application

Dozens of enterprise level Apache projects define the industry and are widely used in artificial intelligence, deep learning, big data, construction management, cloud computing, content management, Devops, IOT and edge computing, mobile, server and web framework. No other software foundation can serve such a wide range of projects. Examples of applications supported by Apache include:

  • SF express, the second largest express company in China, uses Apache skywalking to transport it worldwide during covid-19 coronavirus;
  • Apache guacamole’s client free remote desktop gateway can help thousands of individuals, enterprises and universities work safely at home without binding to specific devices, VPNs or clients;
  • Alibaba uses Apache Flink for real-time large screen and real-time recommendation, processing more than 2.5 billion messages per second
  • The mission control system of the European Space Agency Jupiter spacecraft is supported by Apache karaf, Apache Maven and Apache groovy;
  • Gaffer, the application of the GCHQ, uses Apache accuulo, Apache HBase, and Apache parquet to store and manage Pb level data.
  • Netflix uses Apache druid to manage its 1.5 trillion line data warehouse to manage what users see when they click on the Netflix icon across platforms or log in from their browsers.
  • Uber uses Apache Hudi (in incubation) to manage nearly real-time data lake with more than 100pb data, supporting all functions from data warehouse to machine learning;
  • Using Apache’s genome data in Boston hospital to discover the health phenotype of children;
  • Amazon, datastax, IBM, Microsoft, neo4j, NBC Universal, and many others use the Apache tinkerbop graphical database and write complex traversals.
  • Using Apache beam, Hadoop, HBase, Lucene, spark and other tools, the global biodiversity information fund has integrated nearly 1600 institutions, more than 1 million biodiversity data and nearly 140 million location records for research free;
  • The European Commission developed a new API gateway infrastructure using Apache camera;
  • China Telecom uses Apache shardingsphere (in incubation) to expand mobile payment in more than 30 applications to 10 billion data sets;
  • Apple’s Siri uses Apache HBase to complete full ring replication in 10 seconds;
  • The U.S. Navy uses Apache RYA to provide power for intelligent UAVs, autonomous small robot groups, and advanced tactical communication of manned unmanned teams;
  • Hundreds of millions of websites around the world are supported by Apache HTTP servers.

Other milestones

In addition to ASF’s 21st anniversary, the Apache community celebrates milestone days for the following projects:

  • 25th Anniversary – Apache HTTP server
  • 21st Anniversary – Apache OpenOffice (at the ASF place 2011)
  • 20th Anniversary – Apache Mod_ perl, Tcl, APR/Portable Runtime, Subversion (at the ASF since 2009)
  • 18th anniversary – Apache dB, incabator, ant
  • 17th anniversary – Apache cocoon, James, web services, maven, logging services
  • 16th anniversary – Apache Gump, portals, struts, Geronimo, SpamAssassin, xalan, XML graphics
  • 15th Anniversary – Apache Lucene, directory, myfaces, Xerces, Tomcat

Apache blog Original: