How to evaluate the service capability of serverless, this report gives 40 criteria


Introduction:Now, some evaluation organizations have given 40 standards to evaluate the service ability of serverless. These evaluation rules are not only a manifestation of the prosperity and development of technology ecology, but also a reference for new entrants to evaluate the implementation effect of serverless.

Editor’s note: two years ago, we were still discussing what is serverless and how to implement it. Now, some evaluation organizations have given 40 standards to evaluate the service ability of serverless. These evaluation rules are not only a manifestation of the prosperity and development of technology ecology, but also a reference for new entrants to evaluate the implementation effect of serverless.

In Forrester’s function as a service (FAAS) platform evaluation report, we selected Alibaba, Amazon, Google, Huawei, IBM, Microsoft, nimbella, Oracle and Tencent, the nine most influential providers, and studied, analyzed and scored them according to 40 criteria. The report shows the performance of each provider in various aspects, and aims to help professionals engaged in application development and delivery (AD & D) to find the most suitable provider for their own needs.

Forrester Wave ™: First quarter report of function as a service (FAAS) platform in 2021

FAAS platform helps developers quickly create cloud native services

The abstraction of FAAS platform makes developers no longer need to pay attention to complex container or virtual machine cluster management and expansion work, so that they can quickly create cloud native micro services. After the management of the underlying infrastructure is handed over to FAAS providers, developers can write microservices into simple small functions in the programming environment using familiar languages such as Java, C #, JavaScript or python. Then, FAAS providers will automatically expand or shrink these micro services according to service requirements. Developers using FAAS platform express that by avoiding infrastructure management and eliminating complex operations related to it with the aid of abstraction, they can quickly push new ideas into the deployment stage, and determine infrastructure costs according to the actual resource requirements for executing micro services. When selecting a FAAS provider, developers should analyze whether the provider meets the following requirements:

  • Support function and container packaging. As developers deploy more and more types of workload to FAAS platform, FAAS platform should allow developers to simply package a function into a zip or jar file and deploy it, or package custom code into a container conforming to the open container standard (OCI) and deploy the corresponding framework. FAAS platform should support these two options at the same time to provide maximum flexibility for developers in developing and deploying web, content and event driven workloads.
  • Provide robust security features. As developers continue to expand the application scope of FAAS platform, it becomes very important to ensure that relevant personnel can safely access the data and application program interface (API) encapsulated in virtual private network or virtual private cloud (VPC). In addition, while expanding and shrinking the capacity of functions, relevant personnel also need to be able to access these resources quickly without waiting for the time-consuming “cold start”.
  • Support third party ecosystems and open standards. Unless you are willing to rely entirely on a public cloud provider, you should choose a FAAS provider with more convenient platform integration. The features you need to focus on include third-party observability, event binding, and message protocol.

Assessment Summary

This Forrester wave ™ The evaluation report classifies the objects to be evaluated into the quadrants of “leaders”, “strong performers”, “competitors” and “challengers”. This is an assessment of the leading manufacturers in the market and does not represent the overall situation of the market. You can check out our report on serverless technology for more information about this market. We hope this evaluation report is just a starting point. We suggest that customers use the Excel based vendor comparison tool to view the product evaluation and adjust the standard weight (see Figure 1 and Figure 2). Click the link at the beginning of the Web Report on to download the above tools.

How to evaluate the service capability of serverless, this report gives 40 criteria

Figure 1 Forrester wave ™: Function as a service (FAAS) platform, the first quarter of 2021

*The grey circle indicates the manufacturers who are not involved in Forrester evaluation

How to evaluate the service capability of serverless, this report gives 40 criteria

Figure 2 Forrester wave ™: Function as a service (FAAS) platform, the first quarter of 2021

All scores were from 0 (weak) to 5 (strong)

*Indicates the manufacturer not involved in the evaluation

Products of various manufacturers

Forrester evaluated the products of nine manufacturers: Alibaba, Amazon, Google, Huawei, IBM, Microsoft, nimbella, Oracle and Tencent (see Figure 3).

How to evaluate the service capability of serverless, this report gives 40 criteria

Figure 3 manufacturers and products of this evaluation

Manufacturer profile

Through the analysis, we find that each manufacturer has the following advantages and disadvantages.


  • AmazonIn terms of regional coverage, ecosystem breadth and implementation stability, it is in the lead. Forrester wave in 2020 ™ In the report, we mentioned lambda’s outstanding performance in terms of geographical availability, customer adoption, experience of developers deploying functions, and ecosystem support. These advantages will continue into 2021. In addition, the support for Amazon’s elastic file system (EFS), the ability to run OCI compliant containers, and billing granularity up to 1 millisecond make lambda move forward on the road of innovation, making more types of workloads applicable. Lambda’s strong support for server less architecture security and special workloads will continue to be a leader in 2021. Amazon thinks that the life cycle of FAAS workload should be short. Developers who accept this view will find that lambda provides them with a powerful FAAS platform. The platform is gradually supporting third-party tools and container centric deployment process. The platform also provides unique services for state management, database operation and API.

Amazon Web services (AWS) lambda is highly appraised by reference customers. The platform can provide first-class customer support with excellent performance and integrate various AWS services and third-party services. In terms of maintaining close cooperation with experienced people in the industry, the product has also been unanimously appreciated by customers. Reference customers think that the platform has room for improvement in observability, and expect that it can provide more simplified tools for developers. AWS lambda is the most suitable temporary workload for asynchronous operation, including integration of other AWS services, Internet of things, batch processing, new web applications and event driven integration.

  • AlibabaUse containers and open standards to accelerate web and content workloads. In the 2020 report, we described Alibaba’s function compute as “a robust general platform”. In 2021, we found that the platform’s support for containers and open source projects and standards, including opentracing, opentelemetry, grafana and Jaeger, further improved its capabilities. In addition to the platform’s investment in supporting mainstream web framework, content-based workload, 1-millisecond billing granularity and runtime, we will bring Alibaba functional computing into the “leader” quadrant of FAAS market this time. Alibaba function computing is especially suitable for the workload that needs to deal with a large number of videos, and uses the container centered model to achieve function concurrency. For customers with such needs, the platform is the best choice.  

Referring to the customers’ appreciation of Alibaba’s event integration ability of function computing in China’s ISV ecosystem, the paper also highly praised Alibaba’s practice of strengthening customer support during Christmas, new year and other festivals. In terms of technology, customers are very satisfied with the ability to expand to tens of thousands of instances in a few minutes. Reference customers hope that the platform can provide better programming language support in remote debugging, and improve the support for GPU computing. Although the platform is best suited for deploying workload in Hongkong or Chinese mainland, its international coverage is expanding. It now supports 10 regions, with 21 usable areas.

  • MicrosoftAzure functions retains important enterprise level functionality for advanced programs. The consumption plan of azure functions supports multiple programming languages and stateful durable functions. However, if you want to use reserved instances, docker container support and powerful security functions, developers need to upgrade to an advanced special plan, namely azure app service plan. Persistent function provides stateful function and supports binding with azure event hub, while azure event grid can help developers build event driven microservices. This year, through increasing investment, Microsoft has improved the guidance of new users, provided more tools for developers, and supported development and debugging on local computers. It has performed well in the standard of developer experience and got the highest score.

Reference customers highly recognize the ability of azure functions to integrate with other azure services such as key vault to realize confidential storage, especially its powerful built-in observability. The performance and service quality of azure functions also make customers generally satisfied. However, if customers want to use important enterprise functions such as VPC services and reserved instances, they have to choose the higher cost advanced service plan, which is the focus of customers’ complaints. For customers who need other azure services or a global FAAS platform, azure functions is the most suitable solution. But if your team wants to build or integrate enterprise class workloads, it needs to upgrade to the advanced version of the application service plan.

Strong performers

  • Tencent cloudThe SCF platform optimizes cloud centric development. Tencent has created a set of excellent web-based tools for function development, and combined it with reliable runtime to support multiple programming languages, complex workflow and longer running function instances. Developers can reserve function instances, and Tencent also provides professional support for artificial intelligence (AI) workloads and specific industry clouds. SCF can also be used as an extension mechanism of wechat ecosystem to effectively help enterprises interact with customers through wechat extension function. But for developers who want to develop and debug local functions, Tencent’s cloud centric development concept does not fit them well.

Reference customers highly evaluate SCF’s performance, customer support and low cost of workload execution. Customers also expressed concern about the stability of the platform and hoped to use more cloud services from Tencent and third parties through more built-in event triggers. Tencent SCF is a good choice for customers who need to deploy FAAS services in China, integrate and expand with wechat to attract more customers and accept cloud centric development methods.

  • NimbellaFully support the hybrid cloud through the “FAAS everywhere” approach. Like IBM cloud functions, nimbella is based on the open source Apache openwhisk project. Nimbella has a simple and fast process for developers to get started. At the same time, its runtime can be used under the pure FAAS model, can also be deployed to other public clouds as an installable function platform, and can also be deployed locally. Nimbella realizes “FAAS everywhere” through powerful container support, multiple programming languages and configurable function maximum runtime settings. Such a FAAS solution is very suitable for stateful functions that need to run for a long time. Although the platform has high flexibility, some problems still need to be paid attention to. Its hosting solution provides built-in isolation function for tenants, while the installable solution supports mixed deployment. The team can make specific adjustments according to different organizational requirements and security policies. As a result, teams that choose mixed deployment may spend more time configuring the infrastructure of the production environment.

Reference customers highly valued nimbella’s performance in customer support and helping to implement best practices. Customers will also minimize the risk of vendor lock-in as a reason to choose nimbella. Customers’ concerns include security and observability, and they hope to speed up the process by adding code examples and digital content. Nimbella is a suitable solution for the development team which adopts the hybrid FAAS solution and hopes to provide their own network and integration functions on the basis of the server free architecture platform. In addition, if you prefer to pay monthly or choose an enterprise agreement to the usage based pricing model, nimbella’s licensing model is also the most attractive one.

  • GoogleBuild a “comprehensive” solution with cloud functions and cloud run. It can be seen from Google’s FAAS products that as public cloud providers use containers and kubernetes platforms as the key platform engines for FAAS runtime, multiple FAAS platforms begin to transform. Google’s cloud functions support common function packaging and deployment, its runtime can be well expanded and slowly cooled, and Google’s cloud OPS integration function also ensures excellent observability. With Google cloud run, developers can package functions with Google’s functions framework, deploy them with knative, and run them in mixed deployment scenarios. Compared with the evaluation results in 2020, the platform also has stronger support for programming languages.  

Reference customers gave Google a high score in terms of performance and simplicity, and praised the security and scalability of cloud functions. In addition, the direct participation and support of product management and customer support teams have also been unanimously recognized by customers. The areas to be improved include: better support for packaging third-party libraries and functions (for example, for processing Kafka messages), and the need to enhance the transparency of cloud functions runtime dependencies. If you want to deploy applications around the world and smoothly transition from functions to containers in a mixed FAAS deployment scenario, Google cloud run should be the right choice for you.

  • IBMCloud functions has been improved and code engine is ready to be launched. IBM cloud functions is based on the open source Apache openwhisk project. It adds built-in API gateway, good event flow support and powerful container level function isolation. From the results of this evaluation, IBM is superior to any other manufacturer in terms of service level agreement terms and availability. However, some aspects of cloud functions still need to be improved, such as limited network isolation options and support for reserved function instances. In the evaluation in 2020, we pointed out that after IBM acquired red hat, it is necessary to establish a unified function runtime strategy. It seems that IBM cloud code engine will take on this task. Although it has not yet entered the general availability (GA) stage (so it is not included in our analysis), code engine has added kubernetes and knative support to IBM’s FAAS strategy.

IBM’s fast automatic scaling of function instances and high concurrency of computational workload have been approved by reference customers. Other customers point out that the support for openwhisk enables portability, thus reducing the risk of vendor lock-in. The improvements include perfect layout, observability and support for reserved function instances. IBM’s regular customers will find cloud functions very suitable for integrating their workload, and its extensive support for multiple programming languages can also effectively help most developers switch to a server less architecture. IBM cloud code engine after entering the GA stage is also worth looking forward to, especially for customers who are investing in the infrastructure based on kubernetes and want to include functions into packaging objects.


  • Oracle puts security, container and event driven integration first. When we analyzed Oracle cloud functions in 2020, we thought that the platform was relatively new, and the evaluation results also reflected this. In 2021, Oracle’s FAAS platform has been upgraded from the “Challenger” quadrant to the “competitor” quadrant. Oracle cloud functions is based on the open source FN project, and the functions are packaged into OCI containers. When Oracle established the FAAS platform, it mainly focused on the “secure by design” mode, which also met the needs of many enterprise development teams. By default, function instances are only available in subnets of the customer virtual cloud network. Unlike some other FAAS vendors, Oracle’s private cloud support features do not charge extra. As a result, the adoption rate of Oracle in North America, Europe, Asia and Latin America is steadily increasing.

According to the customer’s evaluation on Oracle cloud functions, the performance, ease of use, development time, document depth and overall execution cost are very high. The deficiency lies in the need to come up with alternative solutions to solve the occasional delay problem, which may be due to the lack of support for reserved instances. Oracle cloud functions is very suitable for enterprises using other OCI services. In addition, the platform has been open to 23 regions, which can help customers implement deployment in a global scope.


  • HuaweiFunction graph has just entered the market, but it has a good prospect. Functional workflow platform is still in the stage of gradual development, but we found that the platform has reached the market average level with strong support for development tools, local development and event driven integration. The platform can execute functions that can run for up to 15 minutes, and supports many programming languages such as node.js, Java, python, go, C # and PHP. As a new face in the market, this platform still has some shortcomings. For example, the documents are not perfect, the support for special workload is poor, and the support for integration is not enough. Availability is also limited in some areas. Therefore, we believe that Huawei’s functional workflow platform is most suitable for developers who want to deploy workloads in China, Brazil, Chile, Mexico, Singapore, South Africa or Thailand where Huawei cloud provides regional support. Huawei refuses to participate in the full Forrester wave ™ Evaluation process.

Overview of evaluation report

We have evaluated manufacturers in strict accordance with 40 criteria, which can be divided into three categories:

  • Current products.Every manufacturer in Forrester wave ™ The position on the vertical axis of the quadrant graph represents the performance of its current product function. Key criteria for evaluating these solutions include developer experience, programming model, runtime execution environment, and security features.
  • Strategic vision.The position on the horizontal axis indicates the strategic performance of the manufacturer. We evaluated the vision, execution capability, ecosystem, participation and pricing strategy of FAAS platform providers.
  • Market share.It is represented by the size of the circle on the graph. Our market performance score reflects the geographical distribution, customer base size and product revenue of each manufacturer.  

Manufacturer selection criteria

Forrester evaluated 9 manufacturers: Alibaba, Amazon, Google, Huawei, IBM, Microsoft, nimbella, Oracle and Tencent. They all meet the following conditions:

  • Own FAAS platform。 Each vendor has a FAAS platform that customers can use and run productive workloads at any time.
  • Participated in Forrester’s evaluation of now tech server free architecture providers。 FAAS platform is a key component of building cloud native applications through a server free architecture. In Forrester’s report “now Tech: serverless architecture, the first quarter of 2021”, we introduced in detail the serverless architecture that developers can choose.

Supplementary materials

Online resources

We’ll take all Forrester waves ™ Scores and weights are organized into an excel file, which contains detailed product evaluation and supports custom sorting. Click the link at the beginning of the Web Report on to download the file. We hope that these scores and default weights are just a starting point, and suggest readers adjust the weights according to their personal needs.

Forrester Wave ™ How to evaluate the quality of products

Forrester Wave ™ The report provides a guide for organizations to buy products in the technology market. To ensure fairness to all participants, Forrester follows the “Forrester wave” principle ™ Method guide to evaluate the participating vendors. In the process of this evaluation, we first conduct a preliminary study and select the manufacturers for later evaluation. Then according to the inclusion criteria, the final list was further screened and determined. Then, detailed information about products and strategies is collected through detailed questionnaires, presentations / presentations, and reference customer surveys / interviews. Finally, the information is combined with the industry experience and professional knowledge of analysts, and a relative scoring system is used to compare a manufacturer with other manufacturers participating in the evaluation. We’re at Forrester wave ™ The Forrester wave is clearly indicated in the title of the report ™ Release date (year and quarter) of. In this Forrester wave ™ In the evaluation process of the report, we used the materials provided to us by the manufacturer before January 21, 2021, and did not accept other information provided after that date. We hope readers can analyze how the market and manufacturers’ products have changed over time. According to “Forrester wave” ™ And new wave ™ Forrester requires manufacturers to review our evaluation results before the release of the report to ensure accuracy. In Forrester wave ™ The manufacturers marked as not participating in the evaluation in quadrant diagram meet the inclusion criteria defined by us, but refuse to participate or only partially participate in this evaluation. According to “Forrester wave ™ Forrester new wave ™ They were scored and included in the quadrant chart together with other manufacturers participating in the evaluation.

Integrity criteria

All our research, including Forrester wave ™ The evaluation follows the “integrity criteria” published on our website.

Link to the original report:

Copyright notice:The content of this article is spontaneously contributed by alicloud real name registered users, and the copyright belongs to the original author. The alicloud developer community does not own its copyright, nor does it bear the corresponding legal responsibility. For specific rules, please refer to the user service agreement of alicloud developer community and the guidelines for intellectual property protection of alicloud developer community. If you find any suspected plagiarism content in the community, fill in the infringement complaint form to report. Once verified, the community will immediately delete the suspected infringement content.

Recommended Today

A detailed explanation of the differences between Perl and strawberry Perl and ActivePerl

Perl is the abbreviation of practical extraction and report language “practical report extraction language”. Application of activestateperl and strawberry PERL on Windows platformcompiler。 Perl   The relationship between the latter two is that C language and Linux system have their own GCC. The biggest difference between activestate Perl and strawberry Perl is that strawberry Perl […]