[micro service technology topic] Netflix dynamic configuration service – micro service configuration component chameleon archaius

Time:2022-5-2

Premise introduction

  • If you want to design and develop a set of micro service infrastructure, parametric configuration is a very important point, and Netflix also open source a configuration center client called chameleon archaius, and archaius can be said to have more production-level features and more flexibility than other clients.

  • In netflixoss microservice technology stack, almost all other components (such as zuul, hystrix, Eureka, ribbon, etc.) depend on archaius. It can be said that understanding archaius is the basis for understanding and using other microservice components of Netflix.

[micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

d0fede9144c76827d9c481c595336207.jpeg

What is archaius

Netflix archaius is a configuration management library that focuses on dynamic properties from multiple configuration stores. It includes a set of Java configuration management APIs for Netflix. It is mainly implemented as an extension of Apache commons configuration library. The main functions provided are:

Note that Netflix only open source the client part of its configuration center (that is, archius), and there is no open source supporting server side. Archaius is actually independent of the implementation of the configuration source. It can connect with various configuration centers as data sources. Later in this paper, we will introduce how archaius integrates with Apollo configuration center.

Origin of archaius project

  • In the microservice environment, the configuration often needs to be adjusted according to different contexts, or the configuration should be multi-dimensional. For example, in Netflix, the context dimension includes the environment (development, test, and production).

  • Netflix hopes to dynamically adjust the service configuration according to the publishing environment and even the context of the request, so as to make the behavior and logic of the whole Netflix system dynamically configurable, so as to meet the rapidly changing needs of Internet applications. To this end, the Netflix platform team developed the configuration center product, which the team vividly called the chameleon archaius, because the chameleon can dynamically adjust the color of its body according to its environment.

Use case scenario of archaius in Netflix

  1. Turn a function on or off according to the request context.
  2. A page displays 10 products by default. In some cases, you can adjust the configuration through archhaius to display only 5 products.
  3. Dynamically adjust the behavior of the hystrix fuse.
  4. Adjust the connection and request timeout parameters of the service call client.
  5. If an online service generates an error alarm, you can dynamically adjust the log output level (the granularity can be fine to the package or component level), so that you can troubleshoot the problem through the detailed log. After the problem is located, restore the log output level to the default level.
  6. For applications deployed in multiple regions or countries, different functions can be enabled according to different regions and countries through dynamic configuration.
  7. The configuration of some basic middleware can be dynamically adjusted according to the user’s actual access mode, such as the cache lifetime TTL (time to live).
  8. The connection pool configuration of the database access client can be configured with different values for different services. For example, a service with a low request frequency RPS (request per second) can be configured with a small number of connections, while a service with a high request frequency can be configured with a large number of connections.
  9. Changes in runtime configuration can take effect in different dimensions, such as a single instance dimension in a cluster, a region dimension under multi region deployment, a service stack dimension, or an application cluster dimension.
  10. The feature flag is released. Although some functions are online, they are not enabled immediately, but are enabled dynamically through the configuration switch. In this way, some online functions can be turned on or off flexibly according to the situation.
  11. Canary release: when the new function is launched, let the new and old clusters coexist for a period of time, and gradually dynamically adjust the traffic to the old cluster to the new cluster through configuration. If the monitoring shows no abnormality, complete the launch of the new cluster. If there is any abnormality, quickly switch back to the old cluster.

Technical basis of archaius

  • Archaius is an open-source dynamic attribute configuration framework of Netflix. Based on Apache commons configuration, it provides the function of obtaining configuration values at runtime.

  • The core of archaius is the concept of composite configuration that can accommodate one or more configurations. Each configuration can be from such as JDBC, rest interface, XXX Properties file and other configuration sources. You can choose to poll the configuration source at run time for dynamic changes,

  • The final value of the property depends on the topmost configuration containing the property (because it is a composite configuration). That is, if an attribute exists in multiple configurations, the actual value seen by the application will be the value in the topmost slot in the configuration hierarchy. Of course, this hierarchy can be configured.

Architecture design of archaius

Archhaius is actually a package and extension of Apache common configuration library. It provides a set of Java based configuration APIs. The main features include:

  • Configuration can be adjusted dynamically: dynamic, type attribute
  • Configure support types (int, long, Boolean, etc.).
  • High performance and thread safety: high throughput and thread safe configuration operations
  • Provides a framework for pulling configuration, which can dynamically pull changed configuration from the configuration source. (a polling framework that allows users to get property changes to configuration sources)
  • It supports the callback mechanism, which is called automatically when the configuration changes.
  • JMX MBeans are supported. You can view and modify the configuration through jconsole.

For applications (and most web applications) that are willing to use the agreed property file location, composite configuration is provided out of the box (which is one of the powerful functions). For the official website that complies with the configuration, an example is given as follows:

[micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

The core of achaius is aComposite configurationThe concept of is simple and can be understood as a hierarchical configuration. The level has priority, and the configuration of high priority level will cover the configuration of low priority. Each level can obtain configuration from a certain configuration source, such as local configuration file, JDBC data source, remote rest API, etc. The configuration source can also dynamically pull changes at runtime. For example, in the above figure, persistent DB configuration means that the configuration is stored in the relational database, and the corresponding configuration source will pull changes from the database regularly). The final value of the configuration is determined by the top-level configuration. For example, if multiple levels contain a configuration item, the final value seen by the application is the value at the top level of the configuration level. The order of configuration layering can be adjusted.

There are two ways to obtain configuration values through archhaius:
  • One is to obtain the configuration center instance through the configuration manager, then obtain the configuration value through propname, and obtain the configuration through the configuration manager

    [micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius
  • Another way is to obtain the dynamicproperty wrapper of the configuration item through the dynamicpropertyfactory.

    [micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

Implementation principle of archaius

What is archaius?

Archaius provides the function of dynamically modifying the value of the configuration. After modifying the configuration, there is no need to restart the application service. Its core idea is to poll the configuration source, detect whether the configuration changes at each iteration, and update the configuration again if there are changes.

The underlying archhaius provides a concrete implementation of abstractconfiguration that implements Apache common configuration.

[micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

  • Concurrentmapconfiguration provides the function of maintaining the configuration values of configuration items in concurrenthashmap.

    [micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

  • Dynamicconfiguration provides the function of dynamically obtaining all configuration values from the data source and updating the configuration values by polling the data source.

    [micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius


  • Dynamicwatchedconfiguration also provides the function of dynamically updating configuration. Unlike dynamicconfiguration, configuration update is triggered when the data source changes.

  • Dynamicconfiguration is pull mode and dynamicwatchedconfiguration is push mode.

    [micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

Concurrentcommiteconfiguration uses a combination mode to combine different abstractconfiguration implementations. For a configuration center with multiple configuration sources, you can use concurrentcommiteconfiguration. For the same configuration item, when multiple configuration sources have configuration values, take the data of the first matching configuration source.

[micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

How does dynamicpropertyfactory work?

  • Dynamicpropertyfactory holds abstractconfiguration instance.

  • When creating a dynamicproperty object, the dynamicproperty object will get the abstractconfiguration instance held by dynamicpropertyfactory.

  • The dynamicproperty object will register the dynamicpropertylistener with the abstractconfiguration instance. When the abstractconfiguration changes in addition, deletion, modification and query, the current dynamicproperty object will be notified.

    [micro service technology topic] Netflix dynamic configuration service - micro service configuration component chameleon archaius

When a large number of dynamicproperty instances are created, there may be performance problems here. Every time a dynamicproperty is created, a listener will be added. At the same time, if any configuration item changes, the listener will be triggered.

It may be considered that there will not be so many configuration item changes in the production environment. It’s better to trigger the watcher only when the corresponding configuration item changes, such as ZK config.

How is abstractconfiguration injected?

Archhaius only allows one implementation class of abstractconfiguration. If there are multiple configuration sources, you can combine different abstractconfigurations using the concurrentcommiteconfiguration mentioned above.

There are several ways to inject abstractconfiguration.
  1. Configure archhaius default. configuration. Class specifies the abstractconfiguration implementation class

  2. Configure archhaius default. configuration. Factory specifies the abstractconfiguration instance factory method class. The factory method class needs to implement the getInstance method to return the abstractconfiguration instance

  3. Call the install (abstractconfiguration config) method of configurationmanager

  4. Call the initwithconfigurationsource (abstractconfiguration config) method of dynamicpropertyfactory

The above methods are mutually exclusive. Only one of them can be used. After one takes effect, the others cannot be called.

A simple example:
Maven dependency configuration
<dependency>
    <groupId>com.netflix.archaius</groupId>
    <artifactId>archaius-core</artifactId>
    <version>0.7.7</version>
</dependency>
Get configuration source
/**
 * Created by longfei on 17/1/19.
 */
public class DynamicConfigurationSource implements PolledConfigurationSource {
    @Override
    public PollResult poll(boolean initial,Object checkPoint) throws Exception {
        Map<String,Object> map = new HashMap<>();
        map.put("test",UUID.randomUUID().toString());
        return PollResult.createFull(map);
    }
}
Define scheduler
AbstractPollingScheduler scheduler = new FixedDelayPollingScheduler(2000,2000,false);
Define dynamic configuration
DynamicConfiguration configuration = new DynamicConfiguration(source,scheduler);
Simple unit test
@org.testng.annotations.Test
    public void testArchaius() throws Exception {
        PolledConfigurationSource source = new DynamicConfigurationSource();
        AbstractPollingScheduler scheduler = new FixedDelayPollingScheduler(2000,2000,false);
        DynamicConfiguration configuration = new DynamicConfiguration(source,scheduler);
        ConfigurationManager.install(configuration);
        final DynamicStringProperty stringProperty = DynamicPropertyFactory.getInstance().getStringProperty("test","nodata");
        Helpers.subscribePrint(Observable.interval(1,TimeUnit.SECONDS).take(20).doOnNext(new Action1<Long>() {
                    @Override
                    public void call(Long aLong) {
                        System.out.println(stringProperty.get());
                    }
                }),"test");
        TimeUnit.MINUTES.sleep(1);
    }

realization

Start polling task
public synchronized void startPolling(PolledConfigurationSource source, AbstractPollingScheduler scheduler) {
    this.scheduler = scheduler;
    this.source = source;
    init(source, scheduler);
    scheduler.startPolling(source, this);
}
Polling runnable and initialization: the implementation is consistent
PollResult result = null;
    try {
       result = source.poll(false,getNextCheckPoint(checkPoint));
       checkPoint = result.getCheckPoint();
       fireEvent(EventType.POLL_SUCCESS, result, null);
       } catch (Throwable e) {
       log.error("Error getting result from polling source", e);
       fireEvent(EventType.POLL_FAILURE, null, e);
       return;
       }
       try {
          populateProperties(result, config);
       } catch (Throwable e) {
           log.error("Error occured applying properties", e);
      }

Notice that source. Is called Poll method, i.e. polled of polledconfigurationsource. The data source interface we implemented can customize the data source (JDBC, file, SCM, etc.)

summary

In the process of in-depth understanding of archaius, there is an unavoidable “obstacle” is Apache commons configuration, because the former strongly depends on the latter for configuration management. Just how much you know about Apache commons configuration determines how much you know about Netflix archaius.

Reference materials