Hytrix practical experience sharing

Time:2021-10-21

1、 Background

Hystrix is an open-source fault-tolerant framework of netlifx. It is a sharp anti avalanche weapon. It has the functions of service degradation, service fusing, dependency isolation, monitoring (hystrix dashboard), etc.

Although hytrix is no longer officially maintained, and there are new frameworks such as Alibaba sentinel, in terms of component maturity and application cases, there are still many projects that continue to use hytrix, one of which is the project I participated in. Therefore, we will share and exchange with you in combination with our own actual combat experience of hystrix.

2、 Experience summary

2.1 selection of isolation strategy

Hystrix provides two resource isolation strategies, thread pool and semaphore. The similarities and differences between them are as follows:

Hytrix practical experience sharing

When using the cache (local memory cache is more suitable for this scenario, and redis and other network caches need to be evaluated), we can use the semaphore isolation strategy, because such services respond quickly, do not occupy the container thread for too long, reduce some expenses of thread switching, and improve the service efficiency.

The specific strategy to use needs to be comprehensively evaluated according to the business scenario. Generally, thread pool isolation is recommended.

two point two   Thread pool size and timeout settings

Under the thread pool isolation strategy, the setting of thread pool size and timeout time is very important, which directly affects the response ability of system services. For example, if the thread pool size is set too large, it will cause waste of resources and overhead such as thread switching; If the setting is too small to support user requests, resulting in request queuing. If the timeout time is set too long, some long-time requests will block the thread, resulting in other normal requests waiting in line; If the setting is too short, too many normal requests will be blown.

For this, the official suggestions given by hystrix are as follows:

Hytrix practical experience sharing

It is converted to the following calculation formula:

  • Thread pool size = service tp99 response time (in seconds) * requests per second + redundant buffer value
  • Timeout (in milliseconds) = 1000 (milliseconds) / requests per second

For example, a service tp99 will receive 30 requests per second, and then the response time of each request is 200ms. According to the above formula, we can get: thread pool size = 0.2 * 30 + 4 (redundant buffer value) = 10, timeout = 300ms

2.3 annotation overlay

In actual development, you may encounter the case that some external calling method has hystrix annotation used together with other annotations, such as query method plus cache annotation. At this time, pay special attention to the execution sequence between annotations to avoid unexpected results:

  • Cache annotation is not valid

At this time, the implementation of the hystrix annotation section is at the outermost layer. Because the internal implementation of hystrix obtains the target object through the proceedingjoinpoint. Gettarget(), it is directly executed to the target object method by reflection call, resulting in the loss of other annotation logic in the middle. You can specify the annotation execution order @ order to ensure that the hystrix annotation is executed at the innermost layer.

  • The query method was blown due to cache exception

If the execution of the hystrix annotation aspect is at the outermost layer, the method logic managed by hystrix includes not only the remote call of the third-party service, but also the cache call logic. If there is an exception in the cache call, it will be counted as the exception of the whole method, causing the whole method to be blown.

2.4 service exception handling

Let’s give you time to look at the following code and check whether there are problems:

@HystrixCommand(fallbackMethod="queryUserByIdFallback")
public User queryUserById(String userId) {
  if(StringUtils.isEmpty(userId)) {
    Throw new bizexception ("illegal parameter");
  }
  
  Result<User> result;
  try {
    result = userFacade.queryById(userId);
  } catch(Exception e) {
    log.error("query user error. id={}", id, e);
  }
  
  if(result != null && result.isSuccess()) {
    return result.getData();
  }
  
  return null;
}

During operation, hystrix will determine whether the fuse of each dependent command is open according to the success rate or failure rate information of the call request. If opened, subsequent requests will be rejected. It can be seen that the control of exceptions has a great impact on the operation effect of hystrix.

Looking back at the above example, you will find two exception handling problems:

  • Exception handling when parameter verification fails

Abnormal failures of non system calls such as illegal parameter verification shall not affect the fusing logic and shall not be counted as failures within the statistical range. The optimization suggestion is to put the parameter verification outside the encapsulated method of remote call, or package it as hystrixbadrequestexception and throw it. Because the exception of hystrixbadrequestexception in the internal logic of hystrix is not counted as failure by default.

  • Exception handling of try catch remote call

Trying to catch a direct call to a remote service will directly “swallow” the exception, which will directly cause the unavailability of services such as network exceptions that cannot be obtained by hystrix. It is recommended to throw the exception after the catch log record is processed.

two point five   Fallback method

When relying on service calls, hystrix supports graceful service degradation by adding the fallback method to return the default value. However, there are many points to pay attention to in the use of fallback, which are roughly summarized as follows:

  1. The fallback method access level and parameters shall be consistent with the corresponding dependent services
  2. The logic executed in the fallback method should be as light as possible, such as using local cache or static default value to avoid remote call
  3. If there is a remote call in the fallback method, it is recommended to wrap it with hystrix and ensure isolation from the main command thread pool
  4. Fallback degradation is not recommended for remote calls to write operations

2.6  groupKey、commandKey、threadPoolKey

You must have seen these three keys in the development of using hystrix, but many people do not understand the meaning of these three keys and their role in hystrix, especially threadpookey. Therefore, I summarize here:

groupKey

Command methods can be grouped through group key to facilitate hystrix data statistics, alarm and dashboard display. It is generally distinguished according to the business type of the remote service. For example, the account service defines a group key and the order service defines another group key.

The default value is the class name of the method annotated by the @ hystrixcommand annotation.

commandKey

The identification name of the specific command method, which is often used to set the dynamic parameters of the command.

The default value is the method name annotated by the @ hystrixcommand annotation.

threadPoolKey

It is used to identify the thread pool to which the command belongs. Commands with the same threadpoolkey use the same thread pool.

If the key is not specified, the default value is groupkey, that is, the class name of the method annotated by @ hystrixcommand annotation.

In the actual project, we will suggest to specify the thread pool through threadpoolkey instead of the default method of groupkey, because there will be scenarios where a command needs to be thread isolated from other commands in the same group to avoid mutual influence.

2.7 parameter priority

By default, hystrix provides four levels of parameter value configuration methods:

Global default value

The default value of hystrix’s own code, which is written in the source code, takes effect when the user does not configure any parameters.

Example: execution.isolation.thread.timeoutinmilliseconds timeout the global default value is 1000, in milliseconds

Dynamic global default property

Such configuration parameters can change the global default values.

Example: the timeout value set by the property name hystrix.command.default.execution.isolation.thread.timeoutinmilliseconds

Instance initial value

The initial value of fuse instance. After configuring such parameters, the default value is no longer used. That is, the attribute value written in the code annotation.

Example: @ hystrixproperty (name)=   “execution.isolation.thread.timeoutInMilliseconds”, value =   “5000”)

Dynamic instance property

The parameter value of a fuse instance can be dynamically adjusted

Example: the timeout value set by the property name hystrix.command.hystrixcommandkey.execution.isolation.thread.timeoutinmilliseconds

Priority relationship:

Dynamic instance property  >  Instance initial value  >  Dynamic global default property  >  Global default value

two point eight   Dynamic configuration of parameters based on configuration center

By default, hystrix uses archius to implement dynamic settings, while archius loads the config.properties file under the classpath by default. You can dynamically control the behavior of hystrix by adding the corresponding attribute key value to the configuration file. It is standard to use the configuration center for unified configuration management in distributed projects. Therefore, it is necessary to realize the dynamic configuration function of hystrix parameters based on the expansion of the configuration center.

By tracking the creation of hystrixcommand, it is found that hystrix finally obtains values according to the parameter property name through the implementation class of hystrixdynamicproperties, and hystrix itself provides the extension mechanism of hystrixdynamicproperties class. See 367 lines of code of hystrixplugins class. It can be seen that hystrix provides four extension methods:

  1. Through system parameters
  2. SPI mechanism based on Java
  3. Archaius dynamic attribute extension implementation class (default)
  4. Hystrix has built-in implementation of hystrixdynamicproperties based on system.getproperty;

2.8.1 SPI mechanism based on Java

The extension implementation based on SPI mechanism depends on two classes, namely, hystrixdynamicproperties and hystrixdynamicproperty. The hystrixdynamicproperties class is the hystrix dynamic property extension SPI interface to be implemented, which provides multiple methods to obtain dynamic properties. The interface definition is as follows:

public interface HystrixDynamicProperties {
    
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name, never <code>null</code>
     * @param fallback default value, maybe <code>null</code>
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<String> getString(String name, String fallback);
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name, never <code>null</code>
     * @param fallback default value, maybe <code>null</code>
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<Integer> getInteger(String name, Integer fallback);
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name, never <code>null</code>
     * @param fallback default value, maybe <code>null</code>
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<Long> getLong(String name, Long fallback);
    /**
     * Requests a property that may or may not actually exist.
     * @param name property name
     * @param fallback default value
     * @return never <code>null</code>
     */
    public HystrixDynamicProperty<Boolean> getBoolean(String name, Boolean fallback);
}


The hystrixdynamicproperty class specifically represents a parameter property and has the ability to change dynamically. The interface definition is as follows:

public interface HystrixDynamicProperty<T> extends HystrixProperty<T>{
    
    public String getName();
    
    /**
     * Register a callback to be run if the property is updated.
     * @param callback callback.
     */
    public void addCallback(Runnable callback);
    
}

The addcallback method is the core to realize the dynamic change of attributes. As explained in the note, it will register the callback callback method to refresh the attributes dynamically when the attributes are changed. The dynamic refresh logic has been implemented inside hystrix. We only need to save the callback when customizing the extension, and then trigger the callback method of the corresponding attribute object when the configuration center is changed.

The implementation steps are as follows:

1. Defines the hystrixdynamicproperty implementation class

Complete the custom implementation of dynamic attribute class, including string / integer / long / Boolean dynamic attribute state implementation.

As mentioned in the description of the hystrixdynamicproperty class above, you need to save the callback and trigger the callback method of these properties when receiving the property change of the configuration center to realize the dynamic change of the properties. This logic can be designed and implemented with reference to observer mode.

The code is as follows:

private abstract static class CustomDynamicProperty<T> implements HystrixDynamicProperty<T>, PropertyObserver {
        protected final String name;
        protected final T defaultValue;
        protected List<Runnable> callbacks;

        protected CustomDynamicProperty(String propName, T defaultValue) {
            this.name = propName;
            this.defaultValue = defaultValue;

            PropertyObserverManager.add(this);
        }

        @Override
        public String getName() {
            return name;
        }

        @Override
        public void addCallback(Runnable callback) {
            if (callbacks == null)
                callbacks = new ArrayList<>(1);
            this.callbacks.add(callback);
        }

        @Override
        public String keyName() {
            return name;
        }

        @Override
        public void update(PropertyItem item) {
            if(getName().equals(item.getName())) {
                for(Runnable r : callbacks) {
                    r.run();
                }
            }
        }
    }

    private static class StringDynamicProperty extends CustomDynamicProperty<String> {
        protected StringDynamicProperty(String propName, String defaultValue) {
            super(propName, defaultValue);
        }

        @Override
        public String get() {
            return ConfigManager.getString(name, defaultValue);
        }
    }

    private static class IntegerDynamicProperty extends CustomDynamicProperty<Integer> {
        protected IntegerDynamicProperty(String propName, Integer defaultValue) {
            super(propName, defaultValue);
        }

        @Override
        public Integer get() {
            String configValue =  ConfigManager.get(name);
            if(StringUtils.isNotEmpty(configValue)) {
                return Integer.valueOf(configValue);
            }
            return defaultValue;
        }
    }

    private static class LongDynamicProperty extends CustomDynamicProperty<Long> {
        protected LongDynamicProperty(String propName, Long defaultValue) {
            super(propName, defaultValue);
        }

        @Override
        public Long get() {
            String configValue =  ConfigManager.get(name);
            if(StringUtils.isNotEmpty(configValue)) {
                return Long.valueOf(configValue);
            }
            return defaultValue;
        }
    }

    private static class BooleanDynamicProperty extends CustomDynamicProperty<Boolean> {
        protected BooleanDynamicProperty(String propName, Boolean defaultValue) {
            super(propName, defaultValue);
        }

        @Override
        public Boolean get() {
            String configValue =  ConfigManager.get(name);
            if(StringUtils.isNotEmpty(configValue)) {
                return Boolean.valueOf(configValue);
            }
            return defaultValue;
        }
    }

The configmanager class temporarily defaults to the configuration management class of the configuration center, which provides functions such as parameter acquisition and parameter listener. The propertyobserver class (the keyname / update method belongs to its definition) and propertyobserver manager class are implemented with reference to the observer mode definition. They are responsible for observer registration and notification management to complete the linkage between dynamic properties and configuration center change notifications. The implementation of these two classes is relatively simple and do not show the description.

2. Defines the hystrixdynamicproperties implementation class

Customize the hystrixdynamicproperties based on the hystrixdynamicproperty extension class defined in step 1. The code is as follows:


public class DemoHystrixDynamicProperties implements HystrixDynamicProperties {
    @Override
    public HystrixDynamicProperty<String> getString(String name, String fallback) {
        return new StringDynamicProperty(name, fallback);
    }

    @Override
    public HystrixDynamicProperty<Integer> getInteger(String name, Integer fallback) {
        return new IntegerDynamicProperty(name, fallback);
    }

    @Override
    public HystrixDynamicProperty<Long> getLong(String name, Long fallback) {
        return new LongDynamicProperty(name, fallback);
    }

    @Override
    public HystrixDynamicProperty<Boolean> getBoolean(String name, Boolean fallback) {
        return new BooleanDynamicProperty(name, fallback);
    }
}

3. Register SPI implementation class

Add a text file named com.netflix.hystrix.strategy.properties.hystrixdynamicproperties in meta-inf / services / with the content of the full path name of the custom implementation class of hystrixdynamicproperties in step 2.

2.8.2 extension based on default archius

By default, hystrix realizes dynamic parameter acquisition through archius, and archius also provides customized parameter acquisition methods, including polledconfigurationsource interface and abstractpollingscheduler class. Polledconfigurationsource interface represents configuration acquisition source and abstractpollingscheduler class represents configuration timing refresh mechanism.

The implementation steps are as follows:

1. Create configuration acquisition source:

public class CustomCfgConfigurationSource implements PolledConfigurationSource {
    private final static String CONFIG_KEY_PREFIX = "hystrix";
 
    @Override
    public PollResult poll(boolean initial, Object checkPoint) throws Exception {
        Map<String, Object> map = load();
        return PollResult.createFull(map);
    }
 
    private Map<String, Object> load() throws Exception{
        Map<String, Object> map = new HashMap<>();
 
        Set<String> keys = ConfigManager.keys();
        for(String key : keys) {
            if(key.startsWith(CONFIG_KEY_PREFIX)) {
                map.put(key, ConfigManager.get(key));
            }
        }
 
        return map;
    }
}

The implementation is very simple. The core implementation is the poll method, which traverses all the configuration parameters starting with hystrix in the configuration center and returns to save.

2. Define the configuration refresh method:

public class CustomCfgPollingScheduler extends AbstractPollingScheduler {
    private final static Logger logger = LoggerFactory.getLogger("CustomCfgPollingScheduler");
 
    private final static String CONFIG_KEY_PREFIX = "hystrix";
 
    @Override
    public void startPolling(PolledConfigurationSource source, final Configuration config) {
        super.startPolling(source, config);
        //
        ConfigManager.addListener(new ConfigListener() {
            @Override
            public void eventReceived(PropertyItem item, ChangeEventType type) {
                String name = item.getName();
                if(name.startsWith(CONFIG_KEY_PREFIX)) {
                    String newValue = item.getValue();
                    //Add & modify
                    if(ChangeEventType.ITEM_ADDED.equals(type) || ChangeEventType.ITEM_UPDATED.equals(type)) {
                        addOrChangeProperty(name, newValue, config);
                    }
                    //Delete
                    else if(ChangeEventType.ITEM_REMOVED.equals(type)) {
                        deleteProperty(name, config);
                    }
                    else {
                        logger.error("error config change event type {}.", type);
                    }
                }
            }
        });
    }
 
    private void addOrChangeProperty(String name, Object newValue, final Configuration config) {
        if (!config.containsKey(name)) {
            config.addProperty(name, newValue);
        } else {
            Object oldValue = config.getProperty(name);
            if (newValue != null) {
                if (!newValue.equals(oldValue)) {
                    config.setProperty(name, newValue);
                }
            } else if (oldValue != null) {
                config.setProperty(name, null);
            }
        }
    }
 
    private void deleteProperty(String key, final Configuration config) {
        if (config.containsKey(key)) {
            config.clearProperty(key);
        }
    }
 
    @Override
    protected void schedule(Runnable pollingRunnable) {
        //IGNORE OPERATION
    }
 
    @Override
    public void stop() {
        //IGNORE OPERATION
    }
}

However, for actual projects, the method of regular refresh is not very real-time, and the other is to check whether the configuration center has been modified every time. The logic is complex, so we use configmanager.addlistener to increase the monitoring of the configuration center.

3. Define and initialize autoconfiguration:

DynamicConfiguration dynamicConfiguration = new DynamicConfiguration(new CustomCfgConfigurationSource(), new CustomCfgPollingScheduler());
ConfigurationManager.install(dynamicConfiguration);

Finally, you only need to execute the above initialization script when the container starts.

Careful students may find that in step 3 of the above steps, the dynamicconfiguration class is finally “installed” into the hystrix configuration management class, and the scheduled refresh class in step 2 is also weak. They think about whether they can continue to simplify the above scheme. They only need to implement a self-defined “dynamicconfiguration” to include the functions of configuration source acquisition and monitoring configuration modification, The implementation is as follows:

public class CustomCfgDynamicConfiguration extends ConcurrentMapConfiguration {
    private final static Logger logger = LoggerFactory.getLogger("CustomCfgDynamicConfiguration");
 
    private final static String CONFIG_KEY_PREFIX = "hystrix";
 
    public CustomCfgDynamicConfiguration() {
        super();
        load();
        initEvent();
    }
 
    /**
     *Load the full amount of hystrix configuration parameter information from the configuration center
     */
    private void load() {
        Set<String> keys = ConfigManager.keys();
        for(String key : keys) {
            if(key.startsWith(CONFIG_KEY_PREFIX)) {
                map.put(key, ConfigManager.get(key));
            }
        }
    }
 
    /**
     *Listen to the event callback processing through the configuration center and synchronize the changes of hystrix configuration parameters
     */
    private void initEvent() {
        ConfigManager.addListener(new ConfigListener() {
            @Override
            public void eventReceived(PropertyItem item, ChangeEventType type) {
                String name = item.getName();
                if(name.startsWith(CONFIG_KEY_PREFIX)) {
                    String newValue = item.getValue();
                    //Add & modify
                    if(ChangeEventType.ITEM_ADDED.equals(type) || ChangeEventType.ITEM_UPDATED.equals(type)) {
                        addOrChangeProperty(name, newValue);
                    }
                    //Delete
                    else if(ChangeEventType.ITEM_REMOVED.equals(type)) {
                        deleteProperty(name);
                    }
                    else {
                        logger.error("error config change event type {}.", type);
                    }
                }
            }
        });
    }
 
    /**
     *Add or modify parameter values
     * @param name
     * @param newValue
     */
    private void addOrChangeProperty(String name, Object newValue) {
        if (!this.containsKey(name)) {
            this.addProperty(name, newValue);
        } else {
            Object oldValue = this.getProperty(name);
            if (newValue != null) {
                if (!newValue.equals(oldValue)) {
                    this.setProperty(name, newValue);
                }
            } else if (oldValue != null) {
                this.setProperty(name, null);
            }
        }
    }
 
    /**
     *Delete parameter value
     * @param key
     */
    private void deleteProperty(String key) {
        if (this.containsKey(key)) {
            this.clearProperty(key);
        }
    }
}

Finally, through configurationmanager. Install (New customcfgdynamicconfiguration()); “Install” this implementation.

3、 Write at the end

The author summarizes and shares the use of hystrix in combination with the actual project. There are explanations on knowledge points such as isolation strategy, thread pool setting and parameter priority, as well as solutions to specific problems such as annotation superposition, exception handling and parameter dynamic configuration. I hope it will be helpful to you.

Author: vivo official website mall development team

Recommended Today

Swift advanced (XV) extension

The extension in swift is somewhat similar to the category in OC Extension can beenumeration、structural morphology、class、agreementAdd new features□ you can add methods, calculation attributes, subscripts, (convenient) initializers, nested types, protocols, etc What extensions can’t do:□ original functions cannot be overwritten□ you cannot add storage attributes or add attribute observers to existing attributes□ cannot add parent […]