How to prefix topic in batch in spring Kafka

Time:2021-4-17

preface

Recently, the business development department has put forward a requirement to our department, because their development environment and test environment share a set of Kafka. They hope our department can help them automatically add environment prefix to the topic of Kafka. For example, the topic of development environment is dev_ Topic, test environment, then topic is test_ Their Kafka client uses spring Kafka. At the beginning, when I received this requirement, I refused in my heart. Why not deploy a Kafka for the development environment and the test environment respectively, but it’s so troublesome. But Dadu agreed to meet this demand. As a little Luo, he can only meet it

Realization idea

1. Producer side

You can prefix topics with producer interceptors

2. Implementation steps

a. Write a producer interceptor

@Slf4j
public class KafkaProducerInterceptor implements ProducerInterceptor<String, MessageDTO> {



    /**
     * runs in the user main thread and calls before the message is serialized.
     * @param record
     * @return
     */
    @Override
    public ProducerRecord<String, MessageDTO> onSend(ProducerRecord<String, MessageDTO> record) {
        log.info (original topic: {}), record.topic ());
        return new ProducerRecord<String, MessageDTO>(TOPIC_KEY_PREFIX + record.topic(),
                record.partition(),record.timestamp(),record.key(), record.value());
    }




    /**
     *It is called before the message is answered or when the message fails to be sent. It usually runs in the IO thread of the producer before the producer callback logic is triggered
     * @param metadata
     * @param exception
     */
    @Override
    public void onAcknowledgement(RecordMetadata metadata, Exception exception) {
      log.info (actual topic: {}), metadata.topic ());
    }


    /**
     *Clean up work
     */
    @Override
    public void close() {
    }


    /**
     *Initialization work
     * @param configs
     */
    @Override
    public void configure(Map<String, ?> configs) {

    }

b. Configure interceptors

kafka:
    producer:
      #Producer interceptor configuration
      properties:
        interceptor.classes: com.github.lybgeek.kafka.producer.interceptor.KafkaProducerInterceptor

c. Testing

How to prefix topic in batch in spring Kafka

2. Consumer side

This is a little difficult because the business development department uses @ kafkalistener’s annotation directly, as follows

 @KafkaListener(id = "msgId",topics = {Constant.TOPIC})

There is no good way like this, only through the source code, through the source code can be found in the following places

KafkaListenerAnnotationBeanPostProcessor#postProcessAfterInitialization

You will assign the value of @ kafkalistener to the consumer. If you have friends who know about spring, you may know that postprocessafterinitialization is a method of spring postprocessor, which is mainly used for some operations after bean initialization. Since we know that @ kafkalistener will assign the value after bean initialization, we can modify the value of @ kafkalistener before bean initialization Value. The specific implementation is as follows

@Component
public class KafkaListenerFactoryBeanPostProcesser implements BeanFactoryPostProcessor {

    @SneakyThrows
    @Override
    public void postProcessBeanFactory(ConfigurableListableBeanFactory beanFactory) throws BeansException {

        List<String> packageNames = AutoConfigurationPackages.get(beanFactory);

        for (String packageName : packageNames) {
            Reflections reflections = new Reflections(new ConfigurationBuilder()
                    . forpackages (packagename) // specify the path URL
                    . addscanners (New subtypesscanner()) // add subclass scanner
                    . addscanners (New fieldannotations scanner()) // add attribute annotation scanner
                    . addscanners (new method annotations scanner()) // add method annotation scanner
                    . addscanners (New methodparameterscanner()) // add method parameter scanner
            );

            Set<Method> methodSet = reflections.getMethodsAnnotatedWith(KafkaListener.class);
            if(!CollectionUtils.isEmpty(methodSet)){
                for (Method method : methodSet) {
                    KafkaListener kafkaListener = method.getAnnotation(KafkaListener.class);
                    changeTopics(kafkaListener);
                }
            }
        }

    }


    private void changeTopics(KafkaListener kafkaListener) throws Exception{
        InvocationHandler invocationHandler = Proxy.getInvocationHandler(kafkaListener);
        Field memberValuesField = invocationHandler.getClass().getDeclaredField("memberValues");
        memberValuesField.setAccessible(true);
        Map<String,Object> memberValues = (Map<String,Object>)memberValuesField.get(invocationHandler);
        String[] topics = (String[])memberValues.get("topics");
        System.out.println (topics before modification:+ Lists.newArrayList (topics));
        for (int i = 0; i < topics.length; i++) {
            topics[i] = Constant.TOPIC_KEY_PREFIX + topics[i];
        }
        memberValues.put("topics", topics);
        System.out.println (modified topics:+ Lists.newArrayList ( kafkaListener.topics ()));

    }
}

test

How to prefix topic in batch in spring Kafka
How to prefix topic in batch in spring Kafka

summary

Although the topic has been dynamically modified, I still think that the topic should not be changed casually. If conditions permit, Kafka should be isolated based on the physical environment. Secondly, the objective conditions really do not allow it. To change the topic dynamically, we need to do a good job in the propaganda of topic dynamic change and the compilation of related wikis, otherwise it is easy to fall into the pit

Demo link

https://github.com/lyb-geek/springboot-learning/tree/master/springboot-mq-idempotent-consume