Tag:kafka

  • Only one instance of Spring Cloud Stream multi-consumer is consuming

    Time:2022-11-28

    environment and phenomena The technology stack of spring cloud stream + kafka is used. # kafka connection configuration spring.cloud.stream.binders.common.type=kafka spring.cloud.stream.binders.common.environment.spring.cloud.stream.kafka.binder.brokers=192.168.110.110:9092 spring.cloud.stream.binders.common.environment.spring.cloud.stream.kafka.binder.zk-nodes=192.168.110.110:2181 #Consumer configuration spring.cloud.stream.bindings.input.binder=common spring.cloud.stream.bindings.input.destination=test-issue spring.cloud.stream.bindings.input.group=group0 spring.cloud.stream.bindings.input.content-type=text/plain # kafka connection configuration spring.cloud.stream.binders.common.type=kafka spring.cloud.stream.binders.common.environment.spring.cloud.stream.kafka.binder.brokers=192.168.110.110:9092 spring.cloud.stream.binders.common.environment.spring.cloud.stream.kafka.binder.zk-nodes=192.168.110.110:2181 # Production side configuration spring.cloud.stream.bindings.output.binder=common spring.cloud.stream.bindings.output.destination=test-issue spring.cloud.stream.bindings.output.content-type=text/plain Currently, Kafka is deployed on a single machine. Topic is automatically created by stream, […]

  • Proxy type upgrade, APISIX supports Kafka as upstream

    Time:2022-11-27

    Apache Kafka is an open source stream processing platform. It is a high-throughput distributed publish-subscribe message system that can process all action stream data of consumers in the website. All its functions are distributed, highly scalable, elastic , fault-tolerant and secure manner. A new feature was released in Apache APISIX 2.14, that is, to provide […]

  • Kafka log and consumer offset expiration test

    Time:2022-11-27

    consumer consumption The consumer will record two offsets locally:• Consume the fetched offset, it will be initialized and rebalancedfetchInitialOffsetGet the offset of the last consumption from the broker and then pull the message from the specified offset. Next, the Consumer will continue to fetch messages from the specified offset through fetchNewMessages() to the broker.• Consume […]

  • [Zhao Yuqiang] The new book “Big Data Principles and Practical Combat” is on the market! ! !

    Time:2022-11-27

    After nearly a year of waiting, the new book “Big Data Principles and Practice” is on the market! ! Sneak peek! ! ! Click here for details ​ ​ Click here for details ​ Click here for details Click here for details ​ Click here for details ​ Click here for details Click here for […]

  • python operation docker kafka

    Time:2022-11-26

    this machineIPYes10.30.6.24, the following configuration process needs to be based on your ownIPInformation configuration modification kafkause by default127.0.0.1access configurationcompose.yamlThe file is as follows services: zookeeper: image: zookeeper container_name: demo-zookeeper ports: – “2181:2181” restart: always kafka: image: wurstmeister/kafka container_name: demo-kafka ports: – “9092:9092” ulimits: nofile: soft: 262144 hard: 262144 environment: DOCKER_API_VERSION: 1.41 KAFKA_ADVERTISED_LISTENERS: “PLAINTEXT://10.30.6.24:9092” KAFKA_LISTENERS: “PLAINTEXT://0.0.0.0:9092” […]

  • Analysis of front-end log collection scheme

    Time:2022-11-25

    foreword In the front-end deployment process, nginx is usually used as the deployment server, and for the default nginx service, it provides corresponding log records, which can be used to record related logs of server access, for system stability and robustness monitoring In other words, log collection and analysis can provide more quantitative index construction. […]

  • A weird Kafka message OOM problem, take the opportunity to pick up StringBuilder

    Time:2022-11-24

    1 Introduction The origin is an OOM in the online environment of a certain customer. It is said that this kind of weird OOM has happened for the second time. First describe the relevant information of OOM: Heap memory 13G; Use the G1 garbage collector; The automatically generated dump file after OOM is 7G; The […]

  • Alibaba Cloud Message Queue Kafka-Message Retrieval Practice

    Time:2022-11-23

    author: Kafka&Tablestore This article mainly introduces the troubleshooting methods for pain points such as message loss and repeated consumption encountered in the use of message queues, as well as the scenario practice of message queue Kafka “retrieval component”, and interprets its key technologies. The purpose is to help you become more familiar with the characteristics […]

  • One of kafka – log files

    Time:2022-11-22

    There are two main parts of this article: segment file in patition how to find a message First look at a few basic concepts: partition: topic physical grouping, there can be multiple partitions under a topic, and each partition is ordered segment: Each patition consists of multiple segment files offset: Each message in the partition […]

  • Redis visual management tool GUI client recommendation

    Time:2022-11-21

    Features status monitoring: The dashboard function provided allows you to monitor the status of the Redis service in real time. Redis friendly: Redis Assistant will not block the server when processing millions of keys and fields. data formatting: Redis Assistant will automatically recognize and format different data formats, including Text, JSON, XML, HEX, MessagePack, etc. […]

  • A picture advanced RocketMQ – message storage

    Time:2022-11-20

    foreword The third gentleman has read several books, and read a picture compiled by the source code many times. Advanced RocketMQ picture, you only need to remember this picture about RocketMQ! If you think it is good, remember to like and follow.[Important] The video is updated synchronously on station B. Welcome to watch and raise […]

  • Kafka II – log management

    Time:2022-11-19

    The main contents of this article are: Log management related implementation classes Log flushing log cleanup checkpoint Log management related implementation classes Kafka’s log management is implemented through the LogManager class, and its main function is to create, retrieve, and clean up logs. KafkaServer starts the LogManager thread The LogManager thread is started when the […]