• Kafka pure Java integration code


    This article mainly introduces the spring pure Java configuration integration Kafka code example, the article through the example code introduction is very detailed, has the certain reference study value to everybody’s study or the work, needs the friend may refer to KafkaConfig.java package com.niugang.config; import java.util.HashMap; import java.util.Map; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.OffsetAndMetadata; import org.apache.kafka.clients.consumer.OffsetCommitCallback; import […]

  • [big data practice] Kafka producer programming (2) – producer sending process


    Preface In the previous article [big data practice] Kafka producer programming (1) – Kafka producer detailed explanation, mainly forKafkaProducerThe functions in the class are explained in detail, but only for some of the methods. There is no in-depth explanation of the principle and mechanism behind producer. Therefore, in this article, we try to introduce the […]

  • Kafka principle and Practice


    This article starts with vivo Internet technology WeChat public number https://mp.weixin.qq.com/s/bV8AhqAjQp4a_iXRfobkCQ.About the author: Zheng Zhibin, graduated from computer science and Technology (bilingual class) of South China University of technology. Has been engaged in e-commerce, open platform, mobile browser, recommended advertising and big data, artificial intelligence and other related development and architecture. At present, I am […]

  • Open source Kafka enhancement: okmq-1.0.0


    The core idea of this tool is: gambling. Only when two basic components die at the same time will they receiveserious influence。 Oh, except for power failure. MQ is a good thing, we are all using it. This also determines that MQ should be highly available. Because of this component, a group has had several […]

  • K8s ecological weekly | etcd v3.4.0 brings many updates


    The content of k8s ecological weekly mainly includes some recommended information about k8s ecology that I have been exposed to every week. Welcome to the k8s ecology column. Helm 3 beta 2 release Helm 3 has been in continuous improvement since alpha. Finally, beta2 has been released. According to the current development progress of the […]

  • Big Data Learning Route


    I. Large Data Processing Flow The figure above is a simplified flow chart of large data processing. The main flow chart of large data processing includes data collection, data storage, data processing and data application. Next, we will explain the technology stack for each link one by one. 1.1 Data collection The first step in […]

  • Kafka Quick Start


    Reference website: http://kafka.apache.org/quick… I. Download Kafka Download address http://kafka.apache.org/downl… As of July 8, 2019, the latest version is The compiled version of scala 2.3.0 is the Kafka version. Scala 2.12  – kafka_2.12-2.3.0.tgz (asc, sha512) decompression > tar -xzf kafka_2.12-2.3.0.tgz > cd kafka_2.12-2.3.0 II. Start-up Services To start zookeeper kafka, you can either build one […]

  • [Spring cloud realizes advertising system step by step] 16. Incremental indexing and data delivery to MQ (kafka)


    Implementing Incremental Data Index In the previous section, we made sufficient preparations for the incremental index loading, using themysql-binlog-connector-javaOpen source components to achieve MySQL binlog monitoring, the relevant knowledge of binlog, you can consult on your own network. Or maybemailto:[email protected] In this section, we will implement incremental data processing according to binlog data objects. We […]

  • Kafka Two-Level Scheduling for Distributed Coordination of Microservice Task Assignment Golang Version


    background Two-level Coordination Scheduling Architecture Based on Kafka Message Queue In order to coordinate the work of internal consumer and Kafka connector, a replication protocol is implemented in Kafka. The main work is divided into two steps: Get metadata information such as topic offset by worker (consumer or connect) and give it to kafka‘s broker […]

  • Data Pipeline | Apache Kafka Actual Warfare Author Hu Xi: Apache Kafka Monitoring and Tuning


    Hu Xi, author of Apache Kafka Actual Warfare, graduated from Beihang Computer Master’s degree. He is currently the director of computing platform of a mutual fund company. He has worked in IBM, Sogou, Weibo and other companies. Active Kafka Code Contributor in China. PrefaceAlthough Apache Kafka has evolved into a streaming platform, most users still […]

  • [kafka KSQL] Game Log Statistical Analysis (1)


    [kafka KSQL] Game Log Statistical Analysis (1) Taking game settlement log as an example, this paper shows the process of statistical analysis of log using KSQL. Start confluent cd ~/Documents/install/confluent-5.0.1/ bin/confluent start View the list of Kafka topics bin/kafka-topics –list –zookeeper localhost:2181 Create a topic that accepts the game settlement log bin/kafka-topics –create –zookeeper localhost:2181 […]

  • An Application Example of Kafka Cluster Golang


    Item see: (kafka cluster example) What does this example do? A docker cluster service with 3 nodes Kafka and 3 nodes zookeeper is built. The docker applications of one message publisher and two message subscribers of the same consumer group are created respectively. UseabConduct concurrent testing to verify the subscription/publishing function of the instance message. […]