• PHP test Kafka project example


    This article describes the PHP test Kafka project as an example. Share with you for your reference, as follows: summary Kafka was originally developed by LinkedIn. It is a distributed, partitioned, multi replica, multi subscriber distributed logging system based on zookeeper coordination (which can also be used as an MQ system). It can be commonly […]

  • Docker deployment Kafka and spring Kafka implementation


    This article mainly introduces the docker deployment of Kafka and the implementation of spring Kafka. The sample code is introduced in great detail, which has a certain reference learning value for everyone’s study or work. Friends who need it can refer to it from https://hub.docker.com/ Find Kafka The third one is active and has a […]

  • How to use Kafka in. Net core


    install CentOS install Kafka Kafka : http://kafka.apache.org/downloads ZooLeeper : https://zookeeper.apache.org/releases.html Download and unzip #Download and unzip $ wget https://archive.apache.org/dist/kafka/2.1.1/kafka_2.12-2.1.1.tgz $ tar -zxvf kafka_2.12-2.1.1.tgz $ mv kafka_2.12-2.1.1.tgz /data/kafka #Download zookeeper and unzip it $ wget https://mirror.bit.edu.cn/apache/zookeeper/zookeeper-3.5.8/apache-zookeeper-3.5.8-bin.tar.gz $ tar -zxvf apache-zookeeper-3.5.8-bin.tar.gz $ mv apache-zookeeper-3.5.8-bin /data/zookeeper Start zookeeper #Copy configuration template $ cd /data/kafka/conf $ cp zoo_sample.cfg zoo.cfg […]

  • Kafka pure Java integration code


    This article mainly introduces the spring pure Java configuration integration Kafka code example, the article through the example code introduction is very detailed, has the certain reference study value to everybody’s study or the work, needs the friend may refer to KafkaConfig.java package com.niugang.config; import java.util.HashMap; import java.util.Map; import org.apache.kafka.clients.consumer.ConsumerConfig; import org.apache.kafka.clients.consumer.OffsetAndMetadata; import org.apache.kafka.clients.consumer.OffsetCommitCallback; import […]

  • [big data practice] Kafka producer programming (2) – producer sending process


    Preface In the previous article [big data practice] Kafka producer programming (1) – Kafka producer detailed explanation, mainly forKafkaProducerThe functions in the class are explained in detail, but only for some of the methods. There is no in-depth explanation of the principle and mechanism behind producer. Therefore, in this article, we try to introduce the […]

  • Kafka principle and Practice


    This article starts with vivo Internet technology WeChat public number https://mp.weixin.qq.com/s/bV8AhqAjQp4a_iXRfobkCQ.About the author: Zheng Zhibin, graduated from computer science and Technology (bilingual class) of South China University of technology. Has been engaged in e-commerce, open platform, mobile browser, recommended advertising and big data, artificial intelligence and other related development and architecture. At present, I am […]

  • Open source Kafka enhancement: okmq-1.0.0


    The core idea of this tool is: gambling. Only when two basic components die at the same time will they receiveserious influence。 Oh, except for power failure. MQ is a good thing, we are all using it. This also determines that MQ should be highly available. Because of this component, a group has had several […]

  • K8s ecological weekly | etcd v3.4.0 brings many updates


    The content of k8s ecological weekly mainly includes some recommended information about k8s ecology that I have been exposed to every week. Welcome to the k8s ecology column. Helm 3 beta 2 release Helm 3 has been in continuous improvement since alpha. Finally, beta2 has been released. According to the current development progress of the […]

  • Big Data Learning Route


    I. Large Data Processing Flow The figure above is a simplified flow chart of large data processing. The main flow chart of large data processing includes data collection, data storage, data processing and data application. Next, we will explain the technology stack for each link one by one. 1.1 Data collection The first step in […]

  • Kafka Quick Start


    Reference website: http://kafka.apache.org/quick… I. Download Kafka Download address http://kafka.apache.org/downl… As of July 8, 2019, the latest version is The compiled version of scala 2.3.0 is the Kafka version. Scala 2.12  – kafka_2.12-2.3.0.tgz (asc, sha512) decompression > tar -xzf kafka_2.12-2.3.0.tgz > cd kafka_2.12-2.3.0 II. Start-up Services To start zookeeper kafka, you can either build one […]

  • [Spring cloud realizes advertising system step by step] 16. Incremental indexing and data delivery to MQ (kafka)


    Implementing Incremental Data Index In the previous section, we made sufficient preparations for the incremental index loading, using themysql-binlog-connector-javaOpen source components to achieve MySQL binlog monitoring, the relevant knowledge of binlog, you can consult on your own network. Or maybemailto:[email protected] In this section, we will implement incremental data processing according to binlog data objects. We […]

  • Kafka Two-Level Scheduling for Distributed Coordination of Microservice Task Assignment Golang Version


    background Two-level Coordination Scheduling Architecture Based on Kafka Message Queue In order to coordinate the work of internal consumer and Kafka connector, a replication protocol is implemented in Kafka. The main work is divided into two steps: Get metadata information such as topic offset by worker (consumer or connect) and give it to kafka‘s broker […]