Tag:kafka

  • [big data practice] Kafka producer programming (2) – producer sending process

    Time:2019-12-14

    Preface In the previous article [big data practice] Kafka producer programming (1) – Kafka producer detailed explanation, mainly forKafkaProducerThe functions in the class are explained in detail, but only for some of the methods. There is no in-depth explanation of the principle and mechanism behind producer. Therefore, in this article, we try to introduce the […]

  • Kafka principle and Practice

    Time:2019-12-2

    This article starts with vivo Internet technology WeChat public number https://mp.weixin.qq.com/s/bV8AhqAjQp4a_iXRfobkCQ.About the author: Zheng Zhibin, graduated from computer science and Technology (bilingual class) of South China University of technology. Has been engaged in e-commerce, open platform, mobile browser, recommended advertising and big data, artificial intelligence and other related development and architecture. At present, I am […]

  • Open source Kafka enhancement: okmq-1.0.0

    Time:2019-11-16

    The core idea of this tool is: gambling. Only when two basic components die at the same time will they receiveserious influence。 Oh, except for power failure. MQ is a good thing, we are all using it. This also determines that MQ should be highly available. Because of this component, a group has had several […]

  • K8s ecological weekly | etcd v3.4.0 brings many updates

    Time:2019-10-20

    The content of k8s ecological weekly mainly includes some recommended information about k8s ecology that I have been exposed to every week. Welcome to the k8s ecology column. Helm 3 beta 2 release Helm 3 has been in continuous improvement since alpha. Finally, beta2 has been released. According to the current development progress of the […]

  • Big Data Learning Route

    Time:2019-9-23

    I. Large Data Processing Flow The figure above is a simplified flow chart of large data processing. The main flow chart of large data processing includes data collection, data storage, data processing and data application. Next, we will explain the technology stack for each link one by one. 1.1 Data collection The first step in […]

  • Kafka Quick Start

    Time:2019-9-19

    Reference website: http://kafka.apache.org/quick… I. Download Kafka Download address http://kafka.apache.org/downl… As of July 8, 2019, the latest version is 2.3.02.12. The compiled version of scala 2.3.0 is the Kafka version. Scala 2.12  – kafka_2.12-2.3.0.tgz (asc, sha512) decompression > tar -xzf kafka_2.12-2.3.0.tgz > cd kafka_2.12-2.3.0 II. Start-up Services To start zookeeper kafka, you can either build one […]

  • [Spring cloud realizes advertising system step by step] 16. Incremental indexing and data delivery to MQ (kafka)

    Time:2019-9-1

    Implementing Incremental Data Index In the previous section, we made sufficient preparations for the incremental index loading, using themysql-binlog-connector-javaOpen source components to achieve MySQL binlog monitoring, the relevant knowledge of binlog, you can consult on your own network. Or maybemailto:[email protected] In this section, we will implement incremental data processing according to binlog data objects. We […]

  • Kafka Two-Level Scheduling for Distributed Coordination of Microservice Task Assignment Golang Version

    Time:2019-7-31

    background Two-level Coordination Scheduling Architecture Based on Kafka Message Queue In order to coordinate the work of internal consumer and Kafka connector, a replication protocol is implemented in Kafka. The main work is divided into two steps: Get metadata information such as topic offset by worker (consumer or connect) and give it to kafka‘s broker […]

  • Data Pipeline | Apache Kafka Actual Warfare Author Hu Xi: Apache Kafka Monitoring and Tuning

    Time:2019-7-2

    Hu Xi, author of Apache Kafka Actual Warfare, graduated from Beihang Computer Master’s degree. He is currently the director of computing platform of a mutual fund company. He has worked in IBM, Sogou, Weibo and other companies. Active Kafka Code Contributor in China. PrefaceAlthough Apache Kafka has evolved into a streaming platform, most users still […]

  • [kafka KSQL] Game Log Statistical Analysis (1)

    Time:2019-6-19

    [kafka KSQL] Game Log Statistical Analysis (1) Taking game settlement log as an example, this paper shows the process of statistical analysis of log using KSQL. Start confluent cd ~/Documents/install/confluent-5.0.1/ bin/confluent start View the list of Kafka topics bin/kafka-topics –list –zookeeper localhost:2181 Create a topic that accepts the game settlement log bin/kafka-topics –create –zookeeper localhost:2181 […]

  • An Application Example of Kafka Cluster Golang

    Time:2019-5-28

    Item see: (kafka cluster example) What does this example do? A docker cluster service with 3 nodes Kafka and 3 nodes zookeeper is built. The docker applications of one message publisher and two message subscribers of the same consumer group are created respectively. UseabConduct concurrent testing to verify the subscription/publishing function of the instance message. […]

  • “From 0 to 1 Learn Flink” – Flink reads Kafka data and writes it to MySQL in batches

    Time:2019-5-26

    <!– more –> Preface Previously, I was actually learning Flink from 0 to 1 – How to customize Data Sink? In fact, the article has written a point to write data to MySQL, but some configurations were written to death, not universal. Recently, a friend of the knowledge planet called me: write an example of […]