Tag:Consumer

  • Request with content type of application / JSON; charset = UTF-8 initiated by Lua

    Time:2020-4-4

    Preface Lua is a lightweight and compact scripting language, written in standard C language, and open source. It is designed to be embedded in the application so as to provide flexible extension and customization functions for the application. Lua was developed in 1993 by a research group at the Catholic University of Rio de Janeiro, […]

  • Kafka API use

    Time:2020-4-4

    1. Producer API Introducing dependency <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-clients</artifactId> < version > 0.11.0.0 < / version > // version 0.11.0.0 </dependency> The configuration items of the producer are described in the producerconfig class, and each configuration has a corresponding doc description. The producer uses the API with the callback function demo, and also runs in blocking […]

  • Kafka Series 1: overview of Kafka

    Time:2020-3-31

    Kafka Series 1: overview of Kafka Kafka is one of the most popular message middleware in the current distributed system. With its high throughput design, Kafka is loved by developers in the application scenarios of log collection system and message system.This article talks about some knowledge about Kafka. It mainly includes the following contents: Kafka […]

  • Simple and clear producer consumer model

    Time:2020-3-28

    First of all, it needs to be controlled by three quantities Ini: // initialization Mutex = 1 // mutex, only one read at a time Empty = n // existing empty digits Full = 0 // on hand inventory The separation of empty and full reflects the interaction between producer and consumer behavior. producer: Wait […]

  • Docker: Kafka cluster

    Time:2020-3-27

    1. overview Apache Kafka is a fast, scalable, high throughput, fault-tolerant distributed publish subscribe message system. It has the characteristics of high throughput, built-in partition, supporting data copy and fault tolerance, and is suitable for large-scale message processing scenarios. The author previously worked in the Internet of things company, where Kafka is the factual standard […]

  • On Kafka’s consumption rebalancing from a production error

    Time:2020-3-9

    Problem description The production error log of the project is as follows, [commitSync] processed message to kafka failed, Just Ignore this commit, wait for next commit to make these messages processed.org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent […]

  • Study Kafka course (1)

    Time:2020-3-5

    brief introduction Kafka is used to build real-time data pipeline and flow applications. It has the advantages of horizontal expansion, fault tolerance, wicked fast, and has run in thousands of companies. target Understand the basic principles of Kafka Master the basic operation of Kafka Kafka’s in-depth exploration is in another article. Related concepts Producer: the […]

  • Do you know what performance strategy Kafka uses

    Time:2020-3-4

    This article is my experience in Kafka research. Welcome to point out the mistakesVisit my blog more Preface As an increasingly popular stream processing platform, Kafka makes people wonder why it is so popular and famous. I explore the mystery of Kafka from the perspective of performance. Define Kafka’s performance position First of all, clear […]

  • Kafka introduction

    Time:2020-2-26

    In the previous article, I introduced the use scenario of message queuing, now I will introduce Kafka Main features of Kafka: Provides high throughput for both publishing and subscription. It is understood that Kafka can produce about 250000 messages (50 MB) per second and process 550000 messages (110 MB) per second. Persistent operation is available. […]

  • Apache Kafka Quick Start Guide

    Time:2020-2-25

    brief introduction Kafka is a message system based on publish and subscribe. Originally originated in LinkedIn, it became an open source Apache project in 2011, and then became the top Apache project in 2012. Kafka is written in scala and Java, and is widely used for its distributed and extensible architecture, persistence and high throughput. […]

  • Kafka – PHP uses rdkafka production / consumption data

    Time:2020-2-24

    Kafka cluster deployment Install rdkafka rdkafkarely onlibkafka yum install rdkafka rdkafka-devel pecl install rdkafka php –ri rdkafka Http://pecl.php.net/package/r… See the supportedkafkaClient version Producer Connecting clusters, creatingtopic, production data. <?php $rk = new Rdkafka\Producer(); $rk->setLogLevel(LOG_DEBUG); //Link Kafka cluster $rk->addBrokers(“192.168.20.6:9092,192.168.20.6:9093”); //Create topic $topic = $rk->newTopic(“topic_1”); while (true) { $message = “hello kafka ” . date(“Y-m-d H:i:s”); echo […]

  • Using multithreading to increase Kafka consumption

    Time:2020-2-23

    Premise: this example is suitable for message topics without sequence requirements. Through a series of optimizations, Kafka can achieve tens of thousands of writes and reads per second. By increasing the number of partitions, we can increase the parallel consumption ability by deploying multiple consumers. But in many cases, the execution speed of some businesses […]