On the use of Kafka


Take Python as an example, from pykafka to Kafka Python to confluent Kafka to Flink

Note that it is best to use the latest version
The performance of pykafka and Kafka Python is not very good. You can also find out the specific situation

Coufluent Kafka is a python module and a lightweight package of librdkafka. Librdkafka is also a Kafka Library Based on C / C + +. There is no need to say more about its performance. It is better than Kafka Python in use. It also supports asynchrony and has an event round robin mechanism, so there seems to be a problem when used in combination with tornado, because tornado is a single process with Gil lock. You can see this:

Some lower versions will prompt “confused_kafka / SRC / confused_kafka. H: 22:32: fatal error: librdkafka / rdkafka. H: no such file or directory” after installation,
You can apt get install librdkafka dev – Y

In addition, the first mock exam for producer and consumer is the same version. Recently, the old project Kafka shows that CreateTime is equal to -1.

So it’s best to use Flink in the project