Kafka – PHP uses rdkafka production / consumption data

Time:2020-2-24

Kafka cluster deployment

Install rdkafka

rdkafkarely onlibkafka

yum install rdkafka rdkafka-devel
pecl install rdkafka
php --ri rdkafka

Http://pecl.php.net/package/r… See the supportedkafkaClient version

Producer

Connecting clusters, creatingtopic, production data.

<?php
$rk = new Rdkafka\Producer();
$rk->setLogLevel(LOG_DEBUG);

//Link Kafka cluster
$rk->addBrokers("192.168.20.6:9092,192.168.20.6:9093");

//Create topic
$topic = $rk->newTopic("topic_1");

while (true) {
    $message = "hello kafka " . date("Y-m-d H:i:s");
    echo "hello kafka " . date("Y-m-d H:i:s") . PHP_EOL;

    try {
        $topic->produce(RD_KAFKA_PARTITION_UA, 0, $message);
        sleep(2);
    } catch (\Exception $e) {
        echo $e->getMessage() . PHP_EOL;
    }
}

Consumer – highlevel

Automatic allocationpartitionrebalancecomsumer group

<?php
$conf = new RdKafka\Conf();

// Set a rebalance callback to log partition assignments (optional)
$conf->setRebalanceCb(function (RdKafka\KafkaConsumer $kafka, $err, array $partitions = null) {
    switch ($err) {
        case RD_KAFKA_RESP_ERR__ASSIGN_PARTITIONS:
            echo "Assign: ";
            var_dump($partitions);
            $kafka->assign($partitions);
            break;
        case RD_KAFKA_RESP_ERR__REVOKE_PARTITIONS:
            echo "Revoke: ";
            var_dump($partitions);
            $kafka->assign(null);
            break;
        default:
            throw new \Exception($err);
    }
});

// Configure the group.id. All consumer with the same group.id will consume
// different partitions.
$conf->set('group.id', 'group_1');

// Initial list of Kafka brokers
$conf->set('metadata.broker.list', '192.168.20.6:9092,192.168.20.6:9093');

$topicConf = new RdKafka\TopicConf();

// Set where to start consuming messages when there is no initial offset in
// offset store or the desired offset is out of range.
// 'smallest': start from the beginning
$topicConf->set('auto.offset.reset', 'smallest');

// Set the configuration to use for subscribed/assigned topics
$conf->setDefaultTopicConf($topicConf);

$consumer = new RdKafka\KafkaConsumer($conf);

// Subscribe to topic 'topic_1'
$consumer->subscribe(['topic_1']);

echo "Waiting for partition assignment... (make take some time when\n";
echo "quickly re-joining the group after leaving it.)\n";

while (true) {
    $message = $consumer->consume(3e3);
    switch ($message->err) {
        case RD_KAFKA_RESP_ERR_NO_ERROR:
            var_dump($message);
            break;
        case RD_KAFKA_RESP_ERR__PARTITION_EOF:
            sleep(2);
        case RD_KAFKA_RESP_ERR__TIMED_OUT:
            echo $message->errstr() . PHP_EOL;
            break;
        default:
            throw new \Exception($message->errstr(), $message->err);
            break;
    }
}

Consumer – lowlevel

AppointpartitionConsumption.
php consumer_lowlevel.php [partitonNuo]
LowLevelWithout the concept of consumption group, we can also think that every consumer belongs to an independent consumption group.

<?php
if (!isset($argv[1])) {
    Fwrite (stdout, "please specify consumption partition:");
    $partition = (int) fread(STDIN, 1024);
} else {
    $partition = (int) $argv[1];
}

$topic = "topic_1";

$conf = new RdKafka\Conf();

// Set the group id. This is required when storing offsets on the broker
$conf->set('group.id', 'group_2');

$rk = new RdKafka\Consumer($conf);
$rk->addBrokers('192.168.20.6:9092,192.168.20.6:9093');

$topicConf = new RdKafka\TopicConf();
$topicConf->set('auto.commit.interval.ms', 2000);

// Set the offset store method to 'file'
// $topicConf->set('offset.store.method', 'file');
// $topicConf->set('offset.store.path', sys_get_temp_dir());

// Alternatively, set the offset store method to 'broker'
$topicConf->set('offset.store.method', 'broker');

// Set where to start consuming messages when there is no initial offset in
// offset store or the desired offset is out of range.
// 'smallest': start from the beginning
$topicConf->set('auto.offset.reset', 'smallest');

$topic = $rk->newTopic($topic, $topicConf);

// Start consuming partition 0
$topic->consumeStart($partition, RD_KAFKA_OFFSET_STORED);

while (true) {
    $message = $topic->consume($partition, 3 * 1000);
    switch ($message->err) {
        case RD_KAFKA_RESP_ERR_NO_ERROR:
            var_dump($message);
            break;
        case RD_KAFKA_RESP_ERR__PARTITION_EOF:
        case RD_KAFKA_RESP_ERR__TIMED_OUT:
            echo $message->errstr() . PHP_EOL;
            break;
        default:
            throw new \Exception($message->errstr(), $message->err);
            break;
    }
}

Recommended Today

Usage scenario of Vue calculation attribute computed and listener Watch

Original link: HTTPS: / / dsx2016. COM /? P = 679WeChat official account: Senior brother 2016 Features and differences vueOfcomputedOption is mainly used to synchronize data processing, whilewatchOption is mainly used for event dispatch, asynchronous Both of them can achieve the same effect, but based on their respective characteristics, there will be some differences in […]