Tag:streaming

  • Define the processor of Kafka streams

    Time:2021-7-23

    order This paper analyzes the kstreambuilder of Kafka streams and gives an example of how to customize the processor of Kafka streams example KStreamBuilder builder = new KStreamBuilder(); KStream<String, String> source = builder.stream(“demo-topic”); KafkaStreams streams = new KafkaStreams(builder, props); streams.start(); There is topology hidden in kstreambuilder KStreamBuilder kafka-streams-0.10.2.1-sources.jar!/org/apache/kafka/streams/kstream/KStreamBuilder.java public class KStreamBuilder extends TopologyBuilder { public […]

  • Practice of Flink 1.11 and hive batch flow integrated data warehouse

    Time:2021-6-13

    Guide: Flink has provided the function of integrating with hive since 1.9.0. With the iteration of several versions, the function of integrating with hive has been further deepened in the latest Flink 1.11, and it has begun to try to integrate the streaming computing scene with hive. This paper mainly shares the new features of […]

  • Application of Flink real time computing in microblog

    Time:2021-5-29

    Introduction:By combining the Flink real-time flow computing framework with business scenarios, microblog has done a lot of work in terms of platform and service, and has also done a lot of optimization in terms of development efficiency and stability. We improve development efficiency through modular design and flat platform development. Cao Fuqiang, senior system engineer […]

  • Flink boosts meituan’s incremental production

    Time:2021-5-6

    This paper is shared by Ju Dasheng, a researcher of meituan and the person in charge of real-time computing. It mainly introduces the application practice of Flink in the incremental production of meituan data warehouse. The contents include: Data warehouse incremental production Streaming data integration Streaming data processing Application of streaming OLAP Future planning 1、 […]

  • Go language HTTP request stream writing body

    Time:2021-5-5

    background Recently, when developing a function, we need to report a lot of log content through HTTP protocol, but the API of HTTP client in go standard library is as follows: http.NewRequest(method, url string, body io.Reader) Body is throughio.ReaderInterface to pass, and does not expose oneio.WriterInterface to provide a way to write. First, let’s see […]

  • Excel 2.4.0 official release, add 152 formula function support

    Time:2021-5-2

    Excel is a basic library for operating Office Excel documents written in go language, which is based on ecma-376 and ISO / IEC 29500 international standards. You can use it to read and write from Microsoft Excel ™ 2007 and above. It supports multiple document formats such as xlsx / xlsm / xltm, and is […]

  • Download the TS video of online class

    Time:2021-4-30

    Originally prepared to use crawler, open multithreading, to download a TS segment, and finally use ffmpeg synthesis complete TS video. from concurrent.futures import ThreadPoolExecutor import requests import logging import re import os url = ‘http://v3.julyedu.com/video/259/6390/01a311da6a2cd91-‘ def download(name): str_name = “%05d” % name print(str_name) file_name = str_name + ‘.ts’ print(url + file_name) try: res = requests.get(url=url […]

  • The application of new generation streaming computing framework in financial industry

    Time:2021-4-14

    In the era of big data, data computing has penetrated into all walks of life. Business precipitation data, data computing generates new business value, data computing is constantly using this way to promote business development. The computing modes of big data are mainly divided into batch computing and stream computing, which are suitable for different […]

  • Are you still using pagination? Try mybatis streaming query, really powerful!

    Time:2021-3-25

    Basic concepts Streaming query means that after a successful query, it does not return a collection, but an iterator. The application gets a query result from the iterator each time. The advantage of streaming query is that it can reduce memory usage. If there is no streaming query, if we want to get 10 million […]

  • Kafka actual combat: (4) detailed explanation of Kafka stream

    Time:2021-3-23

    1、 Background of Kafka stream 1. Introduction to Kafka stream It provides the function of stream processing and analyzing the tree stored in Kafka Kafka stream features: Kafka stream provides a very simple and lightweight library, which can be easily embedded in any Java application, or packaged and deployed in any way There are no […]

  • Kafka stream introduction and basic application (including case)

    Time:2021-3-20

    catalog 1. Background of Kafka stream 1.1 what is Kafka stream 1.2 what is flow computing 1.3 why Kafka stream 2. How Kafka stream solves key problems in streaming system 2.1 ktable and ksteam 2.2 time 2.3 window 3. Kafka stream application example 3.1 case 1: write the data of topic a to topic B […]

  • Kwai based optimization based on Apache Flink

    Time:2021-3-20

    This is Kwai Tai teacher Liu Jiangang, which is divided into three parts. First, the basic concepts of streaming computing are introduced, then the key technologies of Flink are introduced. Finally, some applications of Flink in Kwai do production practice are introduced, including real-time index calculation and fast failover. 1、 Introduction of flow calculation Stream […]