Tag:Flow type

  • Flink helps meituan’s warehouse incremental production

    Time:2021-9-4

    Introduction: This article is shared by Ju Dasheng, a researcher of meituan and head of real-time computing. It mainly introduces the application practice of Flink in helping meituan’s digital warehouse incremental production. The contents include: 1. Digital warehouse incremental production; 2. Streaming data integration; 3. Streaming data processing; 4. Streaming OLAP application; 5. Future planning. […]

  • Smartnews: Practice of accelerating hive daily watch production based on Flink

    Time:2021-9-1

    Introduction: technical challenges and solutions to seamlessly integrate Flink into batch processing systems dominated by airflow and hive. This paper introduces the practice that smartnews uses Flink to accelerate the production of hive daily watch and seamlessly integrates Flink into the batch processing system based on airflow and hive. Introduce the technical challenges and solutions […]

  • Application of Flink real-time computing in microblog

    Time:2021-7-27

    Introduction: by combining Flink real-time stream computing framework with business scenarios, microblog has done a lot of work in platform and service, and has also done a lot of optimization in development efficiency and stability. We improve development efficiency through modular design and platform development. Cao Fuqiang, senior system engineer and data computing director of […]

  • Kafka stream errorlog alarm instance

    Time:2021-7-24

    KafkaAppender log4j-core-2.7-sources.jar!/org/apache/logging/log4j/core/appender/mom/kafka/KafkaAppender.java public void append(final LogEvent event) { if (event.getLoggerName().startsWith(“org.apache.kafka”)) { LOGGER.warn(“Recursive logging from [{}] for appender [{}].”, event.getLoggerName(), getName()); } else { try { final Layout<? extends Serializable> layout = getLayout(); byte[] data; if (layout != null) { if (layout instanceof SerializedLayout) { final byte[] header = layout.getHeader(); final byte[] body = layout.toByteArray(event); data […]

  • State management of Flink — fault tolerance — checkpoint

    Time:2020-11-14

    Stateful distributed streaming Stream processing In short, stream processing is an endless source of data, which continuously receives data, takes code as the basic logic of data processing, and then outputs, which is the basic principle of flow processing. Distributed streaming The stream needs to be partitioned, set the same key, and make the same […]

  • Spring boot2.0 integrates Kafka

    Time:2020-10-14

    Kafka overview Apache Kafka is a distributed stream processing platform, which is used to build real-time data pipelines and streaming applications. It can let you publish and subscribe to streaming records, store streaming records, and has good fault tolerance. It can process streaming records when they are generated. Apache Kafka is a distributed publish subscribe […]

  • Alibaba big data practice real time technology

    Time:2020-9-30

    Source: digital intelligence transformation Club The value of data is time sensitive. When a piece of data is generated, if it can not be processed in time and used in the business system, it can not keep the highest “freshness” and maximize the value of the data. Compared with offline batch processing technology, streaming real-time […]

  • Flink — basic components and wordcount

    Time:2020-9-25

    Xiaobai’s novice learn notes, please spray it gently This article is filed atGitHubWe welcome criticism and correction Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. […]

  • 2. My God, this is how Jackson wrote JSON

    Time:2020-9-13

    No one is always 18, but there are always 18. This paper has been published byhttps://www.yourbatman.cnIt includes spring technology stack, mybatis, JVM, middleware, etcspecial columnFor free study. Pay attention to the official account.Bat’s Utopia】Break through one by one, grasp deeply, and refuse to stop. preface Hello, I’m your Batman. The last article introduced Jackson, the […]

  • Application of iceberg in streaming data warehousing scenario based on Flink

    Time:2020-9-7

    Based on the scenario of streaming data warehousing, this paper introduces the benefits of introducing iceberg as the landing format and embedding Flink sink, and analyzes the current framework and key points. Application scenarios Streaming data warehousing is a typical application scenario of big data and data lake. The upstream stream data, such as log […]

  • All secrets of Yixin data center

    Time:2020-8-24

    Content source: the 11th technical Salon of Yixin Institute of Technology Speaker: Pei Guoqiang, solution architect of Yixin data center PPT download: link: https://pan.baidu.com/s/1eSkSdUo6FmYFmcE4xg0vjw Password: 99uh 1、 Data middle station positioning 1.1 overall introduction of ADX – midrange positioning First of all, the service scope of Zhongtai is described Enterprise level: for all business departments […]

  • [2020 sprint annual salary 30W] super big data learning route + mind map

    Time:2020-8-5

    Big data learning route Let’s talk about the big data learning route to help you quickly enter the big data industry. I will combine my own actual experience and explain the learning route. The target of this route is zero base Xiaobai, and the goal is to get junior high school big data engineers. It […]