Tag:Flow type

  • Flink helps meituan’s warehouse incremental production


    Introduction: This article is shared by Ju Dasheng, a researcher of meituan and head of real-time computing. It mainly introduces the application practice of Flink in helping meituan’s digital warehouse incremental production. The contents include: 1. Digital warehouse incremental production; 2. Streaming data integration; 3. Streaming data processing; 4. Streaming OLAP application; 5. Future planning. […]

  • Smartnews: Practice of accelerating hive daily watch production based on Flink


    Introduction: technical challenges and solutions to seamlessly integrate Flink into batch processing systems dominated by airflow and hive. This paper introduces the practice that smartnews uses Flink to accelerate the production of hive daily watch and seamlessly integrates Flink into the batch processing system based on airflow and hive. Introduce the technical challenges and solutions […]

  • Application of Flink real-time computing in microblog


    Introduction: by combining Flink real-time stream computing framework with business scenarios, microblog has done a lot of work in platform and service, and has also done a lot of optimization in development efficiency and stability. We improve development efficiency through modular design and platform development. Cao Fuqiang, senior system engineer and data computing director of […]

  • Kafka stream errorlog alarm instance


    KafkaAppender log4j-core-2.7-sources.jar!/org/apache/logging/log4j/core/appender/mom/kafka/KafkaAppender.java public void append(final LogEvent event) { if (event.getLoggerName().startsWith(“org.apache.kafka”)) { LOGGER.warn(“Recursive logging from [{}] for appender [{}].”, event.getLoggerName(), getName()); } else { try { final Layout<? extends Serializable> layout = getLayout(); byte[] data; if (layout != null) { if (layout instanceof SerializedLayout) { final byte[] header = layout.getHeader(); final byte[] body = layout.toByteArray(event); data […]

  • State management of Flink — fault tolerance — checkpoint


    Stateful distributed streaming Stream processing In short, stream processing is an endless source of data, which continuously receives data, takes code as the basic logic of data processing, and then outputs, which is the basic principle of flow processing. Distributed streaming The stream needs to be partitioned, set the same key, and make the same […]

  • Spring boot2.0 integrates Kafka


    Kafka overview Apache Kafka is a distributed stream processing platform, which is used to build real-time data pipelines and streaming applications. It can let you publish and subscribe to streaming records, store streaming records, and has good fault tolerance. It can process streaming records when they are generated. Apache Kafka is a distributed publish subscribe […]

  • Alibaba big data practice real time technology


    Source: digital intelligence transformation Club The value of data is time sensitive. When a piece of data is generated, if it can not be processed in time and used in the business system, it can not keep the highest “freshness” and maximize the value of the data. Compared with offline batch processing technology, streaming real-time […]

  • Flink — basic components and wordcount


    Xiaobai’s novice learn notes, please spray it gently This article is filed atGitHubWe welcome criticism and correction Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams. Flink has been designed to run in all common cluster environments, perform computations at in-memory speed and at any scale. […]

  • 2. My God, this is how Jackson wrote JSON


    No one is always 18, but there are always 18. This paper has been published byhttps://www.yourbatman.cnIt includes spring technology stack, mybatis, JVM, middleware, etcspecial columnFor free study. Pay attention to the official account.Bat’s Utopia】Break through one by one, grasp deeply, and refuse to stop. preface Hello, I’m your Batman. The last article introduced Jackson, the […]

  • Application of iceberg in streaming data warehousing scenario based on Flink


    Based on the scenario of streaming data warehousing, this paper introduces the benefits of introducing iceberg as the landing format and embedding Flink sink, and analyzes the current framework and key points. Application scenarios Streaming data warehousing is a typical application scenario of big data and data lake. The upstream stream data, such as log […]

  • All secrets of Yixin data center


    Content source: the 11th technical Salon of Yixin Institute of Technology Speaker: Pei Guoqiang, solution architect of Yixin data center PPT download: link: https://pan.baidu.com/s/1eSkSdUo6FmYFmcE4xg0vjw Password: 99uh 1、 Data middle station positioning 1.1 overall introduction of ADX – midrange positioning First of all, the service scope of Zhongtai is described Enterprise level: for all business departments […]

  • [2020 sprint annual salary 30W] super big data learning route + mind map


    Big data learning route Let’s talk about the big data learning route to help you quickly enter the big data industry. I will combine my own actual experience and explain the learning route. The target of this route is zero base Xiaobai, and the goal is to get junior high school big data engineers. It […]