Tag:Flow type

  • Top ten hottest big data technologies

    Time:2021-12-20

    With the rapid penetration of big data analysis market into all walks of life, which big data technologies are just needed? Which technologies have great potential value? According to the index released by Forrester Research, here are the ten hottest big data technologies. 1. Forecast analysis Predictive analysis is a statistical or data mining solution […]

  • Kafka source code analysis series – Directory (collection is not lost)

    Time:2021-12-13

    Continuous update, please pay attention! catalogue The series of articles on Kafka source code analysis is planned to be written in the order of “data transmission”, that is, first analyze the producers, then analyze the data processing on the server, then analyze the consumers, and finally supplement some related contents of transactions and streaming computing. […]

  • Interviewer: how to query tens of millions of data through mybatis and ensure that the memory does not overflow?

    Time:2021-12-11

    Due to the mature application of ORM framework, many small partners are weak in the concept of JDBC. The bottom layer of ORM framework is actually a DB operated through JDBC JDBC (javadatabase connectivity) is a java database connection. To put it bluntly, it is to use the Java language to operate the database Sun […]

  • Python operation MySQL streaming reading data (black technology you haven’t seen)

    Time:2021-12-7

    Finally meet you again! It’s been a long time since I updated my developer notes. Today, let’s share a big trick! Before sharing, you must have encountered the operation of reading a large amount of data by mysql. I think if you read a large amount of data at one time, your memory will not […]

  • Learning Flink from 0 to 1 – Introduction to Apache Flink

    Time:2021-12-2

    preface Flink is a streaming computing framework. Why did I come into contact with Flink? At present, I am responsible for the alarm part of the monitoring platform. The collected monitoring data will be directly inserted into Kafka. Then, on the alarm side, I need to read the monitoring data in real time from Kafka […]

  • Grpc service invocation practice (I)

    Time:2021-11-15

    Service oriented Recently, I was looking at the design related to microservices, includinggrpcIt is the standard configuration of the microservice framework, so we should study it This article is suitable for those who have a little impression of grpc. You can search it in the station you don’t know,golangSomeone in the section wrote it very […]

  • Django streaming response Chinese CSV example

    Time:2021-10-25

    In Django, streaming responseStreamingHttpResponseIt is a good thing to generate a large file quickly and save memory. One of the current projects for streaming response isEventsourceIt is used to improve the user’s feeling of slow speed when communicating across systems. I won’t elaborate on this. Another is to generate a large CSV file. When the […]

  • Alibaba, indeed, began to have the ability to “predict the future”

    Time:2021-10-3

    The ceiling lamp flashes and the flute rings. The ambulance car rushed to the vast sea of cars and ran on the track of time. Gaode map, GPS satellite navigation, road magnetic induction coil and 1300 intersection cameras started at the same time to explore the fastest route for the ambulance; GPS sends back real-time […]

  • Flink helps meituan’s warehouse incremental production

    Time:2021-9-4

    Introduction: This article is shared by Ju Dasheng, a researcher of meituan and head of real-time computing. It mainly introduces the application practice of Flink in helping meituan’s digital warehouse incremental production. The contents include: 1. Digital warehouse incremental production; 2. Streaming data integration; 3. Streaming data processing; 4. Streaming OLAP application; 5. Future planning. […]

  • Smartnews: Practice of accelerating hive daily watch production based on Flink

    Time:2021-9-1

    Introduction: technical challenges and solutions to seamlessly integrate Flink into batch processing systems dominated by airflow and hive. This paper introduces the practice that smartnews uses Flink to accelerate the production of hive daily watch and seamlessly integrates Flink into the batch processing system based on airflow and hive. Introduce the technical challenges and solutions […]

  • Application of Flink real-time computing in microblog

    Time:2021-7-27

    Introduction: by combining Flink real-time stream computing framework with business scenarios, microblog has done a lot of work in platform and service, and has also done a lot of optimization in development efficiency and stability. We improve development efficiency through modular design and platform development. Cao Fuqiang, senior system engineer and data computing director of […]

  • Kafka stream errorlog alarm instance

    Time:2021-7-24

    KafkaAppender log4j-core-2.7-sources.jar!/org/apache/logging/log4j/core/appender/mom/kafka/KafkaAppender.java public void append(final LogEvent event) { if (event.getLoggerName().startsWith(“org.apache.kafka”)) { LOGGER.warn(“Recursive logging from [{}] for appender [{}].”, event.getLoggerName(), getName()); } else { try { final Layout<? extends Serializable> layout = getLayout(); byte[] data; if (layout != null) { if (layout instanceof SerializedLayout) { final byte[] header = layout.getHeader(); final byte[] body = layout.toByteArray(event); data […]