Tag:Dispatch

  • Java Concurrent Programming — a deep understanding of spin lock

    Time:2021-2-27

    1. What is spin lock Spinlock: when a thread is acquiring a lock, if the lock has been acquired by other threads, the thread will wait in a loop, and then continuously judge whether the lock can be acquired successfully. It will not exit the loop until the lock is acquired.The thread that gets the […]

  • Data monitoring based on done file theory

    Time:2021-2-27

    1 question Except for likeAlibabaOfDataworks In addition, it is very difficult for other companies to take data scheduling, data monitoring, data kinship, metadata management, etc. as an integrated platform. Some factories, including our company, often separate these constructions and are responsible by different teams. The data platform scheduling function is the basic platform of most […]

  • Operating system — deep understanding of process and thread

    Time:2021-2-26

    1. Definition of process and thread Process: is a program in execution, a program is loaded into memory and ready to execute, it is a process, is a basic unit of system resource allocation and scheduling. Thread: is an entity of a process, is the basic unit of CPU scheduling and dispatching, it is smaller […]

  • Yard of Hadoop

    Time:2021-2-25

    1. Overview of yarn Horn is a new feature in Hadoop 2. X.In version 1. X, MapReduce takes on too heavy tasks, including resource scheduling, while in version 2. X, the resource scheduling part is separated, that is, yarn, which makes Hadoop more stable and has better scalability, availability and reliability. So we can know […]

  • Spark of big data framework

    Time:2021-2-11

    Spark cluster composition and execution process 1、 Spark cluster compositionSpark cluster is composed of cluster manager, worker, executor, driver and application.1. Driver: this process executes the main method of Spark Program and is responsible for the execution of actual code.1) Start sparkcontext or sparksession to convert user program into job;2) Scheduling tasks between executors;3) Track […]

  • 11 new projects selected in CNCF sandbox

    Time:2021-2-6

    Industry news CNCF set up a new working group WG naming At present, the first task is to eliminate words with racial discrimination, such asnode-role.kubernetes.io/master ,whitelist/blacklist。 KubeCon + CloudNativeCon North America is going online Kubecon + cloudnativecon Na 2020 conference was held online from November 17 to 20, and CFP deadline was extended to July […]

  • Detailed analysis of the underlying principle of spark

    Time:2021-2-3

    Introduction to spark Apache spark is used forLarge scale data processingThe unified analysis engine based on memory computing improves the real-time performance of data processing in the big data environment, and ensures the real-time performanceHigh fault toleranceandHigh Scalability , allowing users to deploy spark on a large number of hardware to form a cluster. Spark […]

  • Give a few simple examples to better understand the working principle of scratch

    Time:2021-1-22

    Description: people who know about crawlers may know that in crawlers, requests are easy to get started, even if there is no basic Xiaobai. After learning for a few days, you can easily request the website, but it is relatively difficult to learn about scratch. This article can list a few simple examples to explain […]

  • Big update coming! Wondertrader new version v0.4.0 released

    Time:2021-1-22

    Today (July 6, 2020) wondertrader released its latest version, withTwo major updates。 The update of v0.4.0 is as follows: Add a new oneStock selection scheduling engine, which is used to schedule the stock selection strategy of the application layer. After getting the target combination, it provides automatic execution service. For the time being, it only […]

  • Computer basic summary of 2021 autumn recruitment interview – operating system

    Time:2021-1-19

    A series of articles: 2021 autumn recruitment interview computer foundation summary – algorithm, data structure, design pattern, Linux Computer foundation summary of 2021 autumn recruitment interview – Java foundation, JVM, spring framework Computer foundation summary of 2021 autumn recruitment interview database, redis Computer basic summary of 2021 autumn recruitment interview – operating system Computer foundation […]

  • High availability combat system based on haproxy

    Time:2021-1-18

    Original address:__ https://www.jianshu.com/p/1fa… Author: Ming_ 96af Introduction to haproxy Software: haproxy — mainly do load balancing 7 layers, can also do 4 layers load balancing, Apache can also do 7 layers load balancing, but it is very troublesome. No one uses it in practice. Load balancing is through the OSI protocol corresponding to the 7-layer […]

  • Two kinds of data scheduling platform systems and their implementation methods and processes

    Time:2021-1-16

    What is dispatching system Dispatching system, to be more precise,Job schedulerOr ratherWorkflow schedulerIt is an indispensable and important part of any big data development platform that has a little scale and is not a simple play. In addition to crontab, quartz and other single machine scheduling programs / libraries. There are also many open source […]