• Java Concurrent Programming — a deep understanding of spin lock


    1. What is spin lock Spinlock: when a thread is acquiring a lock, if the lock has been acquired by other threads, the thread will wait in a loop, and then continuously judge whether the lock can be acquired successfully. It will not exit the loop until the lock is acquired.The thread that gets the […]

  • Data monitoring based on done file theory


    1 question Except for likeAlibabaOfDataworks In addition, it is very difficult for other companies to take data scheduling, data monitoring, data kinship, metadata management, etc. as an integrated platform. Some factories, including our company, often separate these constructions and are responsible by different teams. The data platform scheduling function is the basic platform of most […]

  • Operating system — deep understanding of process and thread


    1. Definition of process and thread Process: is a program in execution, a program is loaded into memory and ready to execute, it is a process, is a basic unit of system resource allocation and scheduling. Thread: is an entity of a process, is the basic unit of CPU scheduling and dispatching, it is smaller […]

  • Yard of Hadoop


    1. Overview of yarn Horn is a new feature in Hadoop 2. X.In version 1. X, MapReduce takes on too heavy tasks, including resource scheduling, while in version 2. X, the resource scheduling part is separated, that is, yarn, which makes Hadoop more stable and has better scalability, availability and reliability. So we can know […]

  • Spark of big data framework


    Spark cluster composition and execution process 1、 Spark cluster compositionSpark cluster is composed of cluster manager, worker, executor, driver and application.1. Driver: this process executes the main method of Spark Program and is responsible for the execution of actual code.1) Start sparkcontext or sparksession to convert user program into job;2) Scheduling tasks between executors;3) Track […]

  • 11 new projects selected in CNCF sandbox


    Industry news CNCF set up a new working group WG naming At present, the first task is to eliminate words with racial discrimination, such asnode-role.kubernetes.io/master ,whitelist/blacklist。 KubeCon + CloudNativeCon North America is going online Kubecon + cloudnativecon Na 2020 conference was held online from November 17 to 20, and CFP deadline was extended to July […]

  • Detailed analysis of the underlying principle of spark


    Introduction to spark Apache spark is used forLarge scale data processingThe unified analysis engine based on memory computing improves the real-time performance of data processing in the big data environment, and ensures the real-time performanceHigh fault toleranceandHigh Scalability , allowing users to deploy spark on a large number of hardware to form a cluster. Spark […]

  • Give a few simple examples to better understand the working principle of scratch


    Description: people who know about crawlers may know that in crawlers, requests are easy to get started, even if there is no basic Xiaobai. After learning for a few days, you can easily request the website, but it is relatively difficult to learn about scratch. This article can list a few simple examples to explain […]

  • Big update coming! Wondertrader new version v0.4.0 released


    Today (July 6, 2020) wondertrader released its latest version, withTwo major updates。 The update of v0.4.0 is as follows: Add a new oneStock selection scheduling engine, which is used to schedule the stock selection strategy of the application layer. After getting the target combination, it provides automatic execution service. For the time being, it only […]

  • Computer basic summary of 2021 autumn recruitment interview – operating system


    A series of articles: 2021 autumn recruitment interview computer foundation summary – algorithm, data structure, design pattern, Linux Computer foundation summary of 2021 autumn recruitment interview – Java foundation, JVM, spring framework Computer foundation summary of 2021 autumn recruitment interview database, redis Computer basic summary of 2021 autumn recruitment interview – operating system Computer foundation […]

  • High availability combat system based on haproxy


    Original address:__ https://www.jianshu.com/p/1fa… Author: Ming_ 96af Introduction to haproxy Software: haproxy — mainly do load balancing 7 layers, can also do 4 layers load balancing, Apache can also do 7 layers load balancing, but it is very troublesome. No one uses it in practice. Load balancing is through the OSI protocol corresponding to the 7-layer […]

  • Two kinds of data scheduling platform systems and their implementation methods and processes


    What is dispatching system Dispatching system, to be more precise,Job schedulerOr ratherWorkflow schedulerIt is an indispensable and important part of any big data development platform that has a little scale and is not a simple play. In addition to crontab, quartz and other single machine scheduling programs / libraries. There are also many open source […]