Image color extraction


This article GitHub project: colorfull color
My codepen link: image color extraction
the demo
Original article, reprint please indicate

Recently, I want to find a small project to practice so that I can be familiar with react, so I came up with the direction of “image color extraction”, which is also said to beImage theme color extractionColor quantization, orGenerate palette from image, because there is no other reason, just because it is beautiful!

Image color extraction

The purpose of “analysis” is as follows:

  • Main colors:Main color is the color with the highest frequency. In this way, color is often used for background color in design, providing immersive experience:

Image color extraction

  • Average color:Average color is the average value of all colors, which can be used as the background color as the main color;
  • Color quantization:Color quantization here is equivalent to extracting theme color. The result is a collection of a series of main colors in the image. These colors can be obtained by statistical analysis or generated by clustering algorithm. At the same time,Main colorsAverage colorandTheme colorThese factors can be used as imageFeaturesFeatures can be used for further image analysis, such as image recognition and retrieval, compression, etc;

Image color extraction

  • Color visualization:The image itself is a container of color, and this “container” is also a kind of visual presentation. I think we can also observe the color from another perspective – remove the image content, and only present the values of different colors and their weights. For example, here’s a visualization scheme like star and sky:

Image color extraction

1、 Common color quantization algorithms

1.1 median segmentation

The median segmentation algorithm firstly maps all the pixels to RGB space, and then repeatedly cuts out the subspace in this three-dimensional space. Finally, the average value of the pixels in the segmentation space is taken as the extraction result. When segmenting blocks, the block with the largest (the longest side length is the largest, or the volume is the largest, or the pixel is the largest) in all blocks shall be selected. The cutting point shall be located in the side direction, so that the pixel positions of the two blocks after segmentation are half of each other. The above isMedian segmentation。 The process is as follows (recommended reading: color quantification):

1. Pixel mapping to RGB space:

Image color extraction

2. Block calculation:

Image color extraction

3. Median segmentation:

Image color extraction

4. Repeated segmentation:

Image color extraction

5. Calculate the average color of the block:

Image color extraction

Here we recommend a color quantization project that uses the middle bit segmentation (JavaScript): Color Thief.

1.2 octree algorithm

The core idea of octree algorithm is to use octree to divide color space, and then merge leaf nodes to gradually gather color (quantization). For the explanation of octree, please refer to “what’s octree algorithm in game scene management?”? 》, the key is the following two pictures:

1. Tree building process:

Image color extraction

2. Merge leaf nodes:

Image color extraction

For specific explanation, please refer to the article: summary of image theme color extraction algorithm. The author also wrote a color quantized node module: a theme color extractor module for node.js

1.3 K-means clustering

The idea of K-means clustering is very simple, which can be divided into these steps:

  1. Select the initial K centroids;
  2. All samples were classified according to the distance from the center of mass;
  3. Recalculate the center of mass and determine whether to exit the condition:

    • If the distance between two centroids is small enough, the exit condition is satisfied;
    • If not, return to step 2;

Let’s see the implementation of JS:

Colors: all samples
Seeds: initial center of mass
Max? Step: maximum number of iterations
kMC(colors, seeds, max_step) {
    let iteration_count = 0;
    while (iteration_count++ < max_step) {
      // divide colors into different categories with duff's device
      classifyColor(colors, seeds);

      // compute center of category
      let len = colors.length;
      let hsl_count = [];
      let category;
      while (len--) {
        category = colors[len].category;
        // ......
      // quit or not
      let flag = hsl_count.every((ele, index) => {
        // ......
      if (flag) {
    console.log("KMC iteration " + iteration_count);

2、 Simple implementation

2.1 general process

  1. Canvas reads the local image and scales it properly;
  2. Statistical color information: color needs to be quantized(Color Quantization)In RGB space, there are more than 16 million colors in the third power of 255. Dividing by 8 can reduce the sampling to more than 32000. RGB value combination isKey value, count the number of times each color appears:
let r_key = Math.floor(r / 8) * 1000000;
let g_key = Math.floor(g / 8) * 1000;
let b_key = Math.floor(b / 8);
let key = r_one + g_one + b_one;
  //If no key is found, new key will be added
  //If found, add 1
  1. Filter color: filter the isolated color (too few times) and the too bright and too black color;
  2. K-means clustering: the K colors with the highest frequency are selected as initial values, and new stable color centers are clustered by the algorithm;
  3. Calculate the main color and mean color;

2.2 experimental results

Image color extraction

The original resolution of this image is 1080 x 1800, and the resolution of zooming to canvas is216 x 360(the scaling rule is to fix the maximum height to 360 and scale to the original width height ratio). Select the interval of color desampling as5, all together6251The number of times filtered out is less than4Remaining after over black and over bright colors2555Color. Set K of K-means clustering as6, the final number of iterations is10Time consuming106ms

The original example of codepen is as follows:

census color

There will be some problems in the implementation of this program:

  • The selection of K-means seed points has great influence on the results;
  • When calculating the clustering center, not only the RGB values, but also the number of color occurrences is added, so when k is small, the new clustering center may not converge toBold embellishmentIn terms of color, this is not consistent with our visual experience, but if K is selected as 10, the above image can converge to red.

3、 Neural network score

This part uses brain, which should be simpleBP neural network。 The training data is based on the popular pictures of the website. At present, there are few image databases with scores, and the scores are often comprehensive, mixed with other factors (composition, theme, light and shadow, characters, etc.), so it is difficult to separate themColor onlySo I scored the training data according to my own preference, so the result will be very close to my personal preference.
In addition, the input term of neural network is also key, because it must correctly reflect the color related image information. What I extract is:

let info = {
  colorCount: (Math.log10(colorInfo.length)),
  variance: 0,
  top50Count: 0,
  top50Average: 0,
  top50Variance: 0,
  top20Count: 0,
  top20Average: 0,
  top20Variance: 0,
  top10Count: 0,
  top10Average: 0,
  top10Variance: 0,
  top5Count: 0,
  top5Average: 0,
  top5Variance: 0

The data were divided into four categories, and the scores were 100, 85, 75 and 65 respectively.

Four, improvement

4.1 selection of color space

Before, RGB space was used. Three cold numbers can’t make us distinguish different colors very well. So here I try to switch to HSL space:Hue(H)、saturation(S)、Lightness(50) The superposition of these three color channels can get a variety of colors. This color space almost includes all colors that human vision can perceive. It is one of the most widely used color systems at present.
For the conversion of RGB and HSL, please refer to JavaScript hex and RGB, HSL color conversion.

Converting to HSL space has the following benefits for our goal of extracting color:

  • The three values in the original RGB are equally important. For HSL, we can use different parameters to deal with three channels respectively. For example, for hue, we can use dense sampling, for lightness and saturation, we can use sparse sampling;
  • The control of different colors is more precise and accurate. In the original RGB space, it is difficult for us to judge the RGB value relationship between two different colors, but for HSL, we only need to pay attention to the hue (the other two channels are also very useful, but here we choose to ignore them);

4.2 binary tree and indexof

The key step that affects the running time of the whole algorithm is the statistics of color information. The most time-consuming part of the statistics is the key detection. The length of the container storing the key will be longer and longerindexOfIt will be more and more time-consuming. Experiments show that most of the time is spent on this step. So try itFind binary treeIn this data structure, the advantage of binary tree is that the time of each search will decrease exponentially, so as to speed up the program operation.
howeverThe result of using JS to implement this data structure is not ideal. The running time is basically the same asindexOfIt’s the same, even a little bit more often. I think the reason is that: Although the time for each duplicate key search is reduced, the steps for each new key are complicated, andindexOf()yesnative code, the running efficiency should be higher than our own JS code. Generally speaking, in a certain range of sample size, the originalindexOfThe efficiency is higher. In this paper, the interval refers to1000~3000Of course, I still believe that when the color is more, the binary tree has its advantages. The code I implemented is as follows:

Traversal before / in / after binary tree

4.3 duff’s device

It’s a very practical skill (I have verified it many times), and I feel that I can’t do without it!

let len = colors.length;
let count = (len / 8) ^ 0;
let start = len % 8;
while (start--) {
  // do something
while (count--) {
  // do something  

Test result: jsprof.

4.4 fuzzy acceleration

It should be feasible to blur the image to reduce the types of colors, so as to speed up the extraction algorithm. But I haven’t joined in the project yet. The implementation of the fuzzy algorithm that I explore is faster and has better effect is as follows:

canvas blur

5、 Svg and canvas animation

At the beginning, I just want to be familiar with react. As a result, the focus of the project will be completely on Algorithm and animation. I think react is more friendly to SVG. All kinds of animation attributes can be put into state. The advantages of SVG animation over CSS are: more flexible, easier to complete complex animation effects, better compatibility, and smoother bottom optimization.
The advantage of canvas animation is that it is relatively smooth. There are still a lot of visible frame dropping and blocking in the mobile end of SVG animation, and SVG will make HTML very messy, which may make you uncomfortable with cleanliness.

SVG halo animation

No matter what kind of animation is ultimately attributed to:MathematicsFor example,

The difficulty of this animation lies in the random layout algorithm, the key lies in the detection of collision and the movement after collision, which essentially depends on the laws of motion in geometry and Physics:

bubble chart

The difficulty of this animation is to find a magic mathematical formula. Although I don’t know what’s going on, sometimes I can achieve regular animation by changing the mathematical formula, and regular plus color, a great probability is a good animation:

canvas wave