Implement SLIC algorithm to generate pixel pictures

preface

Pixel style first appeared in 8bit video games. Restricted by the size of computer memory and single display color, it can only use a small number of pixels to present content, but it has achieved many classic pixel games. With the improvement of memory capacity and screen resolution, the limitation of memory and display media is no longer a problem, and pixel style has slowly evolved into a unique creative style.

The general rendering process of pixel painting includes line drawing, color filling, etc., and pixel by pixel rendering requires a lot of time. Some popular art methods, such as line drawing and painting, have gradually appeared automatic or semi-automatic generation methods. This article will be implemented from scratchSLIC[1] Algorithm, and implement a tool to generate pixel painting.

What is SLIC algorithm

The reason why pixel painting is not simple is that direct down sampling can not accurately capture key pixels, and it is easy to lose edge information. The generated pixel painting is often unsatisfactory. The purpose of manual line drawing and color filling is to select appropriate pixels. Therefore, our problem becomes how to select the appropriate pixels for color filling.

First, we introduce a concept – super pixel. Super pixel is an image segmentation technology proposed and developed by Xiaofeng Ren in 2003. It refers to irregular pixel blocks with certain visual significance composed of adjacent pixels with similar texture, color, brightness and other characteristics [1].

By dividing the picture into super pixels, similar pixel clusters can be obtained. Similar pixels are filled with the same color, and the pixel picture will be more reasonable.

The methods of super-pixel segmentation include contour extraction, clustering, gradient rise and so on. paper[1]ProposedSLICSuper pixel segmentation algorithm (simple linear iterative clustering,simple linear iterative clustering）Is one of them, which is based onK-meansClustering algorithm, clustering according to the color and distance characteristics of pixels to achieve good segmentation results, compared with several super-pixel segmentation algorithms,SLICIt has the advantages of simplicity, flexibility, good effect and fast processing speed.

How to implement SLIC algorithm

SLICThe basic process is as follows:

1. Image preprocessing.

   Image fromRGBColor space conversion toCIE-LabColor space,LabColor space is more in line with human visual perception of color. The distance in this space can reflect the color difference people feel, and the relevant calculation is more accurate.

   LabThe color space also has three channels, which arel，a，bWherelRepresents brightness, and the value range is[0,100]，aIndicates the component from green to red, and the value range is[-128,127]，bRepresents the blue to yellow component, and the value range is[-128,127]。

   RGBandLABThere is no direct conversion formula between. You need toRGBConvert toXYZColor space toLAB, see the complete code at the end of the text for the code.

2. Initialize the cluster center.

   The number of super pixels, that is, the number of areas to be divided, is determined according to the parameters. Suppose the picture hasNPixels, estimated to be divided intoKSuper pixels, each with a size ofN/K, the distance between adjacent centers isS=Sqr(N/K), getKCluster coordinates.

   

3. Optimize the initial cluster center. In the cluster center3*3In the neighborhood, the pixel with the smallest gradient is selected as the new cluster center.

   The image is regarded as a two-dimensional discrete function, and the gradient is the derivation of this function. When the values of adjacent pixels change, there will be a gradient, and the gradient of pixels on the edge is the largest. Moving the cluster center to the place with the minimum gradient can prevent it from falling on the edge contour and affect the clustering effect.

   The gradient calculation of discrete gradient will not be derived in detail here. Since it contains several * squares and square root, it will be simplified to the operation of * like * square and * square root with absolute value. The simplified calculation coordinates are(i,j)The gradient formula of the pixel of is:

   

   among（i+1,j）And（i,j+1）Is the coordinates of the point on the right side of the pixel and the point below the pixel.l(a,b)by（a，b）Luminance channel value of pixels on coordinatesl。

4. Calculate the distance between the pixel and the cluster center.

   In the area with cluster center distance s2S*2SThe distance between pixels and each cluster center is calculated in the neighborhood of.

   

   The distance here is the European distance, the total distanceDfromdcColor distance anddsSpace distance is composed of two parts. The formula is as follows:

   

   If you directlyl，a，b，x，ySpliced into a vector to calculate the distance. When the size of the super pixel changes,x，yFor example, if a graph1000*1000, the space distance can reach1000*Sqr(2), and the maximum color distance is only10*Sqr(2), resulting in the space distancedsThe weight proportion is too large.

   Therefore, it is necessary to normalize and divide by the maximum value, that is, the initial width of the super pixelS, map values to[0,1]。

   And the color space distance will also be given a fixed valuemTo adjust the influence weight of color distance and space distance,mThe value range is[1,40]。

   The distance formula becomes

   

   WhenmLarger, color space divided bymThe smaller the value after, that is, the greater the weight of the spatial distance, the more regular the shape of the generated pixels will bemThe smaller the size, the greater the color distance weight, the more compact the super pixels will be at the edge, and the shape size will be more irregular.

5. Pixel classification.

   The category that marks each pixel is the category that is the smallest from its cluster center.

6. Recalculate cluster centers.

   Calculate the * mean vector values of all pixels belonging to the same cluster, and get the cluster center again.

7. iteration4～6Process.

   Until the distance between the old cluster center and the new cluster center is less than a certain threshold or reaches a certain number of iterations, generally speaking, when the number of iterations reaches10, the algorithm can reach convergence.

8. Clustering optimization.

   At the end of the iteration, there may be isolated pixels that do not belong to the same connectivity domain as the cluster center, which can be assigned to the most * cluster label using the connectivity algorithm.

   The paper does not give a specific implementation algorithm. The application scenario of this paper is to generate pixel paintings, which will be down sampled and not refined to each pixel. Therefore, this paper does not do clustering optimization.

   

To sum up, the SLIC algorithm process is roughly the same asK-meansIt is consistent. The cluster with the smallest distance is calculated iteratively. The difference is that only theSThe calculation of pixels within the distance reduces a lot of calculation.

Generate pixel painting

be based onSLICAlgorithm, we can already divide a graph intoNSuper pixels. The pixels in each super pixel are relative. That is, each pixel is classified as a super pixel with a cluster center. Then assign the color of the pixel to the color of its cluster center to get the desired effect.

Set a certain stepstride, usingCanvas, everystridePixels, and assign the pixel to the color of its cluster center, that is, the final pixelation result is obtained.

Everyone’s subjective feelings about pixel painting are inconsistent. In order to let users have more choices and get their own satisfactory results. More manual intervention parameters can be exposed, such as canceling the termination condition of clustering optimization, and setting the number of iterations and the step size of the final pixel value by the user. Manually set parameters include

• Super pixel sizeblocksize；blocksizeThe smaller the size, the finer the super pixel segmentation.
• Iterationsiters；itersThe larger the size, the more accurate the segmentation result and the longer the calculation time.
• Color space weightsweight；weightThe larger the, the greater the influence of color on the segmentation results.
• Take pixel stepstride；strideThe smaller, the more * super pixels are connected to the generated pixel map, and the more delicate it will be.

Implement user interface

As a tool, user interaction interface is naturally required. The front-end interface is based onHTML/Javascript/CSSBuild, useCanvas APIDraw image content, and the user interaction panel selectsdat.gui[3] Library.dat.guiIs a lightweight graphical interface library, which is very suitable for parameter modification. It is often used as a demonstration of visual demo. Supported parameter types includeNumber、String、Boolean, custom functions, etc. Corresponding response events can be bound for different attributes, and the event will be triggered automatically when the attribute value changes.

Add the following attributes and events to the build pixelation tool:

• Wheniters、stride、blockSize、weight(color space weight m) restart when the parameter changesSLICCalculate the algorithm and redraw the calculation results;
• add toUpload imageAndExport imageButton, which supports users to upload pictures and download pixelated pictures;

When drawing an imageCanvasOverlay one layer on the canvas layerCanvasCanvas to visualize the results of the algorithm and add the following functions

• gridSwitch to control whether to draw pixel grid;
• CentersSwitch to control whether the cluster center is displayed;
• ContoursSwitch to control whether the cluster edge contour is displayed;

Cluster centerCentersDirect use ofctx.fillRectJust pass in the coordinates of the center point.

Superpixel outlineContoursThe contour points need to be calculated first.

Each pixel and the surrounding8Compare pixels. If the number of pixels with different cluster centers is greater than2, it means that there are more than two points of different categories around this pixel, and this point is the outline. The effect is as follows:

Finally, we get a simple tool to generate pixel drawings.

Experience address

Full version code address (JS version)

reference

[1] Achanta R, Shaji A, Smith K, Lucchi A, Fua P, Su ̈sstrunk S. SLIC superpixels. Technical Report. IVRG CVLAB; 2010.

[2] Gerstner T , Decarlo D , Alexa M , et al. Pixelated image abstraction with integrated user constraints[J]. Computers & graphics, 2013.

[3] https://github.com/dataarts/dat.gui

Welcome to bump lab blog:aotu.io

Or follow aotulabs official account and push articles from time to time: