Detailed explanation of the best [golang] map of the year

Time:2020-11-29

This article mainly talks about the specific implementation process of map assignment, deletion, query and expansion, which is still from the perspective of the bottom. Combined with the source code, read this article will thoroughly understand the underlying principle of map.

What I want to make clear is that the basic usage of map is less involved here. I believe that I can learn it by reading other introductory books. The content of this article is more in-depth, but I believe it is easy to understand because I have drawn various pictures.

What is a map

Wikipedia defines it as follows:

In computer science, an associative array, map, symbol table, or dictionary is an abstract data type composed of a collection of (key, value) pairs, such that each possible key appears at most once in the collection.

To make it simple: in computer science, what is called a correlated array, map, symbol table, or dictionary, is a set of<key, value>For the composition of the abstract data structure, and the same key will only appear once.

There are two key points: map is created bykey-valueOf composition; of;keyOnly once.

Map related operations are as follows:

  1. Add a K-V pair — add or insert;
  2. Remove or delete a K-V pair;
  3. Modify the v-reassign corresponding to a certain K;
  4. Query the v-lookup corresponding to a certain K;

In short, it is the most basicAdd, delete, check and modify

The design of map is also known as “the dictionary problem”. Its task is to design a data structure to maintain the data of a set, and to add, delete, query and modify the set at the same time. There are two main data structures:Hash tableSearch tree

Hash lookup table uses a hash function to assign keys to different buckets (i.e. different indexes of the array). In this way, the cost is mainly in the calculation of hash function and the constant access time of array. In many scenarios, hash lookup tables perform well.

Hash lookup tables generally have the problem of “collision”, that is, different keys are hashed to the same bucket. There are generally two ways to deal with itLinked list methodandOpen address methodLinked list methodImplement a bucket into a linked list, and the keys in the same bucket will be inserted into the linked list.Open address methodAfter the collision, the “empty bit” is selected at the back of the array according to certain rules to place a new key.

Search tree method generally uses self balanced search tree, including AVL tree and red black tree. During the interview, they are often asked and even asked to write the red and black tree code. In many cases, the interviewer can’t write it himself, which is too much.

The worst search efficiency of the self balanced search tree method is O (logn), while the worst search efficiency of hash lookup table is O (n). Of course, the average search efficiency of hash lookup table is O (1). If the hash function is well designed, the worst case will not happen. In addition, traversing the self balanced search tree, the returned key sequence is generally in the order from small to large, while the hash lookup table is out of order.

Why map

An excerpt from the go language official blog:

One of the most useful data structures in computer science is the hash table. Many hash table implementations exist with varying properties, but in general they offer fast lookups, adds, and deletes. Go provides a built-in map type that implements a hash table.

Hash table is the most important design of computer data structure. Most hash tables have the functions of quick search, addition and deletion. The built-in map of go language realizes all the above functions.

It’s hard to imagine writing a program that doesn’t use a map, making it difficult to answer the question why you use a map.

So why use map? Because it is too powerful, the operation efficiency of various additions, deletions and modifications is very high.

How to implement the bottom layer of map

First of all, I use the go version:

go version go1.9.2 darwin/amd64

In this paper, we have mentioned several schemes of map implementation. Go language uses hash lookup table, and uses linked list to solve hash conflict.

Next, we’ll explore the core principles of map and take a look at its internal structure.

Map memory model

In the source code, the structure representing map is hmap, which is the abbreviation of HashMap

// A header for a Go map.
type hmap struct {
    //Element number. This value is returned directly when len (map) is called
    count     int
    flags     uint8
    //Log ﹣ 2 of buckets
    B         uint8
    //Bucket approximation of overflow
    noverflow uint16
    //The hash function is passed in when calculating the hash of the key
    hash0     uint32
    //Points to the buckets array with a size of 2 ^ B
    //Nil if the number of elements is 0
    buckets    unsafe.Pointer
    //When the capacity is expanded, the length of buckets will be twice that of oldbuckets
    oldbuckets unsafe.Pointer
    //Indicates the expansion progress. The migration of buckets smaller than this address is completed
    nevacuate  uintptr
    extra *mapextra // optional fields
}

Just to make it clear,BIs the logarithm of the length of the buckets array, that is, the length of the buckets array is 2 ^ B. The bucket stores key and value, which will be discussed later.

Buckets is a pointer that eventually points to a structure:

type bmap struct {
    tophash [bucketCnt]uint8
}

But this is just the surface (SRC / runtime)/ hashmap.go )A new structure is created dynamically by feeding it during compilation

type bmap struct {
    topbits  [8]uint8
    keys     [8]keytype
    values   [8]valuetype
    pad      uintptr
    overflow uintptr
}

bmapThis is what we often call “bucket”. There are at most eight keys in the bucket. The reason why these keys fall into the same bucket is that after hash calculation, the hash result is “one class”. In a bucket, the location of the key in the bucket will be determined according to the high 8 bits of the hash value calculated by the key (there are at most 8 positions in a bucket).

Take a whole picture:

Detailed explanation of the best [golang] map of the year

When the key and value of the map are not pointers, and the size is less than 128 bytes, BMAP will be marked as no pointer, which can avoid scanning the entire hmap during GC. However, we can see that BMAP actually has an overflow field, which is of pointer type, which breaks the assumption that BMAP does not contain pointers. In this case, it will move the overflow field to the extra field.

type mapextra struct {
    // overflow[0] contains overflow buckets for hmap.buckets.
    // overflow[1] contains overflow buckets for hmap.oldbuckets.
    overflow [2]*[]*bmap

    //Next overflow contains idle overflow buckets, which are pre allocated buckets
    nextOverflow *bmap
}

BMAP is the place where K-V is stored. Let’s take a closer look at the internal composition of BMAP.

Detailed explanation of the best [golang] map of the year

The above figure shows the memory model of bucket,HOB HashIt means top hash. Notice that key and value are placed together, not necessarilykey/value/key/value/...In this form. The source code shows that the advantage is that in some cases, you can omit the padding field to save memory space.

For example, there is a map of this type:

map[int64]int8

Ifkey/value/key/value/...In this mode, each key / value pair must be padded by 7 bytes; all the keys and values are bound togetherkey/key/.../value/value/..., you only need to add padding at the end.

Each bucket is designed to hold at most eight key value pairs. If the ninth key value falls into the current bucket, it is necessary to build another bucketoverflowThe pointers are connected.

Create map

At the syntax level, creating a map is simple:

ageMp := make(map[string]int)
//Specify the map length
ageMp := make(map[string]int, 8)

//If the agemp is nil, you can't add elements to it. It will pan directly
var ageMp map[string]int

As can be seen from the assembly language, in fact, what the underlying calls aremakemapFunction, the main job is to initializehmapStructure, such as calculating the size of B, setting hash seed hash0, and so on.

func makemap(t *maptype, hint int64, h *hmap, bucket unsafe.Pointer) *hmap {
    //Omit various condition checks

    //Find a B so that the load factor of the map is in the normal range
    B := uint8(0)
    for ; overLoadFactor(hint, B); B++ {
    }

    //Initialize hash table
    //If B is equal to 0, then buckets will be reallocated at the time of assignment
    //If the length is large, it will take longer to allocate memory
    buckets := bucket
    var extra *mapextra
    if B != 0 {
        var nextOverflow *bmap
        buckets, nextOverflow = makeBucketArray(t, B)
        if nextOverflow != nil {
            extra = new(mapextra)
            extra.nextOverflow = nextOverflow
        }
    }

    //Initialize HAMP
    if h == nil {
        h = (*hmap)(newobject(t.hmap))
    }
    h.count = 0
    h.B = B
    h.extra = extra
    h.flags = 0
    h.hash0 = fastrand()
    h.buckets = buckets
    h.oldbuckets = nil
    h.nevacuate = 0
    h.noverflow = 0

    return h
}

Note that this function returns the following result:*hmapIt’s a pointer, and we talked about it beforemakesliceThe function returnsSliceStructure:

func makeslice(et *_type, len, cap int) slice

Let’s review the structure definition of slice

// runtime/slice.go
type slice struct {
    array  unsafe.Pointer  //Element pointer
    Len int // length 
    Cap int // capacity
}

The structure contains the underlying data pointer.

The difference between makemap and makeslice brings about a difference: when map and slice are used as function parameters, the operation of map within function parameters will affect the map itself; however, for slice, it will not (as mentioned in the previous article on slice).

The main reason: one is the pointer(*hmap)One is a structure(slice)。 In go language, function parameter passing is value passing. In function internal, parameter will be copied to local.*hmapAfter the copy map is finished, it will still affect the internal operation of the same function. After the slice is copied, it will become a new slice, and the operation on it will not affect the actual parameters.

hash function

A key point of map is the choice of hash function. When the program starts, it will detect whether the CPU supports AES. If it does, AES hash will be used, otherwise memhash will be used. This is in the functionalginit()In the path:src/runtime/alg.goNext.

Hash function, there are encryption type and non encryption type.
For general data encryption, shamds256 and 256 are typical for encryption;
Unencrypted is usually a lookup. In the application scenario of map, search is used.
The choice of hash function mainly focuses on two points: performance and collision probability.

As we mentioned earlier, the structure representing the type:

type _type struct {
    size       uintptr
    ptrdata    uintptr // size of memory prefix holding all pointers
    hash       uint32
    tflag      tflag
    align      uint8
    fieldalign uint8
    kind       uint8
    alg        *typeAlg
    gcdata    *byte
    str       nameOff
    ptrToThis typeOff
}

amongalgFields are related to hashes. They are pointers to the following structures:

// src/runtime/alg.go
type typeAlg struct {
    // (ptr to object, seed) -> hash
    hash func(unsafe.Pointer, uintptr) uintptr
    // (ptr to object A, ptr to object B) -> ==?
    equal func(unsafe.Pointer, unsafe.Pointer) bool
}

Typealg contains two functions. The hash function calculates the hash value of the type, while the equal function calculates whether the two types are “hash equal”.

For string type, its hash and equal functions are as follows:

func strhash(a unsafe.Pointer, h uintptr) uintptr {
    x := (*stringStruct)(a)
    return memhash(x.str, h, uintptr(x.len))
}

func strequal(p, q unsafe.Pointer) bool {
    return *(*string)(p) == *(*string)(q)
}

According to the type of key_ The alg field of the type structure is set with hash and equal functions of the corresponding type.

Key location process

Key gets the hash value after hash calculation. There are 64 bit bits in total (64 bit machine, 32-bit machine will not be discussed, now the mainstream is 64 bit machine). When calculating which bucket it will fall in, only the last B bit will be used. Remember the B mentioned earlier? If B = 5, then the number of buckets, that is, the length of the buckets array is 2 ^ 5 = 32.

For example, after a key is calculated by the hash function, the hash result is as follows:

 10010111 | 000011110110110010001111001010100010010110010101010 │ 01010

Use the last five bits, that is01010The value is 10, which is barrel 10. This operation is actually a remainder operation, but the cost is too large, so the code implementation uses bit operation instead.

Then use the high 8 bits of the hash value to find the position of the key in the bucket. This is to find the existing key. At the beginning, there is no key in the bucket. The newly added key will find the first empty position and put it in.

The bucket number is the bucket number. When two different keys fall into the same bucket, a hash conflict occurs. The solution to the conflict is to use the linked list method: in the bucket, find the first empty position from front to back. In this way, when searching for a key, first find the corresponding bucket, and then traverse the key in the bucket.

Here, refer to a picture in Cao Da GitHub blog. The original picture is ASCII diagram, full of geek flavor. You can find Cao Da’s blog from the reference materials. We recommend you to have a look.

Detailed explanation of the best [golang] map of the year

In the figure above, assume B = 5, so the total number of buckets is 2 ^ 5 = 32. First, calculate the hash of the key to be found, and use the lower 5 bits00110, find the corresponding bucket No. 6, and use the upper 8 bits10010111, corresponding to decimal 151, search the key with the top hash value (hob hash) of 151 in bucket 6, and find slot 2, so the whole search process is finished.

If it is not found in the bucket and the overflow is not empty, continue to search in the overflow bucket until all the key slots are found, including all overflow buckets.

Let’s look at the source code, ha ha! From the assembly language, we can see that the underlying function to find a key ismapacessSeries of functions, the function is similar, the difference will be discussed in the next section. Let’s look directly heremapacess1Function:

func mapaccess1(t *maptype, h *hmap, key unsafe.Pointer) unsafe.Pointer {
    // ……
    
    //If h has nothing, it returns zero
    if h == nil || h.count == 0 {
        return unsafe.Pointer(&zeroVal[0])
    }
    
    //Write read conflict
    if h.flags&hashWriting != 0 {
        throw("concurrent map read and map write")
    }
    
    //The hash algorithm used by different types of keys is determined at compile time
    alg := t.key.alg
    
    //Calculate the hash value and add hash0 to introduce randomness
    hash := alg.hash(key, uintptr(h.hash0))
    
    //For example, if B = 5, then M is 31 and binary is all 1
    //When calculating bucket num, hash and m are matched,
    //The result shows that the bucket num is determined by the lower 8 bits of hash
    m := uintptr(1)<<h.B - 1
    
    //B is the address of the bucket
    b := (*bmap)(add(h.buckets, (hash&m)*uintptr(t.bucketsize)))
    
    //Oldbuckets is not nil, indicating that expansion has occurred
    if c := h.oldbuckets; c != nil {
        //If it is not the same size expansion (see the content of expansion later)
        //Solution to condition 1
        if !h.sameSizeGrow() {
            //The number of new buckets is twice that of the old ones
            m >>= 1
        }
        
        //Find the bucket position of key in the old map
        oldb := (*bmap)(add(c, (hash&m)*uintptr(t.bucketsize)))
        
        //If oldB does not move to a new bucket
        //Look for it in the old bucket
        if !evacuated(oldb) {
            b = oldb
        }
    }
    
    //Calculate the high 8-bit hash
    //It is equivalent to shifting 56 bits to the right and only 8 bits higher
    top := uint8(hash >> (sys.PtrSize*8 - 8))
    
    //Add a mintophash
    if top < minTopHash {
        top += minTopHash
    }
    for {
        //Traverse 8 buckets
        for i := uintptr(0); i < bucketCnt; i++ {
            //Tophash does not match, continue
            if b.tophash[i] != top {
                continue
            }
            //Tophash matching to locate the key
            k := add(unsafe.Pointer(b), dataOffset+i*uintptr(t.keysize))
            //Key is a pointer
            if t.indirectkey {
                //Dereference
                k = *((*unsafe.Pointer)(k))
            }
            //If the keys are equal
            if alg.equal(key, k) {
                //Locate the location of value
                v := add(unsafe.Pointer(b), dataOffset+bucketCnt*uintptr(t.keysize)+i*uintptr(t.valuesize))
                //Value dereference
                if t.indirectvalue {
                    v = *((*unsafe.Pointer)(v))
                }
                return v
            }
        }
        
        //After searching for the bucket (not found yet), continue to search in the overflow bucket
        b = b.overflow(t)
        //The overflow bucket is also found, indicating that there is no target key
        //Returns a zero value
        if b == nil {
            return unsafe.Pointer(&zeroVal[0])
        }
    }
}

The function returns a pointer to h [key]. If there is no such key in H, it will return a zero value of the corresponding type of key, and will not return nil.

The whole code is relatively direct, there is nothing difficult to understand. Follow the notes above to understand step by step.

Here, let’s talk about how to locate key and value and how to write the whole loop.

//Key location formula
k := add(unsafe.Pointer(b), dataOffset+i*uintptr(t.keysize))

//Value positioning formula
v := add(unsafe.Pointer(b), dataOffset+bucketCnt*uintptr(t.keysize)+i*uintptr(t.valuesize))

B is the address of the BMAP. Here, BMAP is the structure defined in the source code. It only contains a tophash array. The structure expanded by the compiler contains the fields key, value and overflow. Dataoffset is the offset of the key from the BMAP start address:

dataOffset = unsafe.Offsetof(struct {
        b bmap
        v int64
    }{}.v)

Therefore, the starting address of the key in the bucket is unsafe.Pointer (b)+dataOffset。 On this basis, the address of the i-key must span the size of the i-key. We also know that the address of value is after all keys, so the address of the i-value needs to add the offset of all keys. After understanding these, the positioning formulas of key and value above are easy to understand.

In addition to the writing of the whole great cycle, the outermost layer is an infinite loop, through which

b = b.overflow(t)

Traverse all buckets, which is equivalent to a bucket list.

When a specific bucket is located, the inner loop traverses all the cells in the bucket, or all the slots, that is, bucketcnt = 8 slots. The whole cycle process:

Detailed explanation of the best [golang] map of the year

Let’s talk about mintophash again. When the tophash value of a cell is less than mintophash, it indicates the migration state of the cell. Because this state value is placed in the tophash array, in order to distinguish it from the normal hash value, an increment will be given to the hash value calculated by key: mintophash. In this way, the normal top hash value can be distinguished from the hash value representing the state.

The following states represent the bucket situation:

//An empty cell is also the initial state of the bucket
empty          = 0
//An empty cell indicates that the cell has been migrated to a new bucket
evacuatedEmpty = 1
//Key and value have been relocated, but the key is in the first half of the new bucket,
//We'll talk about the expansion later.
evacuatedX     = 2
//Same as above, key in the second half
evacuatedY     = 3
//The minimum normal value of hash
minTopHash     = 4

In the source code, it is used to determine whether the bucket has been relocated

func evacuated(b *bmap) bool {
    h := b.tophash[0]
    return h > empty && h < minTopHash
}

Only the first value of tophash array is taken to judge whether it is between 0 and 4. Compared with the above constant, when top hash isevacuatedEmptyevacuatedXevacuatedYOne of these three values indicates that all the keys in this bucket have been moved to the new bucket.

Two get operations of map

There are two kinds of syntax to read map in go language: with and without comma. When the key to be queried is not in the map, the usage with comma will return a bool type variable to indicate whether the key is in the map, while the statement without comma will return a zero value of value type. If value is of type int, it returns 0; if value is of type string, it returns an empty string.

package main

import "fmt"

func main() {
    ageMap := make(map[string]int)
    ageMap["qcrao"] = 18

    //No comm usage
    age1 := ageMap["stefno"]
    fmt.Println(age1)

    //Usage with comma
    age2, ok := ageMap["stefno"]
    fmt.Println(age2, ok)
}

Operation results:

0
0 false

I always thought it was amazing. How did it come true? This is actually what the compiler does behind the scenes: after analyzing the code, it corresponds the two syntax to two different functions at the bottom layer.

// src/runtime/hashmap.go
func mapaccess1(t *maptype, h *hmap, key unsafe.Pointer) unsafe.Pointer
func mapaccess2(t *maptype, h *hmap, key unsafe.Pointer) (unsafe.Pointer, bool)

In the source code, the function name is not restricted to small sections, directly with the suffix 1, 2, completely ignoring the “code encyclopedia” in that set of naming practices. The difference can be seen from the statements of the above two functions,mapaccess2If a bool type variable is added to the return value of the function, the codes of the two are exactly the same, except that a false or true is added after the return value.

In addition, depending on the type of key, the compiler will replace the search, insert, and delete functions with more specific functions to optimize efficiency

Key type

lookup

uint32

mapaccess1_fast32(t *maptype, h *hmap, key uint32) unsafe.Pointer

uint32

mapaccess2_fast32(t *maptype, h *hmap, key uint32) (unsafe.Pointer, bool)

uint64

mapaccess1_fast64(t *maptype, h *hmap, key uint64) unsafe.Pointer

uint64

mapaccess2_fast64(t *maptype, h *hmap, key uint64) (unsafe.Pointer, bool)

string

mapaccess1_faststr(t *maptype, h *hmap, ky string) unsafe.Pointer

string

mapaccess2_faststr(t *maptype, h *hmap, ky string) (unsafe.Pointer, bool)

The parameter types of these functions are uint32, unt64 and string directly. Because the type of key is known in advance, the memory layout is very clear, so it can save a lot of operations and improve efficiency.

These functions are all in the filesrc/runtime/hashmap_fast.goInside.

How to expand the capacity

The purpose of using hash table is to quickly find the target key. However, with more and more keys added to the map, the probability of key collision is also increasing. The eight cells in the bucket will be gradually filled, and the efficiency of finding, inserting and deleting keys will be lower and lower. The ideal situation is that only one key is installed in a bucket. In this way, it can be achievedO(1)However, the space consumption is too large and the cost of exchanging space for time is too high.

In fact, when the gokey is used to locate the location of a bucket in a certain time, it needs to change the location of gokey to another space.

Of course, this requires a degree. Otherwise, all keys fall into the same bucket and degenerate into a linked list. The efficiency of various operations is directly reduced to o (n), which is not feasible.

Therefore, there needs to be an indicator to measure the situation described above, which isLoading factor。 Go source code in this definitionLoading factor

loadFactor := count / (2^B)

Count is the number of elements in the map, and 2 ^ B is the number of buckets.

Let’s talk about the timing of triggering map expansion: when a new key is inserted into the map, condition detection will be performed. If the following two conditions are met, the expansion will be triggered:

  1. The loading factor exceeds the threshold. The threshold defined in the source code is 6.5.
  2. There are too many buckets in overflow: when B is less than 15, that is, the total number of buckets is less than 2 ^ 15, if the number of overflow buckets exceeds 2 ^ B; when b > = 15, that is, the total number of buckets 2 ^ B is greater than or equal to 2 ^ 15, if the number of overflow buckets exceeds 2 ^ 15.

Through assembly language, we can find that the function in the source code corresponding to the assignment operation ismapassignThe source code corresponding to the expansion conditions is as follows:

// src/runtime/hashmap.go/mapassign

//Trigger expansion opportunity
if !h.growing() && (overLoadFactor(int64(h.count), h.B) || tooManyOverflowBuckets(h.noverflow, h.B)) {
        hashGrow(t, h)
    }

//The loading factor is more than 6.5
func overLoadFactor(count int64, B uint8) bool {
    return count >= bucketCnt && float32(count) >= loadFactor*float32((uint64(1)<<B))
}

//Too many overflow buckets
func tooManyOverflowBuckets(noverflow uint16, B uint8) bool {
    if B < 16 {
        return noverflow >= uint16(1)<<B
    }
    return noverflow >= 1<<15
}

Explain:

Point 1: we know that each bucket has 8 vacancies. If there is no overflow and all buckets are full, the result of the loading factor is 8. Therefore, when the load factor exceeds 6.5, it indicates that many buckets are almost full, and the search efficiency and insertion efficiency become low. It is necessary to expand the capacity at this time.

Point 2: it is a supplement to point 1. That is to say, when the load factor is small, the efficiency of map search and insertion is also very low, and the first point can not identify this situation. The surface phenomenon is that the number of molecules to calculate the loading factor is small, that is, the total number of elements in the map is small, but the number of buckets is large (the number of real allocated buckets is large, including a large number of overflow buckets).

It’s not hard to imagine the reason for this: constantly inserting and deleting elements. Many elements are inserted first, resulting in the creation of many buckets. However, the load factor fails to reach the critical value of point 1, and the expansion is not triggered to alleviate this situation. After that, delete the elements, reduce the total number of elements, and insert many elements, resulting in the creation of many overflow buckets, but it will not violate the provisions of point 1. What can you do with me? Because the number of overflow buckets is too large, the keys will be scattered, and the efficiency of searching and inserting is frightening. Therefore, the second regulation is issued. It’s like an empty city. There are a lot of houses, but there are few households. They are scattered. It’s very difficult to find people.

For the limit of hit conditions 1 and 2, the expansion will occur. But the strategy of capacity expansion is not the same, after all, the two conditions deal with different scenarios.

For condition 1, there are too many elements and too few buckets. It is very simple: add 1 to B, and the maximum number of buckets (2 ^ b) will directly become twice the number of original buckets. So there are new and old buckets. Note that at this time, the elements are all in the old bucket and have not been migrated to the new bucket. Moreover, the new bucket only changes the maximum quantity to twice the original maximum quantity (2 ^ b * 2).

For condition 2, in fact, there are not so many elements, but the number of overflow buckets is particularly large, indicating that many buckets are not full. The solution is to create a new bucket space and move the elements in the old bucket to the new bucket, so that the keys in the same bucket are arranged more closely. In this way, the key in the overflow bucket can be moved to the bucket. The result is to save space and improve the utilization rate of bucket, and the efficiency of map search and insertion will naturally be improved.

For the solution of condition 2, Cao Da’s blog also puts forward an extreme situation: if all the key hashes inserted into the map are the same, they will fall into the same bucket, and if there are more than 8, the overflow bucket will be generated, and the result will be too many overflow buckets. In fact, moving elements can’t solve the problem, because at this time, the whole hash table has degenerated into a linked list, and the operation efficiency has changedO(n)

Let’s take a look at how to expand the capacity. Due to the expansion of the map, the original key / value needs to be relocated to a new memory address. If a large number of key / values need to be relocated, the performance will be greatly affected. Therefore, the expansion of go map adopts a local style called “progressive”. The original key will not be relocated at one time, and only two buckets will be moved at most each time.

It says ithashGrow()The function doesn’t actually “move”, it just allocates new buckets and hooks the old buckets to the old buckets field. The action of moving buckets isgrowWork()Function, and call thegrowWork()The action of the function is in the mapassign and mapdelete functions. In other words, when you insert, modify, or delete a key, you will try to move buckets. First, check whether the old baskets have been relocated. Specifically, check whether the old baskets are nil.

Let’s look firsthashGrow()Function to see how to move buckets.

func hashGrow(t *maptype, h *hmap) {
    //B + 1 is equivalent to twice the original space
    bigger := uint8(1)

    //Corresponding condition 2
    if !overLoadFactor(int64(h.count), h.B) {
        //For the same amount of memory expansion, so B does not change
        bigger = 0
        h.flags |= sameSizeGrow
    }
    //Hook old buckets to buckets
    oldbuckets := h.buckets
    //Apply for a new bucket space
    newbuckets, nextOverflow := makeBucketArray(t, h.B+bigger)

    flags := h.flags &^ (iterator | oldIterator)
    if h.flags&iterator != 0 {
        flags |= oldIterator
    }
    //Submit the action of grow
    h.B += bigger
    h.flags = flags
    h.oldbuckets = oldbuckets
    h.buckets = newbuckets
    //The relocation progress is 0
    h.nevacuate = 0
    //The number of overflow buckets is 0
    h.noverflow = 0

    // ……
}

It mainly applies for a new bucket space and processes the relevant flag bits: for example, the flag nevacuate is set to 0, which indicates that the current relocation progress is 0.

It is worth saying that it is righth.flagsTreatment of:

flags := h.flags &^ (iterator | oldIterator)
if h.flags&iterator != 0 {
    flags |= oldIterator
}

Here’s the operator: & ^. It’s calledBy position 0Operator. For example:

x = 01010011
y = 01010100
z = x &^ y = 00000011

If the Y bit is 1, then the result Z corresponding bit is 0, otherwise Z corresponding bit is the same as X corresponding bit value.

Therefore, the meaning of the above code for “one meal” operation on flags is: first clear the corresponding bits of iterator and olditerator in h.flags to 0, and then if it is found that the iterator bit is 1, it is transferred to the olditerator bit, so that the olditerator flag bit becomes 1. The subtext is: buckets are now in the name of oldbuckets, and the corresponding flag bit will be transferred.

Several flag bits are as follows:

//There may be iterators using buckets
iterator     = 1
//There may be iterators using oldboxes
oldIterator  = 2
//A coroutine is writing a key to the map
hashWriting  = 4
//Equal capacity expansion (corresponding to condition 2)
sameSizeGrow = 8

Let’s look at the growwork() function that actually performs the relocation.

func growWork(t *maptype, h *hmap, bucket uintptr) {
    //Confirm that the old bucket corresponds to the one in use
    evacuate(t, h, bucket&h.oldbucketmask())

    //Move another bucket to speed up the relocation process
    if h.growing() {
        evacuate(t, h, h.nevacuate)
    }
}

h. The growing() function is very simple:

func (h *hmap) growing() bool {
    return h.oldbuckets != nil
}

IfoldbucketsIf it is not empty, it means that the relocation has not been completed, and it has to be continued.

bucket&h.oldbucketmask()This line of code, as stated in the source code comment, is to confirm that the relocated bucket is the one we are using.oldbucketmask()The function returns the bucket mask of the map before the expansion.

The function of the so-called bucketmask is to compare the hash value calculated by the key with the bucket mask, and the result is the bucket that the key should fall into. For example, if B = 5, then the lower 5 bits of bucketmask are11111, the rest are0In other words, only the lower five bits of the hash value decide which bucket the key falls into.

Next, we focus all our attention on the key function of relocation, evaluate. Source paste in the following, do not be nervous, I will add a large area of comments, through the annotation is absolutely understandable. After that, I will elaborate on the relocation process.

The source code is as follows:

func evacuate(t *maptype, h *hmap, oldbucket uintptr) {
    //Locate the old bucket address
    b := (*bmap)(add(h.oldbuckets, oldbucket*uintptr(t.bucketsize)))
    //The result is 2 ^ B, if B = 5, the result is 32
    newbit := h.noldbuckets()
    //Hash function of key
    alg := t.key.alg
    //If B has not been relocated
    if !evacuated(b) {
        var (
            //Represents the destination address of the bucket move
            x, y   *bmap
            //Point to key / Val in X, y
            xi, yi int
            //Point to the key in X, y
            xk, yk unsafe.Pointer
            //Point to value in X, y
            xv, yv unsafe.Pointer
        )
        //The default is to expand the capacity by the same size, and the previous and subsequent bucket numbers remain unchanged
        //Use X for relocation
        x = (*bmap)(add(h.buckets, oldbucket*uintptr(t.bucketsize)))
        xi = 0
        xk = add(unsafe.Pointer(x), dataOffset)
        xv = add(xk, bucketCnt*uintptr(t.keysize))、

        //If the capacity is not expanded by the same size, the number of the bucket before and after will change
        //Use y for relocation
        if !h.sameSizeGrow() {
            //The bucket number represented by Y has been increased by 2 ^ B
            y = (*bmap)(add(h.buckets, (oldbucket+newbit)*uintptr(t.bucketsize)))
            yi = 0
            yk = add(unsafe.Pointer(y), dataOffset)
            yv = add(yk, bucketCnt*uintptr(t.keysize))
        }

        //Traverse all buckets, including overflow buckets
        //B is the old bucket address
        for ; b != nil; b = b.overflow(t) {
            k := add(unsafe.Pointer(b), dataOffset)
            v := add(k, bucketCnt*uintptr(t.keysize))

            //Traverse all cells in the bucket
            for i := 0; i < bucketCnt; i, k, v = i+1, add(k, uintptr(t.keysize)), add(v, uintptr(t.valuesize)) {
                //Top hash value of current cell
                top := b.tophash[i]
                //If the cell is empty, there is no key
                if top == empty {
                    //It's a sign that it's been moved
                    b.tophash[i] = evacuatedEmpty
                    //Continue to the next cell
                    continue
                }
                //This is not normally the case
                //Cells that have not been relocated can only be empty or empty
                //Normal top hash (greater than mintophash)
                if top < minTopHash {
                    throw("bad map state")
                }

                k2 := k
                //If the pointer is the dereference
                if t.indirectkey {
                    k2 = *((*unsafe.Pointer)(k2))
                }

                //X is used by default, and the capacity is expanded by the same amount
                useX := true
                //If not equal expansion
                if !h.sameSizeGrow() {
                    //Calculate the hash value, which is the same as when the key is first written
                    hash := alg.hash(k2, uintptr(h.hash0))

                    //If a coroutine is traversing the map
                    if h.flags&iterator != 0 {
                        //If the same key value appears, the calculated hash value is different
                        if !t.reflexivekey && !alg.equal(k2, k2) {
                            //Only appears in the case of nan() of the float variable
                            if top&1 != 0 {
                                //B position 1
                                hash |= newbit
                            } else {
                                //Position B 0
                                hash &^= newbit
                            }
                            //Take the upper 8 bits as the top hash value
                            top = uint8(hash >> (sys.PtrSize*8 - 8))
                            if top < minTopHash {
                                top += minTopHash
                            }
                        }
                    }

                    //Depends on whether the oldB + 1 bit of the new hash value is 0 or 1
                    //See the following article in detail
                    useX = hash&newbit == 0
                }

                //If key moves to part X
                if useX {
                    //Mark the top hash value of the old cell, indicating the move to the X part
                    b.tophash[i] = evacuatedX
                    //If Xi is equal to 8, it will overflow
                    if xi == bucketCnt {
                        //Create a bucket
                        newx := h.newoverflow(t, x)
                        x = newx
                        //Xi counts from 0
                        xi = 0
                        //XK indicates the location to which the key is to be moved
                        xk = add(unsafe.Pointer(x), dataOffset)
                        //XV represents the position where value is to be moved
                        xv = add(xk, bucketCnt*uintptr(t.keysize))
                    }
                    //Set the top hash value
                    x.tophash[xi] = top
                    //Key is a pointer
                    if t.indirectkey {
                        //Copy the original key (which is the pointer) to the new location
                        *(*unsafe.Pointer)(xk) = k2 // copy pointer
                    } else {
                        //Copy the original key (which is the value) to the new location
                        typedmemmove(t.key, xk, k) // copy value
                    }
                    //Value is the pointer, and the operation is the same as key
                    if t.indirectvalue {
                        *(*unsafe.Pointer)(xv) = *(*unsafe.Pointer)(v)
                    } else {
                        typedmemmove(t.elem, xv, v)
                    }

                    //Go to the next cell
                    xi++
                    xk = add(xk, uintptr(t.keysize))
                    xv = add(xv, uintptr(t.valuesize))
                }Else {// key moves to the Y part, and the operation is the same as that of the X part
                    // ……
                    //This part is omitted, and the operation is the same as that of section X
                }
            }
        }
        //If no coprocessor is using old buckets, clear the old buckets and help GC
        if h.flags&oldIterator == 0 {
            b = (*bmap)(add(h.oldbuckets, oldbucket*uintptr(t.bucketsize)))
            //Only clear the key and value parts of the bucket, and keep the top hash part to indicate the relocation status
            if t.bucket.kind&kindNoPointers == 0 {
                memclrHasPointers(add(unsafe.Pointer(b), dataOffset), uintptr(t.bucketsize)-dataOffset)
            } else {
                memclrNoHeapPointers(add(unsafe.Pointer(b), dataOffset), uintptr(t.bucketsize)-dataOffset)
            }
        }
    }

    //Update relocation progress
    //If the bucket of this relocation is equal to the current progress
    if oldbucket == h.nevacuate {
        //Progress plus 1
        h.nevacuate = oldbucket + 1
        // Experiments suggest that 1024 is overkill by at least an order of magnitude.
        // Put it in there as a safeguard anyway, to ensure O(1) behavior.
        //Try to look back at 1024 buckets
        stop := h.nevacuate + 1024
        if stop > newbit {
            stop = newbit
        }
        //Looking for buckets that have not been relocated
        for h.nevacuate != stop && bucketEvacuated(t, h, h.nevacuate) {
            h.nevacuate++
        }
        
        //Now all the buckets before h.nevacuate have been relocated
        
        //All buckets have been relocated
        if h.nevacuate == newbit {
            //Clear old buckets
            h.oldbuckets = nil
            //Clear old overflow bucket
            //Recall that: [0] represents the current overflow bucket
            //[1] indicates old overflow bucket
            if h.extra != nil {
                h.extra.overflow[1] = nil
            }
            //Clear the flag bit that is being expanded
            h.flags &^= sameSizeGrow
        }
    }
}

The code comment of evaluate function is very clear. It is easy to understand the whole relocation process with code and comments. Be patient.

The purpose of relocation is to move old buckets to new ones. According to the previous description, we know that the number of new buckets is twice that of the previous one under condition 1, and the number of new buckets is equal to that of the previous one under condition 2.

For condition 2, if you move from an old bucket to a new bucket, because the number of bucktes remains unchanged, you can move it according to the serial number. For example, when you move to a new location, you will still put it in bucket 0.

For condition 1, it’s not that simple. It is necessary to recalculate the hash of the key to determine which bucket it falls in. For example, the original B = 5, after calculating the hash of the key, we can only look at its lower 5 bits to determine which bucket it falls in. After the expansion, B becomes 6, so you need to look at one more bit. Its lower 6 bits determine which bucket the key falls in. This is calledrehash

Detailed explanation of the best [golang] map of the year

Therefore, the bucket serial number of a key before and after the relocation may be the same as the original, or it may be added with 2 ^ B (the original b value) before and after the relocation, depending on whether the 6th bit of the hash value is 0 or 1.

After understanding the change of bucket number above, we can answer another question: why is traversing map out of order?

After the map is expanded, the keys will be moved. After the relocation, some keys will fly away (the bucket number is added with 2 ^ b). The process of traversal is to traverse the bucket in order and the key in the bucket in order. After the relocation, the location of the key has changed greatly. Some keys fly to the high branch, while others stay in place. In this way, it is impossible to traverse the map in the original order.

Of course, if I only use a hard code map, I will not insert or delete the map. In principle, every time I traverse such a map, I will return a fixed sequence of key / value. It’s true, but go has put an end to this practice, because it will give novice programmers a misunderstanding that this is bound to happen, in some cases, it may lead to big mistakes.

Of course, go is even better. When we traverse a map, we do not start from bucket 0. Each time, we start from a bucket with a random number and traverse from a cell with a random number. In this way, even if you are a dead map and just traverse it, it is unlikely to return a fixed sequence of key / value pairs.

In a word, the feature “the result of iterative map is unordered” was added from go 1.0.

One more problem: if B is increased by 1 after capacity expansion, it means that the total number of buckets is twice as much as the original one, and the original bucket 1 is “fission” to two barrels.

For example, if the original B = 2, the hash values of two keys in bucket 1 are 010110. Since the original B = 2, it is 2 bits lower10It’s decided that they landed in barrel 2, and now B becomes 3, so010110They fall into No.2 and No.6 barrels respectively.

Detailed explanation of the best [golang] map of the year

I understand this. I will use it later when I talk about map iteration.

Let’s talk about several key points in the relocation function

The evaluate function only completes the relocation of one bucket at a time. Therefore, it is necessary to traverse all the cells of the bucket and copy the cell with value to a new place. The bucket also links to the overflow bucket, and they also need to be relocated. Therefore, there are two layers of loops, the outer layer traverses the bucket and overflow bucket, and the inner layer traverses all the cells of the bucket. This loop is everywhere in the map source code, to understand.

The reference to X and Y part in the source code is actually what we said if the capacity is doubled, the number of barrels is twice the original. The first half of the bucket is called X part, and the second half is called y part. The key in a bucket may split into two buckets, one in X part and one in Y part. So we need to know which part key is in the cell before the move. Very simply, recalculate the hash of the key in the cell, and “look forward” one more bit to decide which part to fall into. This is also described in detail above.

There is a special case: there is a key, each time hash is calculated, the result is different. This key ismath.NaN()What does it meannot a number, the type is float64. When it is used as the key of a map, it will encounter a problem when it is relocated: the hash value calculated again is different from that calculated when it was inserted into the map!

As you may have thought, one consequence of this is that the key will never be obtained by the get operation! When I usem[math.NaN()]Statement, is not to find out the results. This key only appears when traversing the entire map. So, you can insert any number ofmath.NaN()As a key.

When relocation meetsmath.NaN()To determine whether to allocate to x part or Y part (if it is twice the number of original buckets after expansion), only the lowest bit of tophash is used. If the lowest bit of tophash is 0, it is assigned to x part; if it is 1, it is assigned to y part.

This is obtained by computing the tophash value with the newly calculated hash value

if top&1 != 0 {
    //The lowest bit of top hash is 1
    //B position 1 of the newly calculated hash value
    hash |= newbit
} else {
    //The B position of the newly calculated hash value is 0
    hash &^= newbit
}

//If the B bit of hash value is 0, it will be moved to x part
//When B = 5, newbit = 32, and the lower 6 bits of binary are 100000
useX = hash&newbit == 0

In fact, I can move such a key to any bucket. Of course, I still need to move to two buckets in the fission diagram above. However, it is beneficial to do so. I will explain it in detail later when I talk about map iteration. For the time being, it is OK to know that the allocation is in this way.

After determining the target bucket to move to, the relocation operation is better. Copy the source key / value value value to the corresponding location of the destination.

The hashkey is set to the original bucketevacuatedXorevacuatedY, indicating that the X part or Y part of the new map has been moved. The tophash of the new map normally takes the upper 8 bits of the key hash value.

The following is a macro look at the changes before and after the expansion.

Before the expansion, B = 2, there are 4 buckets in total, and lowbits represent the low order of hash value. Suppose we don’t focus on other buckets and focus on bucket 2. It is also assumed that there are too many overflows, triggering an equal amount of expansion (corresponding to the previous condition 2).

Detailed explanation of the best [golang] map of the year

After the expansion is completed, the overflow bucket disappears and the keys are concentrated in one bucket, which is more compact and improves the efficiency of searching.

Detailed explanation of the best [golang] map of the year

Suppose a double expansion is triggered. After the expansion is completed, the key in the old buckets is split into two new buckets. One in X part and one in Y part. The basis is the lowbits of hash. In the new map0-3It’s called X part,4-7It’s called y part.

Detailed explanation of the best [golang] map of the year

Note that the two figures above ignore the relocation of other buckets, indicating the situation after all buckets have been relocated. In fact, we know that the relocation is a “gradual” process and will not be completed all at once. Therefore, during the relocation process, the oldbuckets pointer will also point to the old [] BMAP, and the tophash value of the key that has been relocated will be a status value, indicating the relocation destination of the key.

Traversal of map

Originally, the traversal process of the map is relatively simple: traverse all the buckets and the overflow buckets hanging behind it, and then traverse all the cells in the bucket one by one. Each bucket contains 8 cells. Take the key and value from the cell with key, and the process is completed.

However, the reality is not so simple. Remember the expansion process mentioned earlier? The expansion process is not an atomic operation. It only carries two buckets at most at a time. Therefore, if the expansion operation is triggered, the map state will be in an intermediate state for a long time: some buckets have moved to new homes, while others are still in the old places.

Therefore, if traversal occurs in the process of capacity expansion, it will involve traversing the old and new buckets, which is the difficulty.

I first write a simple code example, pretending that I don’t know what function the traversal process calls

package main

import "fmt"

func main() {
    ageMp := make(map[string]int)
    ageMp["qcrao"] = 18

    for name, age := range ageMp {
        fmt.Println(name, age)
    }
}

Execute command:

go tool compile -S main.go

Get the assembly command. Here is not a line by line explanation, you can go to see the previous articles, which are very detailed.

The key lines of assembly code are as follows:

// ......
0x0124 00292 (test16.go:9)      CALL    runtime.mapiterinit(SB)

// ......
0x01fb 00507 (test16.go:9)      CALL    runtime.mapiternext(SB)
0x0200 00512 (test16.go:9)      MOVQ    ""..autotmp_4+160(SP), AX
0x0208 00520 (test16.go:9)      TESTQ   AX, AX
0x020b 00523 (test16.go:9)      JNE     302

// ......

In this way, the underlying function call relationship is clear about map iteration. First, callmapiterinitFunction initializes the iterator and then circulatesmapiternextFunction to iterate the map.

Detailed explanation of the best [golang] map of the year

Structure definition of iterator:

type hiter struct {
    //Key pointer
    key         unsafe.Pointer
    //Value pointer
    value       unsafe.Pointer
    //Map type, including key size, etc
    t           *maptype
    // map header
    h           *hmap
    //The bucket pointed to during initialization
    buckets     unsafe.Pointer
    //BMAP currently traversed
    bptr        *bmap
    overflow    [2]*[]*bmap
    //The bucet number of the initial traversal
    startBucket uintptr
    //Cell number at the beginning of traversal (8 cells in each bucket)
    offset      uint8
    //Has it been traversed from the beginning
    wrapped     bool
    //The size of B
    B           uint8
    //Indicates the current cell number
    i           uint8
    //Points to the current bucket
    bucket      uintptr
    //The bucket that needs to be checked because of capacity expansion
    checkBucket uintptr
}

mapiterinitIt is to initialize and assign values to the fields in the hiter structure.

As mentioned earlier, even if a dead map is traversed, the result is out of order every time. Now we can take a closer look at their implementation.

//Generating random number r
r := uintptr(fastrand())
if h.B > 31-bucketCntBits {
    r += uintptr(fastrand()) << 31
}

//Which bucket does the traversal start from
it.startBucket = r & (uintptr(1)<<h.B - 1)
//Which cell of the bucket starts traversing
it.offset = uint8(r >> h.B & (bucketCnt - 1))

For example, B = 2, thenuintptr(1)<<h.B - 1The result is 3, and the lower 8 bits are0000 0011If you add r to it, you get one0~3Bucket CNT – 1 is equal to 7, and the lower 8 bits are0000 0111If R is shifted to the right by 2 bits, then it is added with 7, and then we can get one0~7The cell of No.

So, in themapiternextFrom the it.startBucket Of it.offset The cell of No. 1 starts to traverse, and takes out the key and value, until it returns to the starting bucket again to complete the traversal process.

The source code part is good-looking to understand, especially after understanding the previous comments of a few pieces of code, and then look at this part of the code, there is no pressure. So, next, I will explain the whole traversal process graphically, hoping to be clear and easy to understand.

Suppose we have a map as shown in the figure below. At the beginning, B = 1, there are two buckets, and then the expansion is triggered (here, do not go into the expansion conditions, it is just a setting), and B becomes 2. In addition, the contents of bucket 1 are moved to a new bucket,No. 1Split intoNo. 1andNo. 3No.0Bucket has not been relocated. The old bucket is hanging in the*oldbucketsAbove the pointer, the new bucket is hung on the*bucketsAbove the pointer.

Detailed explanation of the best [golang] map of the year

At this point, we traverse the map. Suppose that after initialization, startbucket = 3 and offset = 2. Therefore, the starting point of traversal will be cell 2 of bucket 3. The following figure shows the state when traversing begins:

Detailed explanation of the best [golang] map of the year

The red one indicates the starting position, and the traversal order of bucket is 3 > 0 > 1 > 2.

Since bucket 3 corresponds to the old bucket 1, check whether the old bucket 1 has been relocated. The judgment method is as follows:

func evacuated(b *bmap) bool {
    h := b.tophash[0]
    return h > empty && h < minTopHash
}

If the value of B. tophash [0] is in the range of flag value, i.e. in the range of (0,4), it indicates that it has been relocated.

empty = 0
evacuatedEmpty = 1
evacuatedX = 2
evacuatedY = 3
minTopHash = 4

In this case, the old bucket 1 has been relocated. Therefore, its tophash [0] value is in the range of (0,4), so only new bucket 3 is traversed.

Traverse the cell of bucket 3 in turn. At this time, you will find the first non empty key: element E. Here, the mapiternext function returns. At this time, our traversal result has only one element:

Detailed explanation of the best [golang] map of the year

Since the returned key is not empty, the mapiternext function will continue to be called.

Continue to traverse back from the last traversal, and find the element F and element g from the new No. 3 overflow bucket.

As a result, the traversal result set grows

Detailed explanation of the best [golang] map of the year

After traversing the new bucket 3, we return to the new bucket 0. Bucket 0 corresponds to the old bucket 0. After checking, the old bucket 0 has not been relocated. Therefore, the traversal of the new bucket 0 is changed to traversing the old bucket 0. Is it to take out all the keys in the old bucket 0?

It’s not so simple. Recall that the old No.0 bucket will split into two buckets: the new No.0 and the new No.2. What we are traversing at this time is only the new bucket 0 (note that the traversal is traversal*bucketPointer, also known as new buckets. Therefore, we will only take out the keys in old bucket 0 allocated to new bucket 0 after fission.

So,lowbits == 00Will enter the traversal result set:

Detailed explanation of the best [golang] map of the year

As in the previous process, continue to traverse the new bucket 1. It is found that the old bucket 1 has been relocated. Just traverse the existing elements in the new bucket 1. The result set becomes:

Detailed explanation of the best [golang] map of the year

Continue to traverse the new bucket 2, which comes from the old bucket 0. Therefore, we need the keys in the old bucket 0 that will split into the new bucket 2, that islowbit == 10The keys.

In this way, the traversal result set becomes:

Detailed explanation of the best [golang] map of the year

Finally, when we continue to traverse to the new No. 3 bucket, we find that all the buckets have been traversed, and the entire iteration process is completed.

By the way, if you run into a keymath.NaN()In this case, the treatment is similar. The core still depends on which bucket it falls into after being split. Just look at the lowest bit of its top hash. If the lowest bit of top hash is 0, it is assigned to x part; if it is 1, it is assigned to y part. According to this, we decide whether to take out the key and put it into the traversal result set.

The core of map traversal is to understand that when the capacity is doubled, the old bucket will split into two new buckets. The traversal operation is performed according to the sequence number of the new bucket. When the old bucket is not relocated, the key to be moved to the new bucket in the old bucket should be found.

The assignment of map

As can be seen from the assembly language, inserting or modifying the key into the map will eventually callmapassignFunction.

In fact, the syntax for inserting or modifying a key is the same, except that the key operated by the former does not exist in the map, while the key operated by the latter exists in the map.

Mapassign has a series of functions. The compiler will optimize it to the corresponding “fast function” according to the key type.

Key type

insert

uint32

mapassign_fast32(t *maptype, h *hmap, key uint32) unsafe.Pointer

uint64

mapassign_fast64(t *maptype, h *hmap, key uint64) unsafe.Pointer

string

mapassign_faststr(t *maptype, h *hmap, ky string) unsafe.Pointer

We only study the most general assignment functionmapassign

On the whole, the process is very simple: calculate the hash value of the key. According to the hash value, according to the previous process, find the position to be assigned (either insert a new key or update the old key), and assign the corresponding position.

The source code is similar to the previous one. The core is a double-layer loop. The outer layer traverses the bucket and its overflow bucket, and the inner layer traverses each cell of the entire bucket. Limited to space, I will not show the comments of this part of the code. If you are interested, you can read it to ensure that you can understand the content of this article.

I would like to make a few important points about this process.

The function first checks the flag bits flags of the map. If the write flag bit of flags is set to 1 at this time, it indicates that other coroutines are performing the “write” operation, which leads to the program panic. This also shows that map is not safe for coroutines.

From the previous article, we know that the expansion is gradual. If the map is in the process of expansion, after the key is located to a certain bucket, it is necessary to ensure that the old bucket corresponding to the bucket has completed the migration process. That is to say, the keys in the old bucket must be migrated to the new bucket (split into two new buckets) before inserting or updating in the new bucket.

The above operation is carried out at the front of the function. Only after the relocation operation is completed, can we safely locate the address of the key to be placed in the new bucket, and then perform the following operations.

Now it’s time to locate where the key should be placed. It’s very important to find your own location. Prepare two pointers, one(inserti)The position of the hash value pointing to the key in the tophash array, and(insertk)Point to the location of the cell (that is, the address where the key is finally placed). Of course, the location of the corresponding value is easy to locate. These three are actually related. The index position in the tophash array determines the position of the key in the whole bucket (8 keys in total), while the position of value needs to “span” the length of 8 keys.

During the loop, inserti and insertk point to the first free cell found, respectively. If no key is found in the map later, that is to say, there is no such key in the original map, which means to insert a new key. The final location of the key is the “empty” first found.

If the eight keys of the bucket are full, after jumping out of the loop, it is found that both inserti and insertk are empty. At this time, you need to hang the overflow bucket behind the bucket. Of course, it is also possible to put an overflow bucket after the overflow bucket. This means that too many key hashes arrive in this bucket.

Before the key is officially installed, check the status of the map to see if it needs to be expanded. If the expansion conditions are met, an expansion operation will be triggered.

After that, the whole process of finding and locating the key has to go again. Because after the expansion, the distribution of keys has changed.

Finally, the map related values will be updated. If a new key is inserted, the count value of the map’s element number field will be increased by 1; it is set at the beginning of the functionhashWritingWrite the flag will clear.

In addition, there is an important point to make. In fact, it is not accurate to find the location of the key and perform the assignment operation. Let’s seemapassignThe prototype of the function knows that the function does not pass in a value value, so when is the assignment performed?

func mapassign(t *maptype, h *hmap, key unsafe.Pointer) unsafe.Pointer

The answer has to be found in assembly language. I will disclose the answer directly. If you are interested, I can study it in private.mapassignKey, the value of the corresponding function is the location of the return function.

Deletion of map

The underlying execution function of the write operation ismapdelete

func mapdelete(t *maptype, h *hmap, key unsafe.Pointer) 

Depending on the key type, the deletion operation will be optimized to a more specific function

Key type

delete

uint32

mapdelete_fast32(t *maptype, h *hmap, key uint32)

uint64

mapdelete_fast64(t *maptype, h *hmap, key uint64)

string

mapdelete_faststr(t *maptype, h *hmap, ky string)

Of course, we only caremapdeleteFunction. It first checks the h.flags flag, and if it finds that the write flag is 1, it directly panics, because it indicates that there are other coroutines writing at the same time.

Calculate the hash of the key to find the dropped bucket. Check this map. If it is in the process of capacity expansion, a relocation operation will be triggered directly.

The deletion operation is also a two-layer loop, and the core is to find the specific location of the key. The search process is similar. Search cells in the bucket.

After finding the corresponding location, reset the key or value

//Clear key
if t.indirectkey {
    *(*unsafe.Pointer)(k) = nil
} else {
    typedmemclr(t.key, k)
}

//Clear value
if t.indirectvalue {
    *(*unsafe.Pointer)(v) = nil
} else {
    typedmemclr(t.elem, v)
}

Finally, subtract the count value by 1 and set the tophash value of the corresponding position toEmpty

This source code is also relatively simple, feeling rise directly to see the code.

Map advanced

Can I delete while traversing

Map is not a thread safe data structure. Reading and writing a map at the same time is an undefined behavior. If it is detected, it will pan directly.

Generally speaking, this can be solved by read-write lock:sync.RWMutex

Call before readRLock()Function, called after reading.RUnlock()Function to unlock; write before calling.Lock()Function, after writing, callUnlock()Unlock.

Besides,sync.MapIs a thread safe map, which can also be used. The principle of its implementation will not be mentioned this time.

Can key be float type?

Grammatically, yes. In go language, any type that is comparable can be used as a key. Except for slice, map and functions, all other types are OK. Specifically, it includes: Boolean value, number, string, pointer, channel, interface type, structure, and array containing only the above types. The common feature of these types is support==and!=Operator,k1 == k2It can be considered that K1 and K2 are the same key. If they are structs, they are considered to be the same key only if their field values are equal.

By the way, any type can be a value, including the map type.

Take an example:

func main() {
    m := make(map[float64]int)
    m[1.4] = 1
    m[2.4] = 2
    m[math.NaN()] = 3
    m[math.NaN()] = 3

    for k, v := range m {
        fmt.Printf("[%v, %d] ", k, v)
    }

    fmt.Printf("\nk: %v, v: %d\n", math.NaN(), m[math.NaN()])
    fmt.Printf("k: %v, v: %d\n", 2.400000000001, m[2.400000000001])
    fmt.Printf("k: %v, v: %d\n", 2.4000000000000000000000001, m[2.4000000000000000000000001])

    fmt.Println(math.NaN() == math.NaN())
}

Program output:

[2.4, 2] [NaN, 3] [NaN, 3] [1.4, 1] 
k: NaN, v: 0
k: 2.400000000001, v: 0
k: 2.4, v: 2
false

In the example, a map whose key type is float type is defined and four keys are inserted into it: 1.4, 2.4, Nan and Nan.

Four keys are also printed out when printing. If you know Nan! = Nan, it’s not surprising. Because the results they compare are not equal. Naturally, in the view of map, they are two different keys.

Next, we query several keys and find that Nan does not exist, 2.400000000001 does not exist, and 2.4000000000001 does.

It’s a little weird, isn’t it?

Then, through the compilation, I found the following facts:

When float64 is used as the key, it should be converted to unit64 type first, and then inserted into the key.

Specifically throughFloat64frombitsFunction completion:

// Float64frombits returns the floating point number corresponding
// the IEEE 754 binary representation b.
func Float64frombits(b uint64) float64 { return *(*float64)(unsafe.Pointer(&b)) }

In other words, the floating-point number is expressed in the format specified by IEEE 754. Such as assignment statement:

0x00bd 00189 (test18.go:9)      LEAQ    "".statictmp_0(SB), DX
0x00c4 00196 (test18.go:9)      MOVQ    DX, 16(SP)
0x00c9 00201 (test18.go:9)      PCDATA  $0, $2
0x00c9 00201 (test18.go:9)      CALL    runtime.mapassign(SB)

"".statictmp_0(SB)The variable is like this:

"".statictmp_0 SRODATA size=8
        0x0000 33 33 33 33 33 33 03 40
"".statictmp_1 SRODATA size=8
        0x0000 ff 3b 33 33 33 33 03 40
"".statictmp_2 SRODATA size=8
        0x0000 33 33 33 33 33 33 03 40

Let’s export something more:

package main

import (
    "fmt"
    "math"
)

func main() {
    m := make(map[float64]int)
    m[2.4] = 2

    fmt.Println(math.Float64bits(2.4))
    fmt.Println(math.Float64bits(2.400000000001))
    fmt.Println(math.Float64bits(2.4000000000000000000000001))
}


4612586738352864255
4612586738352862003
4612586738352862003

To hexadecimal:

0x4003333333333333
0x4003333333333BFF
0x4003333333333333

And the front one"".statictmp_0By comparison, it’s clear.2.4and2.4000000000000000000000001aftermath.Float64bits()The result of function conversion is the same. Naturally, in the view of map, they are the same key.

Let’s take a look at Nan (not a number)

// NaN returns an IEEE 754 ``not-a-number'' value.
func NaN() float64 { return Float64frombits(uvnan) }

Uvan is defined as:

uvnan    = 0x7FF8000000000001

Nan() calls directlyFloat64frombits, pass in the write dead const type variable0x7FF8000000000001And get the Nan type value. Since Nan is parsed from a constant, why is it considered as a different key when inserting a map?

This is determined by the hash function of the type. For example, for a 64 bit floating-point number, its hash function is as follows:

func f64hash(p unsafe.Pointer, h uintptr) uintptr {
    f := *(*float64)(p)
    switch {
    case f == 0:
        return c1 * (c0 ^ h) // +0, -0
    case f != f:
        return c1 * (c0 ^ h ^ uintptr(fastrand())) // any kind of NaN
    default:
        return memhash(p, h, 8)
    }
}

The second case,f != fIt’s aimed atNANI’ll add another random number here.

In this way, all the puzzles are solved.

Due to the characteristics of Nan:

NAN != NAN
hash(NAN) != hash(NAN)

Therefore, when the key searched in the map is Nan, nothing can be found; if you add four nan to it, the traversal will get four NANs.

Finally, the conclusion is: float type can be used as a key, but due to the accuracy problem, it will lead to some strange problems, so be careful to use it.

summary

At the time of writing this article, I have read all the blogs in the Chinese world and can’t find the answers. Of course, the source code can answer any question. However, you can’t jump into the details of the source code, you have to have a whole understanding of it.

Therefore, I began to search English related articles about source code, not too many in this respect. However, I found a high-quality article in the first reference. It led the readers to optimize step by step, and finally realized the random extraction of a key from the map. I recommend you to read. It’s wonderful. This is especially true when you know the underlying traversal and expansion process of the map.

To sum up, in go language, map is realized by hash lookup table, and hash conflict is solved by linked list method.

The key is scattered into different buckets through the hash value of the key. There are 8 cells in each bucket. The low bit of the hash value determines the bucket number, and the high bit identifies different keys in the same bucket.

When many keys are added to the bucket, resulting in too many elements or too many overflow buckets, the expansion will be triggered. Capacity expansion is divided into equal capacity expansion and double capacity expansion. After the expansion, the key in the original bucket is divided into two and will be redistributed to two buckets.

The process of capacity expansion is gradual, mainly to prevent the number of keys that need to be relocated at a time to cause performance problems. When a new element is added to trigger the expansion, the time for bucket relocation occurs during the period of assignment and deletion. At most two buckets are relocated each time.

The core content of finding, assigning and deleting is how to locate the key, which needs to be understood. Once understood, the source code of map can be understood.

Finally, if the article is helpful to you, please help me to share, or click to see, thank you!

Finally, click to read the original text, and you may be involved in witnessing a thousand Star project from scratch.

reference material

[how to select a map key randomly in English is very wonderful] https://lukechampine.com/hackmap.html

[Wikipedia of map] https://en.wikipedia.org/wiki/Associative_ array

[ sync.map Source code analysis] https://github.com/Chasiny/Blog/blob/master/blog/go/sync.Map Source code analysis.md

Flow chart of various map related operations https://www.jianshu.com/p/aa0d4808cbb8

[map source code analysis] https://www.twblogs.net/a/5bd78d5d2b71777ac86b541f

[there is no need to explain Cao Da’s article on map] https://github.com/cch123/golang-notes/blob/master/map.md

[picture in English] https://www.ardanlabs.com/blog/2013/12/macro-view-of-map-internals-in-go.html

[English compares the map implementation of Java and C + + https://dave.cheney.net/2018/05/29/how-the-go-runtime-implements-maps-efficiently-without-generics

[English why go map is sensitive to competition] https://dave.cheney.net/2015/12/07/are-go-maps-sensitive-to-data-races

【golang blog map】https://blog.golang.org/go-maps-in-action

[random map open source code] https://github.com/lukechampine/randmap

[good picture] https://hacpai.com/article/1533916370874

[night reading issue] https://github.com/developer-learning/reading-go/issues/332

[new blog, very deep] https://draveness.me/golang-hashmap

[expansion process + diagram] https://my.oschina.net/renhc/blog/2208417

[operator] https://juejin.im/post/5c0e572fe51d4522ad6e59d5

【english】https://www.digitalocean.com/community/tutorials/understanding-maps-in-go

[simple description of map traversal source code] https://gocn.vip/article/1704

[passage, it is OK to traverse and delete the key at the same time] https://cloud.tencent.com/developer/article/1065474

[belief oriented programming, golang range] https://draveness.me/golang-for-range

[difference between slice and map as parameters] https://stackoverflow.com/questions/47590444/slice-vs-map-to-be-used-in-parameter/47590531#47590531

[go official blog about map] https://blog.golang.org/go-maps-in-action

[comparable types of go languages] https://golang.org/ref/spec#Comparison_ operators

[key type] http://lanlingzi.cn/post/technical/2016/0904_ Go_ map/

[hash function performance comparison] http://aras-p.info/blog/2016/08/09/More-Hash-Function-Tests/

[hash function selection, C + + / Java comparison] https://studygolang.com/articles/15839

[slice and map as function parameters] https://stackoverflow.com/questions/47590444/slice-vs-map-to-be-used-in-parameter/47590531#47590531

[MAP1 of fried fish blog] https://github.com/EDDYCJY/blog/blob/master/golang/pkg/2019-03-04- In depth understanding of go map initialization and access element.md

[MAP2 of fried fish blog] https://github.com/EDDYCJY/blog/blob/master/golang/pkg/2019-03-24- In depth understanding of go map assignment and expansion migration.md

[definition of hash function] http://zhangshuai.ren/2018/05/16/ Hash algorithm to realize map function of go language/

How to compare two maps to be equal https://golangbot.com/maps/

【NAN hash】https://research.swtch.com/randhash

[description of concurrent security] http://zjykzk.github.io/post/cs/golang/map/