Go operation etcd

Time:2021-6-13

Etcd is a popular open source and distributed key value pair data storage system in recent years, which provides shared configuration, service registration and discovery. This paper mainly introduces the installation and use of etcd.

etcd

Etcd introduction

Etcd is an open source and highly available distributed key value storage system developed by go language, which can be used for configuration sharing and service registration and discovery.

Similar projects include zookeeper and consumer.

Etcd has the following characteristics

  • Full replication: each node in the cluster can use the full archive
  • High availability: etcd can be used to avoid single point of hardware failure or network problems
  • Consistency: each read returns the latest write across multiple hosts
  • Simple: includes a well-defined user oriented API (grpc)
  • Security: automated TLS with optional client certificate authentication is implemented
  • Fast: reference speed of 10000 writes per second
  • Reliable: using raft algorithm to achieve strong consistency and high availability of service storage directory

Etcd application scenarios

Service discovery

Service discovery is also one of the most common problems in distributed system, that is, how can a process or service in the same distributed cluster find each other and establish a connection. In essence, service discovery is to find out whether there are processes in the cluster listening to UDP or TCP ports, and then find and connect by name.

Configuration center

Put some configuration information on etcd for centralized management.

This kind of scenario is usually used as follows: when the application starts, it takes the initiative to obtain the configuration information from the etcd. At the same time, it registers a watcher on the etcd node and waits. Every time the configuration is updated in the future, the etcd will notify the subscriber in real time, so as to obtain the latest configuration information.

Distributed lock

Because etcd uses raft algorithm to maintain the strong consistency of data, the value stored in the cluster for a certain operation must be globally consistent, so it is easy to implement distributed lock. Lock service can be used in two ways, one is to keep exclusive, the other is to control timing.

  • Keep exclusive, that is, all users who acquire locks can get only one. Etcd provides a set of CAS for distributed lock atomic operation(CompareAndSwap)API. By settingprevExistValue to ensure that when multiple nodes create a directory at the same time, only one node succeeds. The successful user can be regarded as having obtained the lock.
  • Control timing, that is, all users who want to obtain locks will be scheduled to execute, butThe order of obtaining locks is also globally unique, and determines the execution order. Etcd also provides a set of API (automatic creation of ordered key) for this purpose. When creating a value for a directory, it is specified asPOSTAction, so etcd will automatically generate a current maximum value as the key in the directory, and store the new value (client number). At the same time, you can also use the API to list all the key values in the current directory in order. In this case, the value of these keys is the timing of the client, and the value stored in these keys can be the number representing the client.

Why use etcd instead of zookeeper?

Zookeeper can implement these functions of etcd. So why use etcd instead of zookeeper?

Why not zookeeper?

  1. The deployment and maintenance is complex, and its usage is complexPaxosStrong consistency algorithm is complex and difficult to understand. The official only providedJavaandCInterface between two languages.
  2. useJavaWriting introduces a lot of dependencies. The maintenance of the operation and maintenance personnel is troublesome.
  3. In recent years, the development is slowetcdandconsulAnd so on.

Why etcd?

  1. Simple. It’s easy to write and deploy in go language; Support HTTP / JSON API, easy to use; Raft algorithm is used to ensure strong consistency and make it easy for users to understand.
  2. Etcd is persistent as soon as the default data is updated.
  3. Etcd supports SSL client security authentication.

Finally, etcd, as a young project, is in high-speed iteration and development, which is both an advantage and a disadvantage. The advantage is that it has unlimited possibilities in the future, and the disadvantage is that it can’t be tested by large projects for a long time. However, for the time beingCoreOSKubernetesandCloudFoundryAnd other well-known projects are used in the production environmentetcdSo in general, etcd is worth trying.

Etcd cluster

Etcd, as a highly available key value storage system, is naturally designed for clustering. Because raft algorithm needs the majority of nodes to vote when making decisions, etcd generally deploys clusters and recommends an odd number of nodes. The recommended number is 3, 5 or 7 nodes to form a cluster.

Build a 3-node cluster example:

In order to distinguish different clusters, it is better to configure a unique token at the same time.

Here is the cluster information defined in advance, wheren1n2andn3Represents three different etcd nodes.

TOKEN=token-01
CLUSTER_STATE=new
CLUSTER=n1=http://10.240.0.17:2380,n2=http://10.240.0.18:2380,n3=http://10.240.0.19:2380

stayn1On this machine, execute the following command to start etcd:

etcd --data-dir=data.etcd --name n1 \
	--initial-advertise-peer-urls http://10.240.0.17:2380 --listen-peer-urls http://10.240.0.17:2380 \
	--advertise-client-urls http://10.240.0.17:2379 --listen-client-urls http://10.240.0.17:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

stayn2On this machine, execute the following command to start etcd:

etcd --data-dir=data.etcd --name n2 \
	--initial-advertise-peer-urls http://10.240.0.18:2380 --listen-peer-urls http://10.240.0.18:2380 \
	--advertise-client-urls http://10.240.0.18:2379 --listen-client-urls http://10.240.0.18:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

stayn3On this machine, execute the following command to start etcd:

etcd --data-dir=data.etcd --name n3 \
	--initial-advertise-peer-urls http://10.240.0.19:2380 --listen-peer-urls http://10.240.0.19:2380 \
	--advertise-client-urls http://10.240.0.19:2379 --listen-client-urls http://10.240.0.19:2379 \
	--initial-cluster ${CLUSTER} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

The etcd official website provides an etcd storage address that can be accessed through the public network. You can get the etcd service directory through the following command and use it as the-discoveryParameter usage.

curl https://discovery.etcd.io/new?size=3
https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92

# grab this token
TOKEN=token-01
CLUSTER_STATE=new
DISCOVERY=https://discovery.etcd.io/a81b5818e67a6ea83e9d4daea5ecbc92


etcd --data-dir=data.etcd --name n1 \
	--initial-advertise-peer-urls http://10.240.0.17:2380 --listen-peer-urls http://10.240.0.17:2380 \
	--advertise-client-urls http://10.240.0.17:2379 --listen-client-urls http://10.240.0.17:2379 \
	--discovery ${DISCOVERY} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}


etcd --data-dir=data.etcd --name n2 \
	--initial-advertise-peer-urls http://10.240.0.18:2380 --listen-peer-urls http://10.240.0.18:2380 \
	--advertise-client-urls http://10.240.0.18:2379 --listen-client-urls http://10.240.0.18:2379 \
	--discovery ${DISCOVERY} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}


etcd --data-dir=data.etcd --name n3 \
	--initial-advertise-peer-urls http://10.240.0.19:2380 --listen-peer-urls http://10.240.0.19:2380 \
	--advertise-client-urls http://10.240.0.19:2379 --listen-client-urls http:/10.240.0.19:2379 \
	--discovery ${DISCOVERY} \
	--initial-cluster-state ${CLUSTER_STATE} --initial-cluster-token ${TOKEN}

The etcd cluster is set up and can be usedetcdctlTo connect etcd.

export ETCDCTL_API=3
HOST_1=10.240.0.17
HOST_2=10.240.0.18
HOST_3=10.240.0.19
ENDPOINTS=$HOST_1:2379,$HOST_2:2379,$HOST_3:2379

etcdctl --endpoints=$ENDPOINTS member lis

Go language operation etcd

Here we use the official etcd / client V3 package to connect etcd and perform related operations.

install

go get go.etcd.io/etcd/clientv3

Put and get operations

putCommand is used to set the key value pair data,getThe command is used to get the value based on the key.

package main

import (
	"context"
	"fmt"
	"time"

	"go.etcd.io/etcd/clientv3"
)

// etcd client put/get demo
// use etcd/clientv3

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"127.0.0.1:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		// handle error!
		fmt.Printf("connect to etcd failed, err:%v\n", err)
		return
	}
    fmt.Println("connect to etcd success")
	defer cli.Close()
	// put
	ctx, cancel := context.WithTimeout(context.Background(), time.Second)
	_, err = cli.Put(ctx, "q1mi", "dsb")
	cancel()
	if err != nil {
		fmt.Printf("put to etcd failed, err:%v\n", err)
		return
	}
	// get
	ctx, cancel = context.WithTimeout(context.Background(), time.Second)
	resp, err := cli.Get(ctx, "q1mi")
	cancel()
	if err != nil {
		fmt.Printf("get from etcd failed, err:%v\n", err)
		return
	}
	for _, ev := range resp.Kvs {
		fmt.Printf("%s:%s\n", ev.Key, ev.Value)
	}
}

Watch operation

watchUsed to get notification of future changes.

package main

import (
	"context"
	"fmt"
	"time"

	"go.etcd.io/etcd/clientv3"
)

// watch demo

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"127.0.0.1:2379"},
		DialTimeout: 5 * time.Second,
	})
	if err != nil {
		fmt.Printf("connect to etcd failed, err:%v\n", err)
		return
	}
	fmt.Println("connect to etcd success")
	defer cli.Close()
	// watch key:q1mi change
	rch := cli.Watch(context.Background(), "q1mi") // 

Save the above code for compilation and execution, and the program will wait for etcdq1miThe change of the key.

For example: we open the terminal and execute the following commands to modify, delete and setq1miThe key.

etcd> etcdctl.exe --endpoints=http://127.0.0.1:2379 put q1mi "dsb2"
OK

etcd> etcdctl.exe --endpoints=http://127.0.0.1:2379 del q1mi
1

etcd> etcdctl.exe --endpoints=http://127.0.0.1:2379 put q1mi "dsb3"
OK

The above program can receive the following notification.

watch>watch.exe
connect to etcd success
Type: PUT Key:q1mi Value:dsb2
Type: DELETE Key:q1mi Value:
Type: PUT Key:q1mi Value:dsb3

Lease lease

package main

import (
	"fmt"
	"time"
)

// etcd lease

import (
	"context"
	"log"

	"go.etcd.io/etcd/clientv3"
)

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"127.0.0.1:2379"},
		DialTimeout: time.Second * 5,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("connect to etcd success.")
	defer cli.Close()

	//Create a 5 second lease
	resp, err := cli.Grant(context.TODO(), 5)
	if err != nil {
		log.Fatal(err)
	}

	//After five seconds, the / Na Zha / key will be removed
	_, err = cli.Put(context.TODO(), "/nazha/", "dsb", clientv3.WithLease(resp.ID))
	if err != nil {
		log.Fatal(err)
	}
}

keepAlive

package main

import (
	"context"
	"fmt"
	"log"
	"time"

	"go.etcd.io/etcd/clientv3"
)

// etcd keepAlive

func main() {
	cli, err := clientv3.New(clientv3.Config{
		Endpoints:   []string{"127.0.0.1:2379"},
		DialTimeout: time.Second * 5,
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println("connect to etcd success.")
	defer cli.Close()

	resp, err := cli.Grant(context.TODO(), 5)
	if err != nil {
		log.Fatal(err)
	}

	_, err = cli.Put(context.TODO(), "/nazha/", "dsb", clientv3.WithLease(resp.ID))
	if err != nil {
		log.Fatal(err)
	}

	// the key 'foo' will be kept forever
	ch, kaerr := cli.KeepAlive(context.TODO(), resp.ID)
	if kaerr != nil {
		log.Fatal(kaerr)
	}
	for {
		ka := 

Implementation of distributed lock based on etcd

go.etcd.io/etcd/clientv3/concurrencyImplement concurrent operations on etcd, such as distributed lock, barrier and election.

Import the package:

import "go.etcd.io/etcd/clientv3/concurrency"

An example of distributed lock based on etcd

cli, err := clientv3.New(clientv3.Config{Endpoints: endpoints})
if err != nil {
    log.Fatal(err)
}
defer cli.Close()

//Create two separate sessions to demonstrate lock contention
s1, err := concurrency.NewSession(cli)
if err != nil {
    log.Fatal(err)
}
defer s1.Close()
m1 := concurrency.NewMutex(s1, "/my-lock/")

s2, err := concurrency.NewSession(cli)
if err != nil {
    log.Fatal(err)
}
defer s2.Close()
m2 := concurrency.NewMutex(s2, "/my-lock/")

//Session S1 acquire lock
if err := m1.Lock(context.TODO()); err != nil {
    log.Fatal(err)
}
fmt.Println("acquired lock for s1")

m2Locked := make(chan struct{})
go func() {
    defer close(m2Locked)
    //Wait until session S1 releases the lock of / my lock /
    if err := m2.Lock(context.TODO()); err != nil {
        log.Fatal(err)
    }
}()

if err := m1.Unlock(context.TODO()); err != nil {
    log.Fatal(err)
}
fmt.Println("released lock for s1")

Output:

acquired lock for s1
released lock for s1
acquired lock for s2

See the documentation to learn more

Other operations

For other operations, please refer to etcd / clientv3 official documents.

Reference link:

  • https://etcd.io/docs/v3.3.12/demo/
  • https://www.infoq.cn/article/etcd-interpretation-application-scenario-implement-principle/