Super detailed! Introduction and practice of CODIS



In 3.0, redis launched a cluster solution: redis cluster. It allows us to build our redis cluster in an official way. But before, our production environment was based on the instance of redis 2. X version. If cluster management is needed, take vipshop as an example, the solution adopted by many colleagues is Twitter’s tweetroxy.

Twitter itself is a static distributed scheme. When we expand and shrink the capacity, we have high requirements for Devops, and it is difficult to achieve smooth expansion and shrink. And there is no dashboard for cluster management, which is very inconvenient.

In order to solve these pain points, Peapod open source their CODIS, thank the authors for bringing such a good tool, whether it is to use or learn cap related development, CODIS is a good reference.

CODIS official project address:…

CODIS architecture

Super detailed! Introduction and practice of CODIS

The figure above is an overall architecture of CODIS. We can get several key components: CODIS server, CODIS proxy, CODIS dashboard, CODIS admin, CODIS Fe, COIs ha, etc.

  • CODIS server: Based on redis-2.8.21 branch development. Additional data structures are added to support slot related operations and data migration instructions. For specific modification, please refer to the modification of redis.
  • CODIS proxy: a client connected redis proxy service, which implements the redis protocol. Except that some commands are not supported (list of unsupported commands), there is no difference between the performance and native redis (like tweetproxy).
  • For the same business cluster, multiple CODIS proxy instances can be deployed at the same time;
  • The state synchronization between different CODIS proxies is guaranteed by CODIS dashboard.
  • CODIS dashboard: a cluster management tool, which supports the addition, deletion and data migration of CODIS proxy and CODIS server. When the cluster state changes, CODIS dashboard maintains the consistency of all CODIS proxy states in the cluster.
  • For the same business cluster, there can only be 0 or 1 CODIS dashboard at the same time;
  • All changes to the cluster must be done through CODIS dashboard.
  • CODIS admin: a command line tool for cluster management.
  • It can be used to control the state of CODIS proxy, CODIS dashboard and access external storage.
  • CODIS Fe: cluster management interface.
  • Multiple cluster instances can share the same front-end display page;
  • Through the configuration file management back-end CODIS dashboard list, the configuration file can be automatically updated.
  • CODIS ha: provides high availability for clusters.
  • Depending on the instance of CODIS dashboard, it can automatically capture the status of each component in the cluster;
  • According to the current status of the cluster, the master-slave switch strategy is automatically generated, and the master-slave switch is completed through CODIS dashboard when necessary.
  • Storage: provides external storage for cluster state.
  • Provide the concept of namespace, and different clusters will be organized according to different product names;
  • At present, only zookeeper and etcd are provided, but an abstract interface is provided, which can be extended by itself.

Understanding the meaning of the above components is very helpful for us to deploy and maintain CODIS, so we must carefully understand the meaning of the above components before building the environment.

CODIS and configuration of dependent environment

CODIS is mainly developed by golang, and relies on ZK or etcd for configuration management (as a gopher, I mainly use etcd), so at least, we need to build up the go environment first.

1. Construction of go development environment

It is recommended to refer to the official guidelines:

After the installation, we check whether it is successful:

$ go version
go version go1.7 linux/amd64
2. Go compilation environment settings

Add to path, for example:

export GOROOT=/usr/local/go
export GOPATH=/home/apps/devgo
export GOBIN=/home/apps/devgo/bin
export PATH=$PATH:$GOROOT/bin
3. Install godep
$ go get -u

If you enter the command “command not found” in which goal, you can solve the problem as follows:

$ sudo cp $GOPATH/bin/godep  /usr/local/bin/
4. Install CODIS
$ mkdir -p $GOPATH/src/
$ cd $_ && git clone -b release3.0

After the project is checked out successfully, we can see the complete directory of CODIS; because the extern of CODIS contains a basic tool of redis. Therefore, we need to execute the make command in the main directory of CODIS

$ cd $GOPATH/src/
$ make
make -j -C extern/redis-2.8.21/
... ...
go build -i -o bin/codis-dashboard ./cmd/dashboard
go build -i -o bin/codis-proxy ./cmd/proxy
go build -i -o bin/codis-admin ./cmd/admin
go build -i -o bin/codis-ha ./cmd/ha
go build -i -o bin/codis-fe ./cmd/fe
$ ls bin/
The total dosage is less than 74528
Drwxrwxr-x 3 apps 4096 Oct 2 09:52
Drwxrwxr-x 12 apps 4096 Oct 2 09:51
Drwxrwxr-x 4 apps 4096 Oct 2 09:52 assets
-Rwxrwxr-x 1 apps 15710271 Oct 2 09:52 CODIS admin
-Rwxrwxr-x 1 apps 16978383 Oct 2 09:52 CODIS dashboard
-Rwxrwxr-x 1 apps 15810040 Oct 2 09:52 CODIS Fe
-Rwxrwxr-x 1 apps 8730288 Oct 2 09:52 CODIS ha
-Rwxrwxr-x 1 apps apps 10267404 Oct 2 09:52 CODIS proxy
-Rwxrwxr-x 1 apps 4308197 Oct 2 09:52 CODIS server
-Rwxrwxr-x 1 apps apps 2166709 Oct 2 09:52 redis benchmark
-Rwxrwxr-x 1 apps 2314614 Oct 2 09:52 redis cli
-Rw-rw-r -- 1 apps apps 146 Oct 2 09:52 version
... ...
$ cat bin/version
version = 2016-09-29 13:58:59 +0800 @29199bb81e7b0b3cdb4bd7e005c96c9fd674a6ea
compile = 2016-10-02 09:52:18 +0800 by go version go1.7 linux/amd64

Next, we need to deal with the command tool under bin /

sudo cp codis-* /usr/local/bin
sudo cp redis-* /usr/local/bin
5. Install etcd

Because I use etcd as configuration management, I need to install etcd. But if you like zookeeper, you can ignore this point and install ZK by yourself

$ curl -L
$ tar xzvf etcd-v2.3.7-linux-amd64.tar.gz
$ cd etcd-v2.3.7-linux-amd64 && sudo cp etcd /usr/local/bin/ &&  cp etcdctl /usr/local/bin/

Check whether etcd installation is successful

$ etcd --version                                                     
etcd Version: 2.3.7
Git SHA: fd17c91
Go Version: go1.6.2
Go OS/Arch: linux/amd64

Running CODIS demo

Source code provides scripts for local testing/ The script generates a local cluster.
migrate slot-[ 512, 767] to group-3
migrate slot-[ 768,1023] to group-4
migrate slot-[   0, 255] to group-1
migrate slot-[ 256, 511] to group-2

The script will output the PID of each process and the log of each instance to the scripts / tmp directory;

After startup, you canhttp:// 8080 to access CODIS Fe.

Super detailed! Introduction and practice of CODIS

Super detailed! Introduction and practice of CODIS

Super detailed! Introduction and practice of CODIS


We have been able to run a demo version of CODIS, but this is not enough. We need to further study the development and operation of CODIS.

User defined parameters and environment construction

In the previous section, we run a demo scheme of CODIS named “CODIS test”. In this section, we try to run our own custom environment through the official related command components.

As in the previous section, I continue to use etcd as my configuration management tool.

Note: Please complete the operation step by step in order.

1、 Start etcd

We use etcd’s default port 2379

nohup etcd --name=codis-demo &>/home/apps/codis/log/etcd/etcd.log &

If the lock exception occurs during startup, you can clean up the relevant history

etcdctl rm --recursive /codis3
2、 Start CODIS server

Let’s start a single node server. CODIS server itself is a redis instance. Let’s start a server instance with port 16379

nohup ./bin/codis-server --port 16379 &>/home/apps/codis/log/redis/redis-16379.log &
[44886] 02 Oct 16:50:11.381 # You requested maxclients of 10000 requiring at least 10032 max file descriptors.
[44886] 02 Oct 16:50:11.381 # Redis can't set maximum open files to 10032 because of OS error: Operation not permitted.
[44886] 02 Oct 16:50:11.381 # Current maximum open files is 4096. maxclients has been reduced to 4064 to compensate for low ulimit. If you need higher maxclients increase 'ulimit -n'.
           _.-``__ ''-._
      _.-``    `.  `_.  ''-._           Redis 2.8.21 (29199bb8/0) 64 bit
  .-`` .-```.  ```/    _.,_ ''-._
 (    '      ,       .-`  | `,    )     Running in stand alone mode
 |`-._`-...-` __...-.``-._|'` _.-'|       Port: 16379
 |    `-._   `._    /     _.-'    |      PID: 44886
  `-._    `-._  `-./  _.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    | 
  `-._    `-._`-.__.-'_.-'    _.-'
 |`-._`-._    `-.__.-'    _.-'_.-'|
 |    `-._`-._        _.-'_.-'    |
  `-._    `-._`-.__.-'_.-'    _.-'
      `-._    `-.__.-'    _.-'
          `-._        _.-'
[44886] 02 Oct 16:50:11.385 # Server started, Redis version 2.8.21
[44886] 02 Oct 16:50:11.385 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
[44886] 02 Oct 16:50:11.385 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled.
[44886] 02 Oct 16:50:11.385 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
[44886] 02 Oct 16:50:11.385 * DB loaded from disk: 0.000 seconds
[44886] 02 Oct 16:50:11.385 * The server is now ready to accept connections on port 16379
3、 Start the dashboard
nohup ./bin/codis-dashboard --config=/home/apps/codis/dashboard.toml --log=/home/apps/codis/log/dashboard/dashboard.log --log-level=WARN &

Here we use the custom dashboard.toml Its contents are as follows:

#                                           #
#                  Codis-Dashboard           #
#                                           #
# Set Coordinator, only accept "zookeeper" & "etcd"
coordinator_name = "etcd"
coordinator_addr = ""
# Set Codis Product {Name/Auth}.
product_name = "codis-demo"
product_auth = ""
# Set bind address for admin(rpc), tcp only.
admin_addr = ""
4、 Start CODIS proxy
nohup ./bin/codis-proxy  --config=/home/apps/codis/proxy.toml --log=/home/apps/codis/log/proxy/proxy.log --log-level=WARN &

proxy.toml The contents are as follows:

product_name = "codis-demo"
product_auth = ""
proto_type = "tcp4"
admin_addr = ""
proxy_addr = ""
5、 Start Fe
nohup ./bin/codis-fe  -d /home/apps/codis/codis.json --listen= &

condis.json The contents are as follows:

        "name": "codis-demo",
        "dashboard": ""
6、 Configure CODIS admin

We have started the basic components above. Next, we need to group our proxy through CODIS admin.

Add proxy group

./bin/codis-admin --dashboard=  --create-group --gid 1

Add group

./bin/codis-admin --dashboard=  --group-add --gid 1 -x

Create proxy

./bin/codis-admin --dashboard=  --create-proxy -x

Configure slot

./bin/codis-admin --dashboard=  --slot-action --interval=100
./bin/codis-admin --dashboard=  --rebalance --confirm

Log in to the management panel, as shown in the following figure:

Super detailed! Introduction and practice of CODIS

Author: domac’s garden
Original text:…

Super detailed! Introduction and practice of CODIS