Go operation NSQ

Time:2021-6-11

NSQ is a popular distributed message queue. This paper mainly introduces NSQ and how go language operates NSQ.

NSQ

NSQ introduction

NSQ is an open source real-time distributed memory message queue written in go language. NSQ has the following advantages:

  1. NSQ advocates distributed and decentralized topology, no single point of failure, supports fault tolerance and high availability, and provides reliable message delivery guarantee
  2. NSQ supports scale out without any centralized agents.
  3. NSQ is easy to configure and deploy, and has a built-in management interface.

Application scenarios of NSQ

Generally speaking, message queuing is suitable for the following scenarios.

Asynchronous processing

As shown in the figure below, message queue is used to asynchronize the non critical processes in the business process, so as to significantly reduce the response time of business requests.

Application decoupling

By using message queue to decouple different business logic, the coupling between systems is reduced, and the robustness of the system is improved. If other businesses want to use the order data, they can directly subscribe to the message queue to improve the flexibility of the system.  

Flow peak clipping

In scenarios like second kill (large second), a large number of requests may be generated at a certain time. Using message queue can provide a certain buffer for back-end processing requests and ensure the stability of back-end services.  

install

The official download page can be downloaded and unzipped according to its own platform.

NSQ components

nsqd

Nsqd is a daemon that receives, queues and sends messages to clients.

start-upnsqd, specify-broadcast-address=127.0.0.1To configure the broadcast address

./nsqd -broadcast-address=127.0.0.1

If it’s a matchnsqlookupdThe mode to be used also needs to be specifiednsqlookupdAddress:

./nsqd -broadcast-address=127.0.0.1 -lookupd-tcp-address=127.0.0.1:4160

If multiplensqlookupdNode cluster, which can also specify multiple nodes-lookupd-tcp-address

nsqdqRelated configuration items are as follows:

-auth-http-address value
    : to query auth server (may be given multiple times)
-broadcast-address string
    address that will be registered with lookupd (defaults to the OS hostname) (default "PROSNAKES.local")
-config string
    path to config file
-data-path string
    path to store disk-backed messages
-deflate
    enable deflate feature negotiation (client compression) (default true)
-e2e-processing-latency-percentile value
    message processing time percentiles (as float (0, 1.0]) to track (can be specified multiple times or comma separated '1.0,0.99,0.95', default none)
-e2e-processing-latency-window-time duration
    calculate end to end latency quantiles for this duration of time (ie: 60s would only show quantile calculations from the past 60 seconds) (default 10m0s)
-http-address string
    : to listen on for HTTP clients (default "0.0.0.0:4151")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-https-address string
    : to listen on for HTTPS clients (default "0.0.0.0:4152")
-log-prefix string
    log message prefix (default "[nsqd] ")
-lookupd-tcp-address value
    lookupd TCP address (may be given multiple times)
-max-body-size int
    maximum size of a single command body (default 5242880)
-max-bytes-per-file int
    number of bytes per diskqueue file before rolling (default 104857600)
-max-deflate-level int
    max deflate compression level a client can negotiate (> values == > nsqd CPU usage) (default 6)
-max-heartbeat-interval duration
    maximum client configurable duration of time between client heartbeats (default 1m0s)
-max-msg-size int
    maximum size of a single message in bytes (default 1048576)
-max-msg-timeout duration
    maximum duration before a message will timeout (default 15m0s)
-max-output-buffer-size int
    maximum client configurable size (in bytes) for a client output buffer (default 65536)
-max-output-buffer-timeout duration
    maximum client configurable duration of time between flushing to a client (default 1s)
-max-rdy-count int
    maximum RDY count for a client (default 2500)
-max-req-timeout duration
    maximum requeuing timeout for a message (default 1h0m0s)
-mem-queue-size int
    number of messages to keep in memory (per topic/channel) (default 10000)
-msg-timeout string
    duration to wait before auto-requeing a message (default "1m0s")
-node-id int
    unique part for message IDs, (int) in range [0,1024) (default is hash of hostname) (default 616)
-snappy
    enable snappy feature negotiation (client compression) (default true)
-statsd-address string
    UDP : of a statsd daemon for pushing stats
-statsd-interval string
    duration between pushing to statsd (default "1m0s")
-statsd-mem-stats
    toggle sending memory and GC stats to statsd (default true)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement) (default "nsq.%s")
-sync-every int
    number of messages per diskqueue fsync (default 2500)
-sync-timeout duration
    duration of time per diskqueue fsync (default 2s)
-tcp-address string
    : to listen on for TCP clients (default "0.0.0.0:4150")
-tls-cert string
    path to certificate file
-tls-client-auth-policy string
    client certificate auth policy ('require' or 'require-verify')
-tls-key string
    path to key file
-tls-min-version value
    minimum SSL/TLS version acceptable ('ssl3.0', 'tls1.0', 'tls1.1', or 'tls1.2') (default 769)
-tls-required
    require TLS for client connections (true, false, tcp-https)
-tls-root-ca-file string
    path to certificate authority file
-verbose
    enable verbose logging
-version
    print version string
-worker-id
    do NOT use this, use --node-id

nsqlookupd

Nsqlookupd is the daemons that maintain all nsqd States and provide service discovery. It can find specific information for consumerstopicNsqd under provides runtime automatic discovery service. It does not maintain a persistent state and does not need to coordinate with any other nsqlookupd instance to satisfy the query. So deploy as much as possible according to the redundancy requirements of your systemnsqlookupdNode. They have few resources and can coexist with other services. Our recommendation is to run at least three clusters for each data center.

nsqlookupdRelated configuration items are as follows:

-broadcast-address string
    address of this lookupd node, (default to the OS hostname) (default "PROSNAKES.local")
-config string
    path to config file
-http-address string
    : to listen on for HTTP clients (default "0.0.0.0:4161")
-inactive-producer-timeout duration
    duration of time a producer will remain in the active list since its last ping (default 5m0s)
-log-prefix string
    log message prefix (default "[nsqlookupd] ")
-tcp-address string
    : to listen on for TCP clients (default "0.0.0.0:4160")
-tombstone-lifetime duration
    duration of time a producer will remain tombstoned if registration remains (default 45s)
-verbose
    enable verbose logging
-version
    print version string

nsqadmin

A web management platform for real-time monitoring the status of cluster and performing various management tasks. start-upnsqadmin, specifynsqlookupdAddress:

./nsqadmin -lookupd-http-address=127.0.0.1:4161

We can open it with a browserhttp://127.0.0.1:4171/Access the following management interface.  

nsqadminThe related configuration items are as follows:

-allow-config-from-cidr string
    A CIDR from which to allow HTTP requests to the /config endpoint (default "127.0.0.1/8")
-config string
    path to config file
-graphite-url string
    graphite HTTP address
-http-address string
    : to listen on for HTTP clients (default "0.0.0.0:4171")
-http-client-connect-timeout duration
    timeout for HTTP connect (default 2s)
-http-client-request-timeout duration
    timeout for HTTP request (default 5s)
-http-client-tls-cert string
    path to certificate file for the HTTP client
-http-client-tls-insecure-skip-verify
    configure the HTTP client to skip verification of TLS certificates
-http-client-tls-key string
    path to key file for the HTTP client
-http-client-tls-root-ca-file string
    path to CA file for the HTTP client
-log-prefix string
    log message prefix (default "[nsqadmin] ")
-lookupd-http-address value
    lookupd HTTP address (may be given multiple times)
-notification-http-endpoint string
    HTTP endpoint (fully qualified) to which POST notifications of admin actions will be sent
-nsqd-http-address value
    nsqd HTTP address (may be given multiple times)
-proxy-graphite
    proxy HTTP requests to graphite
-statsd-counter-format string
    The counter stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.counters.%s.count")
-statsd-gauge-format string
    The gauge stats key formatting applied by the implementation of statsd. If no formatting is desired, set this to an empty string. (default "stats.gauges.%s")
-statsd-interval duration
    time interval nsqd is configured to push to statsd (must match nsqd) (default 1m0s)
-statsd-prefix string
    prefix used for keys sent to statsd (%s for host replacement, must match nsqd) (default "nsq.%s")
-version
    print version string

NSQ architecture

NSQ working mode

Topic and channel

Each nsqd instance is designed to process multiple data streams at once. These data streams are called“topics”OnetopicHas one or more“channels”. eachchannelYou’ll get ittopicCopies of all messages, in fact, the downstream service is through the correspondingchannelTo consumetopicNews.

topicandchannelIt’s not pre configured.topicCreated on first use by publishing it to the specifiedtopic, or subscribe to the specifiedtopicOnchannelchannelIs specified by subscriptionchannelCreated when first used.

topicandchannelBuffer data independently to prevent slow consumers from causing other problemschennel(the same applies totopicLevel).

channelMultiple clients can and usually will be connected. Assuming that all connected clients are ready to receive messages, each message is delivered to a random client. For example:

All in all, the news comes fromtopic -> channel(each channel receives copies of all messages of the topic) multicast, but fromchannel -> consumersEvenly distributed (each consumer receives part of the channel’s messages).

NSQ receive and send message flow

NSQ features

  • Messages are not persistent by default and can be configured in persistent mode. NSQ adopts the mode of memory + hard disk. When the memory reaches a certain level, it will persist the data to the hard disk.
    • If--mem-queue-sizeSet to 0, all messages will be stored to disk.
    • When the server restarts, it will also persist the messages in memory at that time.
  • Each message is delivered at least once.
  • Messages are not guaranteed to be orderly.

Go operation NSQ

Go language version of the official client: go NSQ, more client support please see client libraries.

install

go get -u github.com/nsqio/go-nsq

producer

A simple producer example code is as follows:

// nsq_producer/main.go
package main

import (
	"bufio"
	"fmt"
	"os"
	"strings"

	"github.com/nsqio/go-nsq"
)

// NSQ Producer Demo

var producer *nsq.Producer

//Initialize producer
func initProducer(str string) (err error) {
	config := nsq.NewConfig()
	producer, err = nsq.NewProducer(str, config)
	if err != nil {
		fmt.Printf("create producer failed, err:%v\n", err)
		return err
	}
	return nil
}

func main() {
	nsqAddress := "127.0.0.1:4150"
	err := initProducer(nsqAddress)
	if err != nil {
		fmt.Printf("init producer failed, err:%v\n", err)
		return
	}

	Reader: = bufio. Newreader (OS. Stdin) // read from standard input
	for {
		data, err := reader.ReadString('\n')
		if err != nil {
			fmt.Printf("read string from stdin failed, err:%v\n", err)
			continue
		}
		data = strings.TrimSpace(data)
		If strings. Toupper (data) = = q "{// enter Q to exit
			break
		}
		//To 'topic'_ Demo 'publish data
		err = producer.Publish("topic_demo", []byte(data))
		if err != nil {
			fmt.Printf("publish msg to nsq failed, err:%v\n", err)
			continue
		}
	}
}

Compile and execute the above code, and then input two pieces of data in the terminal123and456

$ ./nsq_producer 
123
2018/10/22 18:41:20 INF    1 (127.0.0.1:4150) connecting to nsqd
456

Open with browserhttp://127.0.0.1:4171/You can view a page similar to the following: you can see the currenttopicInformation:  

Click on thetopic_demoYou can enter a page showing more details, where we can view and managetopicAt the same time, we can see theLWZMBP:4151 (127.0.01:4151)thisnsqdThere are two messages on the. And because there is no consumer access, so there is no creationchannel。 

stay/nodesThis page is very convenient for us to view the current accesslookupdOfnsqdNode.  

this/counterThe page shows the number of messages processed, because we have no access to consumers, so the number of messages processed is 0.  

stay/lookupInterface support creationtopicandchannel。 

consumer

A simple consumer example code is as follows:

// nsq_consumer/main.go
package main

import (
	"fmt"
	"os"
	"os/signal"
	"syscall"
	"time"

	"github.com/nsqio/go-nsq"
)

// NSQ Consumer Demo

//Myhandler is a consumer type
type MyHandler struct {
	Title string
}

//Handlemessage is a method that needs to be implemented to process messages
func (m *MyHandler) HandleMessage(msg *nsq.Message) (err error) {
	fmt.Printf("%s recv from %v, msg:%v\n", m.Title, msg.NSQDAddress, string(msg.Body))
	return
}

//Initial consumer
func initConsumer(topic string, channel string, address string) (err error) {
	config := nsq.NewConfig()
	config.LookupdPollInterval = 15 * time.Second
	c, err := nsq.NewConsumer(topic, channel, config)
	if err != nil {
		fmt.Printf("create consumer failed, err:%v\n", err)
		return
	}
	consumer := &MyHandler{
		Title: "Shahe 1",
	}
	c.AddHandler(consumer)

	// if err := c.ConnectToNSQD(address);  err !=  Nil {// directly connected to nsqd
	if err := c.ConnectToNSQLookupd(address);  err !=  Nil {// query through lookupd
		return err
	}
	return nil

}

func main() {
	err := initConsumer("topic_demo", "first", "127.0.0.1:4161")
	if err != nil {
		fmt.Printf("init consumer failed, err:%v\n", err)
		return
	}
	C: = make (Chan OS. Signal) // define a signal channel
	Signal. Notify (C, syscall. SIGINT) // forwards the keyboard interrupt signal to C

After the above code is saved and compiled, we can get the two messages we published before:

$ ./nsq_consumer 
2018/10/22 18:49:06 INF    1 [topic_demo/first] querying nsqlookupd http://127.0.0.1:4161/lookup?topic=topic_demo
2018/10/22 18:49:06 INF    1 [topic_demo/first] (127.0.0.1:4150) connecting to nsqd
Shahe No.1 recv from 127.0.0.1:4150, msg:123
Shahe No.1 recv from 127.0.0.1:4150, msg:456

At the same time, in nsqadmin/counterThe number of processed data is 2.  

aboutgo-nsqFor more information on go NSQ, please read the official document of go NSQ.

 

Reprinted from Li Wenzhou’s blog

Recommended Today

The road of high salary, a more complete summary of MySQL high performance optimization in history

preface MySQL for many linux practitioners, is a very difficult problem, most of the cases are because of the database problems and processing ideas are not clear. Before the optimization of MySQL, we must understand the query process of MySQL. In fact, a lot of query optimization work is to follow some principles so that […]