Lua web rapid development guide (10) – asynchronous task, subscription / publishing, message queuing with MQ

Time:2019-12-2

In this chapter, we will learn how to useMQLibrary.

Introduction to MQ Library

MQThe library implements the connection protocols of various message broker middleware. At present, it supports:redismqttstompAgreement.

MQBased on the above protocol, the library implements:Producer - > consumerAndSubscription - > publishThe model can accomplish tasks independently without relying on other services

API introduction

CF framework provides a variety ofMQWhen we need to use it, we need to choose according to the actual protocol:

-- local MQ = require "MQ.mqtt"
-- local MQ = require "MQ.redis"
-- local MQ = require "MQ.stomp"

MQ:new(opt)

This method will create an MQ object instance of

optIt is atableParameter of type, you can pass the following values:

  • Host – string type, the domain name or IP address of the message queue
  • Port – int type, the port on which message queuing listens
  • Auth / DB – string type, only used for login authentication or DB selection under redis protocol (none can be left blank)
  • Username / password – string type, only used for login authentication under the stomp / mqtt protocol (none can be left blank)
  • Vhost – string type, only filled in when using certain message queuing servers (for example: rabbit)
  • Keepalive – int type, which is used only when mqtt is used to start the client to actively send out the heartbeat packet

Take redis broker as an example:

local MQ = require "MQ.redis"
local mq = MQ:new {
  host = "localhost",
  port = 6379,
  -- db = 0,
  -- auth = "123456789",
}

MQ:on(pattern, function)

This method is used to subscribe to a specifiedpatternWhenbrokerAfter passing the message to CF,functionWill be called

MQLibrary will befunctionInjecting onetableParameters of typemsg, this parameter will benil.

msgDepending on the protocol usedmsgThe content will be differentloggingThe printing of the library shall prevail

Standard use example:

local Log = require("logging"):new()
mq:on("/notice", function(msg)
  if not msg then
    Return log: error ("['/ notice'] subscribe error: connection disconnected.")
  end
  Log:DEBUG(msg)
end)

Developers can subscribe to multipleparttern.

MQ:emit(pattern, msg)

This method is used to specifypatternSend message. MSG is a message of string type

Use example:

mq:emit('/notice', '{"code":200,"data":[1,2,3,4,5,6,7,8,9,10]}')

singleMQIt can be reused all the time, and a write queue will be created internally to complete the sequential sending of messages

MQ:start()

This method is called when running as a standalone server

Use example:

mq:start()

MQ:clsoe()

This method can shut down MQ that is no longer used; in any case, MQ needs to call this method to release resources after using it

Use example:

mq:close()

Begin to practice

In order to make the demonstration more intuitive, we only use redis as broker technical secondary message

1. Simulate producers and consumers

We simulate 100 producers to redis/queuePost message and define a consumer subscription/queueContinuous consumption

The code is as follows:

local cf = require "cf"
local json = require "json"
local Log = require("logging"):new()
local MQ = require "MQ.redis"

cf.fork(function ()
  local consumer = MQ:new {
    host = "localhost",
    port = 6379
  }

  local count = 0
  consumer:on("/queue", function (msg)
    if not msg then
      Log: error ("[/ queue] connection failed", "has consumed".. count.. "messages")
      return
    end
    count = count + 1
    Log: debug ("start consuming:", MSG, "has consumed".. count.. "messages")
  end)

  Consumer: start() -- no need to use this method inside websocket
end)

for i = 1, 100 do
  cf.fork(function()

    local producer = MQ:new {
      host = "localhost",
      port = 6379
    }

    producer:emit("/queue", json.encode({
      code = 200,
      data = {
        id = math.random(1, 1 << 32)
      },
    }))

    producer:close()
  end)
end

The output is as follows:

[[email protected]:~/Documents/core_framework] $ ./cfadmin
[2019-06-25 16:05:36240] [@ script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 3912595079}", ["source"] = "/ queue", ["type"] = "pmessage"}, 1 message has been consumed
[2019-06-25 16:05:36240] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 2938696189}", ["source"] = "/ queue", ["type"] = "pmessage"}, and 2 messages have been consumed
[2019-06-25 16:05:36240] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 3499397173}", ["source"] = "/ queue", ["type"] = "pmessage"}, three messages have been consumed
[2019-06-25 16:05:36240] [@ script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 1711272453}", ["source"] = "/ queue", ["type"] = "pmessage"}. Four messages have been consumed
[2019-06-25 16:05:36240] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 3968420025}", ["source"] = "/ queue", ["type"] = "pmessage"}, and 5 messages have been consumed
[2019-06-25 16:05:36240] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 1887895479}", ["source"] = "/ queue", ["type"] = "pmessage"}, and 6 messages have been consumed
[2019-06-25 16:05:36240] [@ script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 3687986737}", ["source"] = "/ queue", ["type"] = "pmessage"}. Seven messages have been consumed
[2019-06-25 16:05:36240] [@ script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 2823099353}", ["source"] = "/ queue", ["type"] = "pmessage"}. Eight messages have been consumed
[2019-06-25 16:05:36240] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 2528190121}", ["source"] = "/ queue", ["type"] = "pmessage"}, and 9 messages have been consumed
[2019-06-25 16:05:36240] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 410799865}", ["source"] = "/ queue", ["type"] = "pmessage"}, 10 messages have been consumed
.
..
...
....
.....
[2019-06-25 16:05:36247] [script / main. Lua: 19] [debug]: start consumption: {["pattern"] = "/ queue", ["payload"] = "{" code ": 200," data ": {" Id ": 3608578767}", ["source"] = "/ queue", ["type"] = "pmessage"}, and 100 messages have been consumed

In order to facilitate reading, we take out the first 10 items and the last 100 items and print out the data structure of MSG for reading

The consumer’s processing method is synchronous and non blocking (the next message will not be processed if the current business is not completed). If you don’t want to block the current message queue event cycle, you can consider yourselfforkA process to deal with

2. Push message to a user

Users can subscribe to their own channels after accessing the server after authentication. When there are user-specific messages, any service can use this method to push business messages

We

The code implementation is as follows:

local cf = require "cf"
local json = require "json"
local Log = require("logging"):new()
local MQ = require "MQ.redis"

for uid = 1, 10 do
  cf.fork(function ()
    local client = MQ:new {
      host = "localhost",
      port = 6379
    }

    client:on("/user/"..uid.."/*", function (msg)
      if not msg then
        Log: error ("[/ user / 9257] connection failed")
        return
      end
      Log: debug ("uid: ["... Uid.. "] received push message", MSG)
    end)

    Client: start() -- no need to use this method inside websocket
  end)
end

local server = MQ:new {
  host = "localhost",
  port = 6379
}

cf.at(1, function (...)
  server:emit("/user/"..math.random(1, 10).."/ad", json.encode({
    code = 200,
    data = {}
  }))
end)

server:start()

The terminal output after operation is as follows:

^C[[email protected]:~/Documents/core_framework] $ ./cfadmin
[2019-06-25 16:20:23506] [script / main. Lua: 18] [debug]: uid: [9] received push message, {["source"] = "/ user / 9 / ad", ["pattern"] = "/ user / 9 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:24504] [script / main. Lua: 18] [debug]: uid: [4] push message received, {["source"] = "/ user / 4 / ad", ["pattern"] = "/ user / 4 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:25506] [script / main. Lua: 18] [debug]: uid: [8] push message received, {["source"] = "/ user / 8 / ad", ["pattern"] = "/ user / 8 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:26506] [script / main. Lua: 18] [debug]: uid: [8] received push message, {["source"] = "/ user / 8 / ad", ["pattern"] = "/ user / 8 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:27505] [script / main. Lua: 18] [debug]: uid: [10] received push message, {["source"] = "/ user / 10 / ad", ["pattern"] = "/ user / 10 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:28506] [script / main. Lua: 18] [debug]: uid: [2] received push message, {["source"] = "/ user / 2 / ad", ["pattern"] = "/ user / 2 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:29506] [script / main. Lua: 18] [debug]: uid: [4] push message received, {["source"] = "/ user / 4 / ad", ["pattern"] = "/ user / 4 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:30506] [script / main. Lua: 18] [debug]: uid: [8] received push message, {["source"] = "/ user / 8 / ad", ["pattern"] = "/ user / 8 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:31505] [script / main. Lua: 18] [debug]: uid: [3] received push message, {["source"] = "/ user / 3 / ad", ["pattern"] = "/ user / 3 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:32506] [script / main. Lua: 18] [debug]: uid: [6] received push message, {["source"] = "/ user / 6 / ad", ["pattern"] = "/ user / 6 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:33506] [script / main. Lua: 18] [debug]: uid: [5] received push message, {["source"] = "/ user / 5 / ad", ["pattern"] = "/ user / 5 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:34503] [script / main. Lua: 18] [debug]: uid: [7] push message received, {["source"] = "/ user / 7 / ad", ["pattern"] = "/ user / 7 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:35506] [script / main. Lua: 18] [debug]: uid: [4] push message received, {["source"] = "/ user / 4 / ad", ["pattern"] = "/ user / 4 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:36506] [script / main. Lua: 18] [debug]: uid: [6] push message received, {["source"] = "/ user / 6 / ad", ["pattern"] = "/ user / 6 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
[2019-06-25 16:20:37505] [script / main. Lua: 18] [debug]: uid: [10] received push message, {["source"] = "/ user / 10 / ad", ["pattern"] = "/ user / 10 / *", ["type"] = "pmessage", ["payload"] = "{" data ": {}," code ": 200}"}
^C[[email protected]:~/Documents/core_framework] $

Here we can see that from the news release to/user/9527/*LowertopicWe can pass it oncewildcardSubscription can receive all subordinate routing messages

3. Message broadcasting

In various fields, message push has become the most common business. Now we try to use MQ to implement message push business

First, we willscript/main.luaThe following code is written to the file of:

-- main.lua
local cf = require "cf"
local json = require "json"
local Log = require("logging"):new()
local MQ = require "MQ.redis"

for i = 1, 3 do
  cf.fork(function ()
    local uid = math.random(1, 1 << 32)
    local client_mq = MQ:new {
      Host = "localhost", -- host name
      Port = 6379, -- port number
      --DB = nil, -- default database
      --Auth = nil, -- password
    }
    client_mq:on("/system/notice", function (msg)
      if not msg then
        Log: error ("['/ system / notice'] subscribe error: connection disconnected.")
        return
      end
      Log: debug ("uid: ["... Uid.. "] received message:", MSG)
    end)

    client_mq:start()
  end)
end

local server_mq = MQ:new {
  Host = "localhost", -- host name
  Port = 6379, -- port number
  --DB = nil, -- default database
  --Auth = nil, -- password
}

cf.at(3, function (args)
  server_mq:emit("/system/notice", json.encode({
    code = 200,
    MSG = "system is about to shut down"
  }))
end)

server_mq:start()

Here we start three processes to simulate the user subscribing to messages, and each process uses a different uid to print. Then we start a timer to simulate the message push business every three seconds

Open terminal operation./cfadminAfter that, the output is as follows:

[[email protected]:~/Documents/core_framework] $ ./cfadmin
[2019-06-25 15:43:24842] [script / main. Lua: 19] [debug]: uid: [3363385555] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[2019-06-25 15:43:24842] [script / main. Lua: 19] [debug]: uid: [1693861773] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" the system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[2019-06-25 15:43:24842] [script / main. Lua: 19] [debug]: uid: [3608578767] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}

[2019-06-25 15:43:27841] [script / main. Lua: 19] [debug]: uid: [3363385555] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[2019-06-25 15:43:27841] [script / main. Lua: 19] [debug]: uid: [1693861773] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[2019-06-25 15:43:27841] [script / main. Lua: 19] [debug]: uid: [3608578767] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}

[2019-06-25 15:43:30841] [script / main. Lua: 19] [debug]: uid: [3363385555] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[2019-06-25 15:43:30841] [script / main. Lua: 19] [debug]: uid: [1693861773] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[2019-06-25 15:43:30841] [script / main. Lua: 19] [debug]: uid: [3608578767] received message: {["pattern"] = "/ system / notice", ["payload"] = "{" MSG ":" system is about to shut down "," code ": 200}", ["type"] = "pmessage", ["source"] = "/ system / notice"}
[[email protected]:~/Documents/core_framework] $

From the output of the terminal, we can see that we do receive a message push every 3 seconds

4. Implement business push for client based on websocket protocol

First of all, we need to build a system based onhttpdLibraryWebsocketRouting. Let’s turn onscript/main.luaFile and write the following code in

local httpd require "httpd"

local app = httpd:new("Web")

app:ws('/ws', require "ws")

app:listen("0.0.0.0", 8080)

app:run()

WebsocketYou must use theMQLibrary subscription/chat. trigger every time the client sends a messageon_messageAt that time, the information will be published directly to/chatPush chat is realized after internal transfer

And then we use what we learned in the previous chapterWebsocket GuideBecause there is no uid generation mechanism in the sample code, we randomly generate 32-bit integers as unique ID identifiers to facilitate debugging

script/ws.luaThe specific codes are as follows:

local MQ = require "MQ.redis"
local class = require "class"

local websocket = class("websocket")

function websocket:ctor (opt)
  self.ws = opt.ws
  self.id = math.random(1, 1 << 32)
end

function websocket:on_open ()
  self.mq = MQ:new { host = 'localhost', port = 6379 }
  self.mq:on("/chat", function (msg)
    if not msg then
      return
    end
    self.ws:send(msg.payload)
  end)
end

function websocket:on_message (data, typ)
  if self.mq then
    self.mq:emit("/chat", data)
  end
  Print ("client ["... Self. Id.. "] sent message: [". Data.. "])
end

function websocket:on_error (error)

end

function websocket:on_close ()
  if self.mq then
    self.mq:close()
    self.mq = nil
  end
end

return websocket

Be careful: we need to remember to turn off subscription recycling when the client is disconnected./cfadminTo see if it works properly

Let’s download the client tools and install them into ourChromeBrowser. Extraction code:cgwr

Now, let’s run the client tool and type in the address barlocalhost:8080/wsConnect to the websocket server we just started, and then start sending messages to the server

If we see similar output from the terminal and the client, our example is finished

[[email protected]:~/Documents/core_framework] $ ./cfadmin
[2019 / 06 / 25 20:11:59] [info] httpd is listening: 0.0.0.0:8080
[2019 / 06 / 25 20:11:59] [info] httpd is running web server service
[2019/06/25 20:12:01] - ::1 - ::1 - /ws - GET - 101 - req_time: 0.000095/Sec
[2019/06/25 20:12:17] - ::1 - ::1 - /ws - GET - 101 - req_time: 0.000080/Sec
The client [1693861773] sent a message: [Hello! I am 2]
The client [1693861773] sent a message: [Hello! I am 2]
The client [1693861773] sent a message: [Hello! I am 2]
The client [1693861773] sent a message: [Hello! I am 2]
The client [1693861773] sent a message: [Hello! I am 2]
The client [1693861773] sent a message: [Hello! I am 2]
The client [1693861773] sent a message: [Hello! I am 2]
[2019/06/25 20:12:23] - ::1 - ::1 - /ws - GET - 101 - req_time: 0.000052/Sec
The client [3363385555] sent a message: [Hello! I'm 1]
The client [3363385555] sent a message: [Hello! I'm 1]
The client [3363385555] sent a message: [Hello! I'm 1]
The client [3363385555] sent a message: [Hello! I'm 1]
The client [3363385555] sent a message: [Hello! I'm 1]
The client [3363385555] sent a message: [Hello! I'm 1]
The client [3363385555] sent a message: [Hello! I'm 1]
The client [1693861773] sent a message: [Hello! I am 2]

Last

The above code is only usedredisThe protocol is simulated. For other protocols, please refer to wiki

Completion of study

thusLua Web Development GuideIt has been written. In the field of software development, it is not only the master’s guide, but also the embodiment of personal cultivation

There are many built-in libraries in the CF framework, and there are many tutorials to be written at the same time to maintain the framework. It is impossible for the author to introduce all of them. The CF framework already has a dedicated QQ discussion community:727531854, click Add group

At present, the author is alone in it. If you are interested in it, you are welcome to come to the group to exchange technology