Implementation of reliable node.js HTTP proxy based on UNIX socket (supporting websocket protocol)

Time:2021-6-6

The most common way to implement proxy service is that the proxy server proxy the corresponding protocol body to request the source station and forward the response from the source station to the client. In this scenario, the proxy service and the source service use the same technology stack (node. JS). The source service is the business service forked out by the proxy service (as shown in the figure below). The proxy service is not only responsible for request reverse proxy and forwarding rule setting, but also responsible for business service expansion, log output and related resource monitoring alarm. Hereinafter referred to as the source serviceBusiness services
Implementation of reliable node.js HTTP proxy based on UNIX socket (supporting websocket protocol)

At first, the author adopts the architecture shown in the figure above. The business service is a real HTTP service or websocket service, which listens to a port of the server and processes the forwarding request of the proxy service. But there are some problems that will trouble us:

  • Business services need to listen on ports, which are capped and may conflict (though conflict can be avoided)
  • When the proxy service forwards the request, it goes through another TCP / IP stack parsing in the kernel, and there is performance loss (TCP slow start, ACK mechanism and other reliability guarantees lead to transmission performance degradation)
  • The forwarding strategy needs to be coupled with the port, so there is a risk in service migration

Therefore, the author tries to find a better solution.

HTTP server based on UNIX socket protocol

To be honest, when I learned Linux network programming, I never tried HTTP server based on domain socket. However, in terms of protocol, HTTP protocol does not strictly require that the transport layer protocol must be TCP. Therefore, if the underlying layer uses UNIX socket transmission based on byte stream, it should also be able to achieve the requirements.

At the same time, compared with TCP protocol, UNIX socket has some advantages as IPC

  • UNIX socket only copies data, does not perform protocol processing, does not need to add or delete network header, does not need to calculate the check sum, does not need to generate sequence number, and does not need to send confirmation message
  • It only depends on named pipes and does not occupy ports

UNIX socket is not a protocol, it is a way of inter process communication (IPC) to solve the communication between two processes of the machine

The HTTP module and net module of node.js provide relevant interfaces“listen(path, cb)”The difference is that the HTTP module encapsulates the HTTP protocol parsing and related specifications on UNIX socket, so it can be seamlessly compatible with the HTTP service based on TCP.

The following is an example of HTTP server and client based on UNIX socket:

const  http  =  require('http');
const  path  =  require('path');
const  fs  =  require('fs');
const  p  =  path.join(__dirname,'tt.sock');

fs.unlinkSync(p);
let  s  =  http.createServer((req, res)=> {
req.setEncoding('utf8')
req.on('data',(d)=>{
    console.log('server get:', d)
});
res.end('helloworld!!!');
});

s.listen(p);

setTimeout(()=>{
    let  c  =  http.request( {
        method:  'post',
        socketPath:  p,
        path:  '/test'
    }, (res) => {
        res.setEncoding('utf8');
        res.on('data', (chunk) => {
            Console.log (` response body: ${chunk} ');
        });

        res.on('end', () => {
        });
    });
    c.write(JSON.stringify({abc:  '12312312312'}));
    c.end();
},2000)

Agent service and creation of business service process

Proxy service is not only a proxy request, but also responsible for the creation of business service process. In the more advanced demand, the proxy service is also responsible for the expansion and scaling of business service process. When the business flow comes up, in order to improve the throughput of business services, the proxy service needs to create more business service processes, and recover the appropriate process resources after the peak flow dissipates. From this perspective, we can find that this kind of demand is related to cluster and child_ Process module is closely related to each other, so the specific implementation of business service cluster will be introduced below.

In order to implement websocket service with sticky session function, the agent in this paper adopts child_ The process module creates a business process. The sticky session here mainly refers to that the handshake message of socket.io needs to always negotiate with the fixed process, otherwise the socket.io connection cannot be established (the socket.io connection here specifically refers to the connection above the successful operation of socket.io). For details, see my article socket.io with PM2 (cluster) cluster solution. However, in the fork business process, the pre_ How to rewrite a subprocess in hook scripthttp.Server.listen()In order to realize the reliable transmission of the underlying layer based on UNIX socket, this method refers to the cluster module’s relevant processing of the sub process. For the cluster module’s override of the sub process’s listen, please refer to the section of “multi sub process and port reuse” in my another article nodejs cluster module.

//Subprocess pre_ Hook script to realize HTTP server based on UNIX socket reliable transmission
function  setupEnvironment() {
    process.title  =  'ProxyNodeApp: '  +  process['env']['APPNAME'];
    http.Server.prototype.originalListen  =  http.Server.prototype.listen;
    http.Server.prototype.listen  =  installServer;
    loadApplication();
}
function  installServer() {
    var  server  =  this;
    var  listenTries  =  0;
    doListen(server, listenTries, extractCallback(arguments));
    return  server;
}

function  doListen(server, listenTries, callback) {
    function  errorHandler(error) {
        // error handle
    }
    //Generate pipe
    var  socketPath  =  domainPath  =  generateServerSocketPath();
    server.once('error', errorHandler);
    server.originalListen(socketPath, function() {
        server.removeListener('error', errorHandler);
        doneListening(server, callback);
        process.nextTick(finalizeStartup);
    });

    process.send({
        type:  'path',
        path:  socketPath
    });
}

In this way, the underlying infrastructure of business services is completed. In the coding stage of business services, there is no need to pay attention to the specific implementation of the transport layer, and http.server.listen (${any) is still used_ Port}. At this time, business services can listen to any port, because the port is not used in the transport layer, so as to avoid the waste of system ports.

Traffic forwarding

Traffic forwarding includes HTTP request and websocket handshake message. Although websocket handshake message is still implemented based on HTTP protocol, it needs different processing, so it is discussed separately here.

HTTP traffic forwarding

In this section, you can refer to the example of “HTTP server and client based on UNIX socket”, create a new HTTP client request service based on UNIX socket in proxy service, and send the response pipe to the client.

class  Client  extends  EventEmitter{
    constructor(options) {
        super();
        options  =  options  || {};
        this.originHttpSocket  =  options.originHttpSocket;
        this.res  =  options.res;
        this.rej  =  options.rej;
        if (options.socket) {
            this.socket  =  options.socket;
        } else {
            let  self  =  this;
            this.socket  =  http.request({
                method:  self.originHttpSocket.method,
                socketPath:  options.sockPath,
                path:  self.originHttpSocket.url,
                headers:  self.originHttpSocket.headers
            }, (res) => {
                self.originHttpSocket.set(res.headers);
                self.originHttpSocket.res.writeHead(res.statusCode);
                //Proxy response
                res.pipe(self.originHttpSocket.res)
                self.res();
            });
        }
    }
    send() {
        //Proxy request
        this.originHttpSocket.req.pipe(this.socket);
    }
}
// proxy server
const  app  =  new  koa();
app.use(async  ctx  => {
    await  new  Promise((res,rej) => {
        //Proxy request
        let  client  =  new  Client({
            originHttpSocket:  ctx,
            sockPath:  domainPath,
            res,
            rej
        });
        client.send();
    });
});

let  server  =  app.listen(8000);

Websocket message processing

If the websocket message is not processed, so far socket.io can only use the “polling” mode, that is, the false long connection can be realized through XHR polling, and the websocket connection cannot be established. Therefore, for better performance experience, websocket packets need to be processed. Here, we mainly refer to the implementation of “HTTP proxy” and do some operations for the message

  1. Header protocol upgrade field check
  2. Proxy request for protocol upgrade based on UNIX socket

The core of message processing lies in the second point: create a “long connection” between proxy service and business service process (the connection is based on UNIX socket pipeline instead of TCP long connection), and use the HTTP upgrade request of this connection overlay to upgrade the protocol.

The implementation here is more complex, so only the processing of proxy service is presented. For the detailed process of websocket message processing, please refer to proxy based UNIX socket.

//Initialize WS module
wsHandler  =  new  WsHandler({
    target: {
        socketPath:  domainPath
    }
}, (err, req, socket) => {
    Console. Error (` agent wshandler error ', ERR));
});

//Handshake upgrade of proxy WS protocol
server.on('upgrade',(req, socket, head) =>{
    wsHandler.ws(req, socket, head);
});

Review and summary

As we all know, to implement HTTP service cluster in node.js, we should use cluster module instead of “child”_ This is due to the adoption of the child module_ The HTTP service cluster implemented by process has the problem of uneven scheduling (the “optimization move” made by the kernel in order to save the context switching cost can refer to the section of “request distribution strategy” of nodejs cluster module for details). Why is child still used in this paper_ What about the process module?

The answer is: different scenes. As a proxy service, it can use cluster module to implement the cluster of proxy service; For business services, in the scenario of session, the proxy service needs to implement the corresponding forwarding strategy, and in other cases, the roundrobin strategy can be used, so the child_ The process module is more suitable.

This paper does not implement the load balancing strategy of proxy service, and its implementation is still described in the in-depth exploration of nodejs cluster module, so please refer to this article.

Finally, on the premise of keeping the process model stable, the underlying protocol is changed to achieve higher performance proxy service.
Implementation of reliable node.js HTTP proxy based on UNIX socket (supporting websocket protocol)]~~~~

Recommended Today

Modern Enterprise Architecture Framework – Application Architecture

Modern Enterprise Architecture Framework:https://mp.weixin.qq.com/s/SlrEu0_t0slijrNZ6DP4Ng Business Architecture:https://mp.weixin.qq.com/s/zQCjiHuxFvAg5QiOAuLAcQ 4. Application Architecture The core concern of application architecture is which applications carry the business requirements, how they interact with users, how and how they interact, and what data they access or change. The design of the application architecture is mainly based on the application (Application) design as the […]