Lua openresty containerization

Original address:Lua openresty containerization

background

The company has several “ancient” projects, which have been relatively stable, but the projects always request the peak QPS every minute in some time every day800KAround, leading to some bottlenecks in the performance of the machine. Every peak period, there will always be an alarm, which is really a headache. To make matters worse, this is only one of the projects in ancient times, and they are all deployed on physical machines. All machines add up to nearly 100.

In consideration of stability (peak shaving) and cost, we finally decided to add all Lua openresty projects to the k8s cluster.

Select the appropriate openresty base image

By viewing the openresty version information used online:

/usr/local/openresty/nginx/sbin/nginx -V
nginx version: openresty/1.13.6.2
built by gcc 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)
built with OpenSSL 1.1.0h  27 Mar 2018 (running with OpenSSL 1.1.0k  28 May 2019)
TLS SNI support enabled
configure arguments: --prefix=/usr/local/openresty/nginx ...

lua -v
Lua 5.1.4  Copyright (C) 1994-2008 Lua.org, PUC-Rio

Learned that what is being used isopenresty/1.13.6.2andLua 5.1.4 :

docker pull openresty/openresty:1.13.6.2-2-centos

Q: Can you choose to use smaller Alpine series?

A: Many project libraries depend on soglibcCompiled, alpine ismusl-lib, incompatible.

Q: Why not recompile?

A: On the one hand, there are risks. On the other hand, some so libraries may not be found.

Find dynamic library dependencies for a project

Nginx profile

$ tree -L 3 nginx/conf
nginx/conf
├── vhosts/
│    ├── inner.prometheus.nginx.conf
│    └── project.nginx.conf
└── nginx.conf

Self compiled C dynamic library files, such asbinary_protocol.so

Write a dockerfile, then package the project into a container and execute:

/usr/local/openresty/nginx/sbin/nginx nginx -t

Sure enough, an error is reported:

/usr/local/openresty/nginx/lua/init.lua:1: module 'binary_protocol' not found:
no field package.preload['binary_protocol']
no file '/usr/local/openresty/nginx/lua/binary_protocol.lua'
no file '/usr/local/openresty/nginx/lua_lib/binary_protocol.lua'
no file '/usr/local/openresty/nginx/luarocks/share/lua/5.1/binary_protocol.lua'
no file '/usr/local/openresty/site/lualib/binary_protocol.ljbc'
…… ……
no file '/usr/local/openresty/nginx/luarocks/lib64/lua/5.1/binary_protocol.so'
no file '/usr/local/openresty/site/lualib/binary_protocol.so'
no file '/usr/local/openresty/lualib/binary_protocol.so'
no file '/usr/local/openresty/site/lualib/binary_protocol.so'
no file '/usr/local/openresty/lualib/binary_protocol.so'
no file './binary_protocol.so'
no file '/usr/local/lib/lua/5.1/binary_protocol.so'
no file '/usr/local/openresty/luajit/lib/lua/5.1/binary_protocol.so'
no file '/usr/local/lib/lua/5.1/loadall.so'
no file '/usr/local/openresty/luajit/lib/lua/5.1/binary_protocol.so'

Q: After careful observation, it is found that so dynamic libraries are internally compiled and provided to Lua calls. How to find them?

A: Yesldd、plddOr uselsofView dynamic library files.

Through LDD and PLDD commands, you can view the dependencies related to so

ldd binary_protocol.so
linux-vdso.so.1 =>  (0x00007fff40bd4000)
libtolua++. So = > not found ## will tell us that LDD lacks this dependency
libcrypto.so.6 => not found
liblog4cplus.so.2 => not found        
libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f458d9ef000)
libm.so.6 => /lib64/libm.so.6 (0x00007f458d6ed000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f458d4d7000)
libc.so.6 => /lib64/libc.so.6 (0x00007f458d10a000)
/lib64/ld-linux-x86-64.so.2 (0x00007f458df1e000)

Through these methods, you can trace a little and find all the dependent libraries.

Lualocks external package file

From onlinenginx.conffindlua_package_pathandlua_package_cpathIncluded inluarocksPath, and then from this path, findmanifestFile that describes which lualocks libraries are installed.

Lualocks external dependency installation

RUN luarocks --tree=${WORK_DIR}/luarocks install lua-cjson \
    && luarocks --tree=${WORK_DIR}/luarocks install penlight \
    && luarocks --tree=${WORK_DIR}/luarocks install version \
    && luarocks --tree=${WORK_DIR}/luarocks install lua-resty-http \
    && luarocks --tree=${WORK_DIR}/luarocks install luaunit \
    && luarocks --tree=${WORK_DIR}/luarocks install ldoc \
    && luarocks --tree=${WORK_DIR}/luarocks install lua-discount \
    && luarocks --tree=${WORK_DIR}/luarocks install serpent \
    && luarocks --tree=${WORK_DIR}/luarocks install luacov \
    && luarocks --tree=${WORK_DIR}/luarocks install cluacov \
    && luarocks --tree=${WORK_DIR}/luarocks install mmdblua \
    && luarocks --tree=${WORK_DIR}/luarocks install lua-resty-jit-uuid \
    && luarocks --tree=${WORK_DIR}/luarocks install luasocket

RUN luarocks --tree=/usr/local/openresty/nginx/luarocks install nginx-lua-prometheus

Problems encountered and their solutions

Question 1: the container is always killed by oom

After analysis, it does occupy a very large memory:

There are many workers located through PS command

resolvent:

Limit the number of workers:worker_processes 4;

Q: Why are there so many workers?

A: On k8s, the worker process started by nginx does not follow the limit we set for the pod, but is related to the node where the pod is located.

Question 2: nginx worker process exited on signal 9

This is because the memory limit set by deployment is too small

Solution: turn it uprequestsResource limit

resources:
  limits:
      cpu: "2000m"
      memory: "1Gi"
  requests:
      cpu: "1000m"
      memory: "512Mi"

PS: it takes about 200mi to start four workers.

Q3: attempt to index upvalue ‘result_ dict’ (a nil value)

The reason is online nginx Conf has relevant definitions
But not at the code level. Add:

lua_shared_dict monitor_status 150m;

A trick to reduce the size of the image

Borrow a chicken to lay an egg

How to access Prometheus monitoring

Access Prometheus in openresty,https://github.com/knyar/ngin…

Installation dependency

luarocks --tree=/usr/local/openresty/nginx/luarocks install nginx-lua-prometheus

New configuration

bynginx/conf/vhosts/project.nginx.confAdd:

lua_shared_dict prometheus_metrics 10M;
log_by_lua_block {
    metric_requests:inc(1, {ngx.var.server_name, ngx.var.status})
    metric_latency:observe(tonumber(ngx.var.request_time), {ngx.var.server_name})
}

New profile

newly addednginx/conf/vhosts/inner.prometheus.nginx.conf

server {
    listen 8099;
    location /metrics {
        content_by_lua_block {
            metric_connections:set(ngx.var.connections_reading, {"reading"})
            metric_connections:set(ngx.var.connections_waiting, {"waiting"})
            metric_connections:set(ngx.var.connections_writing, {"writing"})
            prometheus:collect()
        }
    }
}

Update deployment configuration

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: ${name}
  namespace: ${namespace}
  labels:
    test-app: test-server
spec:
  replicas: ${replicas}
  template:
    metadata:
      labels:
        test-app: test-server
      Annotations: # < ------------------ NEW
        prometheus.io/scrape: "true"
        prometheus.io/path: "/metrics"
        prometheus.io/port: "8099"

summary

So far, the containerization of a Lua project has been completed, and there are still many problems encountered in the process. Only a few main steps and problems are recorded above.