MySQL container startup failure recovery case

Time:2022-5-7

After restarting the NAS yesterday, it was found that the NAS crashed again this morning. You can only forcibly shut down and restart again.
Before starting the docker container, I adjusted the memory of MySQL container to 512M. Then the tragic discovery couldn’t start. Later, it was found that the failure to start was not due to my memory adjustment.

View the log as follows:

2020-12-27T02:43:53.375776Z 0 mysqld: [Warning] World-writable config file ‘/etc/mysql/my.cnf’ is ignored.
2020-12-27T02:43:53.375776Z 0 mysqld: Error on realpath() on ‘/var/lib/mysql-files’ (Error 2 – No such file or directory)
2020-12-27T02:43:53.375776Z 0 [ERROR] [MY-010095] [Server] Failed to access directory for –secure-file-priv. Please make sure that directory exists and is accessible by MySQL Server. Supplied value : /var/lib/mysql-files
2020-12-27T02:43:53.376005Z 0 [ERROR] [MY-010119] [Server] Aborting

The solution is to re run a MySQL container and add it to the previous configuration-v /mnt/md0/User/wzp/home/www/mysql-files:/var/lib/mysql-files/

Now the problem comes. I didn’t record the complete run command before running MySQL container. Fortunately, you can view it through the rekcod tool. What’s more, this tool can be run through docker. Because my NAS system is not a distributed version of Linux, NPM and yum, many tools cannot be installed.

For the use of rekcod, please refer toLinuxea: how to reproduce and view the docker run parameter command

In short, the usage is as follows:

1. Docker installation rekcod

$ docker pull nexdrew/rekcod
$ alias rekcod="docker run --rm -v /var/run/docker.sock:/var/run/docker.sock nexdrew/rekcod"

2. Usage:

[ [email protected] -012664 ~]# rekcod MySQL = = > MySQL is my container name
docker run --name mysql --runtime runc -v /mnt/md0/User/wzp/home/www/mysql/:/var/lib/mysql -p 3306:3306/tcp --net bridge --restart no -h 39964e9e508a --expose 3306/tcp --expose 33060/tcp -e 'MYSQL_ROOT_PASSWORD=123456' -e 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin' -e 'GOSU_VERSION=1.12' -e 'MYSQL_MAJOR=5.7' -e 'MYSQL_VERSION=5.7.30-1debian10' -d -t -i --entrypoint "docker-entrypoint.sh" mysql 'mysqld'

You can see that my database files are placed in/mnt/md0/User/wzp/home/www/mysql/Under the directory, I checked the directory file, and the data should not be lost.

So I created mysql2 container with the following command
docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456 --name mysql2 -v /mnt/md0/User/wzp/home/www/mysql/:/var/lib/mysql -v /mnt/md0/User/wzp/home/www/mysql-files:/var/lib/mysql-files/ mysql

Then, enter mysql2 and find that you can’t connect to the database
docker exec -it mysql2 /bin/bash

View MySQL status

[email protected]:/# mysqld status
mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/docker.cnf' is ignored.
mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/mysql.cnf' is ignored.
2020-12-27T02:39:41.865252Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-12-27T02:39:41.865455Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20) starting as process 105
2020-12-27T02:39:41.871715Z 0 [ERROR] [MY-010123] [Server] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
2020-12-27T02:39:41.872541Z 0 [ERROR] [MY-010119] [Server] Aborting
2020-12-27T02:39:41.872776Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.20)  MySQL Community Server - GPL.
[email protected]:/# Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!^C

Start mysqld as root with the following method

[email protected]:/# mysqld --user=root
mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/docker.cnf' is ignored.
mysqld: [Warning] World-writable config file '/etc/mysql/conf.d/mysql.cnf' is ignored.
2020-12-27T02:40:57.169719Z 0 [Warning] [MY-011070] [Server] 'Disabling symbolic links using --skip-symbolic-links (or equivalent) is the default. Consider not using this option as it' is deprecated and will be removed in a future release.
2020-12-27T02:40:57.169896Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.20) starting as process 116
2020-12-27T02:40:57.184807Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2020-12-27T02:40:57.241048Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2020-12-27T02:40:58.241783Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2020-12-27T02:40:59.242983Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2020-12-27T02:41:00.244290Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2020-12-27T02:41:01.245762Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11
2020-12-27T02:41:02.247539Z 1 [ERROR] [MY-012574] [InnoDB] Unable to lock ./ibdata1 error: 11

The screen keeps printing unable to lock/ Ibdata1 error: 11, can only be interrupted by ctrl-c. Baidu said that there are two ways to solve this problem

1 reference[error] InnoDB of MySQL: unable to lock/ ibdata1, error: 11
According to the analysis of the article, the error is mainly caused by the following two reasons. Unfortunately, the methods in the article can not solve my problem.

  1. Insufficient disk space directory
  2. The ibdata1 file is occupied by another process

2 referenceReport. Sqlto: Unlock/ ibdata1 error: 11My article solved the problem perfectly. The main operations are as follows.

2.1 first, enter the directory where my MySQL database is located
cd /mnt/md0/User/wzp/home/www/mysql

2.2 then rename ibdata1 and other files, and then CP come back. I don’t quite understand why.

[[email protected] www]# cd /mnt/md0/User/wzp/home/www/mysql/
[[email protected] mysql]# ls
 auto.cnf          binlog.index      client-key.pem   dci            '#ib_16384_0.dblwr'   ib_logfile0     mautic               nianbao    performance_schema   robot             shangbiao          undo_001         zeng
 baike             ca-key.pem        company_works    dianzicaipiao  '#ib_16384_1.dblwr'   ib_logfile1     mysql                niuwan     private_key.pem      sara_wiki         sjzt_ry6           undo_002
 bigdatapaltfrom   ca.pem            copyrightdata    gs_data         ib_buffer_pool      '#innodb_temp'   mysql.ibd            pachong    public_key.pem       server-cert.pem   sys                wenshu
 binlog.000001     client-cert.pem   db_huayun        hy_dci_admin    ibdata1              integrate       mysql_upgrade_info   pachong2   qianliu_wiki         server-key.pem    ucenter_huayunyy   yuanqixiaoshuo
[[email protected] mysql]# mv ibdata1 ibdata1.bak
[[email protected] mysql]# mv ib_logfile0 ib_logfile0.bak
[[email protected] mysql]# mv ib_logfile1 ib_logfile1.bak
[[email protected] mysql]# cp -a ibdata1.bak ibdata1
[[email protected] mysql]# cp -a ib_logfile0.bak ib_logfile0
[[email protected] mysql]# cp -a ib_logfile1.bak ib_logfile1

2.3 then I delete the previous mysql2 container, run a mysql2 container again with the following command, and then I can connect to the MySQL database.

[[email protected] ~]# docker run -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=123456  --name mysql2 -v /mnt/md0/User/wzp/home/www/mysql/:/var/lib/mysql  -v /mnt/md0/User/wzp/home/www/mysql-files:/var/lib/mysql-files/    mysql
1e031247ea46e82f6205db68e7fb1b55389c87e5e2cb13517f9e1ac17d514509
[[email protected] ~]# docker exec -it mysql2 /bin/bash
[email protected]:/# mysql -uroot -p
mysql: [Warning] World-writable config file '/etc/mysql/my.cnf' is ignored.
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.20 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show databases;

Think everything will be all right when the MySQL container is ready? No,too naive! There’s another episode waiting for me. I just solved the MySQL container here, and suddenly found that my wiki container was mysteriously gone. But fortunately, with the database files, everything can be restored as before.

1 create two new wikis with the following command

docker run --name sarawiki --link mysql2:mysql -p 8086:80 -d sarawang85/mediawiki:1.0.0
docker run --name qianliuwiki --link mysql2:mysql -p 8083:80 -d sarawang85/mediawiki:1.0.0

2. Then backup the local settings Copy PHP files and logo images into the container and finish it.

PS, if you edit a container in stop status in the docker GUI interface, such as setting a memory limit, you will actually create a new container with the same name. Because I found that after I did this, I visited my Wiki and prompted me to reinstall.

Again, MySQL backup must be put on the agenda immediately. MySQL memory limit cannot be limited by container alone, my CNF also has to make changes in response.