Using SSH forward proxy and mongodump to realize remote scheduled backup of mongodb database

Time:2021-5-1

Recently, nodejs, express and mongodb have been used to do some back-end projects. The amount of data is not very large. At the beginning, we chose to create a dump folder in the main directory in the database backup section. When the program starts, we start a scheduled task to mongodump the current database at 1 a.m. every day. But always worried about the current server hard disk broken how to do, my data is not gone? So here’s an idea:On another server, I also backup the database on my project server regularly. Mongodump itself has powerful functions, which can back up the database remotely

Mongodump - H remote IP address - U login user - P login password - D database name -- port remote mongod port (default 27017) - O output path to the local machine

But now the servers in enterprises are all accessed through SSH,Mongodump provides the way of user name and password, but does not provide the function configuration of SSH accessSo this is my problem.

We know that in addition to login, SSH command has three proxy functions: forward proxy (- L), reverse proxy (- R), Socks5 proxy (- D). Here we can use forward proxy to proxy to a certain port of our local machine, and then mongodump the local machine to complete my purpose?

SSH - L: local agent port: localhost: remote mongod process port remote login user name @ remote server address

This machine here also refers to a server, because I want to write a nodejs application for backup and deploy it to a backup server.

The project contents are as follows:

Download - dump // database directory
Θ - app.js // server startup
Config.js // configuration file
(?) - cronfunc.js // timed task
├ - init.js // initialization function
// mongodump.js // back up the database
(?) - sslproxy.js // SSH forward proxy
├ - util. JS // tool function
├─package.json
├─README.md
├─server.json

Three packages are used in the project

"dependencies": {
    "Express": "^ 4.17.1" // start the service
    "Node CMD": "^ 4.0.0" // call the system terminal to execute the command
    "Node cron": "^ 2.0.3" // start timing task
  }
  • configuration file
// config.js
module.exports = {
    APP_ Port: 7003, // the port of this application

    DUMPPROXY_ Port: 9876, // Mongo local proxy port
    REMOTE_ MONGOD_ Port: 27017, // mongod service port of the server where the database is located
    REMOTE_ Host: '* * *', // server address of the database
    REMOTE_ Port: 22, // the SSH login port of the server where the database is located
    REMOTE_ User: 'root', // the login user name of the server where the database is located
    REMOTE_ DB_ Name: 'my db' // the name of the database to be backed up

    CRON_ RULE: '0 0 0/6 * * ?', // Specify the cron rule to perform scheduled backup
}

The core code includes two parts

  • SSH forward proxy
// sslProxy.js
const cmd = require('node-cmd');
const {DUMPPROXY_PORT,REMOTE_MONGOD_PORT,REMOTE_HOST,REMOTE_PORT,REMOTE_USER} = require('./config')
let fun = {}
fun.startProxy = async function () {
    try{
        const line = `ssh -L :${DUMPPROXY_PORT}:localhost:${REMOTE_MONGOD_PORT} ${REMOTE_USER}@${REMOTE_HOST} -p ${REMOTE_PORT}`;
        Console.log ('Start SSH forward proxy '+ line)
        cmd.get(line, (err, data) => {
            if (!err){
                Console.log ('agent succeeded -------- ')
            }else {
                Throw new error ('agent failed ')
            }
        })
    } catch(error) {
        Console.log ('agent failed + + + + + ~ + '+ error. MSG)
    }
}
module.exports = fun

Here, I don’t use the cmd.get method after the promise like in mongodump.js, because the program starts and runs startproxy first. In this case, when await, it will continue to block, resulting in the timing task can’t run, and the asynchronous method won’t affect it.

  • Backup server
// mongodump.js
const DUMP_ DIR = path.join(__ dirname, './dump'); //  Database backup address

/**
 *@ param {*} dirname directory name
 *@ param {*} dbname corresponds to the database name
 */
async function backup(dirname, dbname) {
    try {
        //Judgment questionnaire folder
        const saveDir = `${DUMP_DIR}${path.sep}${dirname}`;
        if (!fs.existsSync(DUMP_DIR)) {
            fs.mkdirSync(DUMP_DIR);
        }
        if (!fs.existsSync(saveDir)) {
            fs.mkdirSync(saveDir);
        }

        const line = `mongodump --port ${DUMPPROXY_PORT} -d ${dbname} -o ${saveDir}`;
        Console.log ('execute command: '+ line)
        //Make a backup
        await cmdPromise(line);

        Return dbname + 'database backup completed' + dirname;

    } catch (error) {
        Return ` backup failed - ${error. Message} ';
    }
};

In each backup, I simply use the timestamp as the path name, and the backup data database name is consistent with the database name.
After the project runs, you can first shorten the timer interval to test itCRON_RULE: '*/60 * * * * ?',
It’s OK for me to start on Linux (unbuntu16), and the functions are normal. The backup files are not neatly arranged by the timestamp pathname
Using SSH forward proxy and mongodump to realize remote scheduled backup of mongodb database
Open one, you can see the standard bson database file, you can use mongorestore for recovery.
Using SSH forward proxy and mongodump to realize remote scheduled backup of mongodb database

But when I start on windows, I will report this error, which leads to the failure of the function. Although I won’t use it on windows, I still want to know the reason. I hope you can tell me in the comments area!
Using SSH forward proxy and mongodump to realize remote scheduled backup of mongodb database

Attach GitHub project address:https://github.com/WkArtist/m…

finish