Build from scratch Node.js Enterprise web server (11): timed tasks

Time:2021-1-28

Configure scheduled tasks

Timing task, which is the execution process triggered by time, is a very common business logic. In the early version of UNIX, cron, a timing task scheduling module, has been used in various Linux systems. Crontab, crontab’s configuration file, has a comprehensive but clear format, which can solve the problem of timing task configuration in most scenarios. Enterprise servers can use crontab like format to flexibly configure various timing task logic. The following is the crontab format:

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

This chapter will be based on the work done in the previous chapterhost1-tech/nodejs-server-examples – 10-logadoptnode-scheduleThe timing task is configured in a way similar to crontab to detect the shop information that may contain network attacks, and then the shop information is sent through thenodemailerSend suspicious store information e-mail to the administrator. Execute the installation command of node schedule and nodemailer in the project root directory

$yarn add node schedule nodemailer ᦇ local installation of node schedule and nodemailer
# ...
info Direct dependencies
├─ [email protected]
└─ [email protected]
# ...

Network attack inspection

Now the logic of timing detection and alarm of network attack information is realized. First, add the service layer logic

// src/services/shop.js
const { Shop } = require('../models');

class ShopService {
  async init() {}

-  async find({ id, pageIndex = 0, pageSize = 10, logging }) {
+  async find({ id, pageIndex = 0, pageSize = 10, where, logging }) {
    if (id) {
      return [await Shop.findByPk(id, { logging })];
    }

    return await Shop.findAll({
      offset: pageIndex * pageSize,
      limit: pageSize,
+      where,
      logging,
    });
  }
  // ...
}
// ...
// src/services/mail.js
const { promisify } = require('util');
const nodemailer = require('nodemailer');
const { mailerOptions } = require('../config');

class MailService {
  mailer;

  async init() {
    this.mailer = nodemailer.createTransport(mailerOptions);
    await promisify(this.mailer.verify)();
  }

  async sendMail(params) {
    return await this.mailer.sendMail({
      from: mailerOptions.auth.user,
      ...params,
    });
  }
}

let service;
module.exports = async () => {
  if (!service) {
    service = new MailService();
    await service.init();
  }
  return service;
};
// src/config/index.js
const merge = require('lodash.merge');
const logger = require('../utils/logger');
const { logging } = logger;

const config = {
  //Default configuration
  default: {
    // ...
+    mailerOptions: {
+      host: 'smtp.126.com',
+      port: 465,
+      secure: true,
+      logger: logger.child({ type: 'mail' }),
+      auth: {
+        user: process.env.MAILER_USER,
+        pass: process.env.MAILER_PASS,
+      },
+    },
  },
  // ...
};
// ...
# .env.local
GITHUB_CLIENT_ID='b8ada004c6d682426cfb'
GITHUB_CLIENT_SECRET='0b13f2ab5651f33f879a535fc2b316c6c731a041'
+
+MAILER_USER='[email protected]'
+MAILER_PASS='CAEJHSTBWNOKHRVL'

Note that since there may be more than one application node, distributed locks will be used to limit the number of execution nodes to avoid repeated alarms

$# generate the model file and schema migration file of timing task lock
$ yarn sequelize model:generate --name scheduleLock --attributes name:string,counter:integer

$# Src / Models/ schedulelock.js  Named Src / models/ scheduleLock.js
$ mv src/models/schedulelock.js src/models/scheduleLock.js

$tree Src / models # show Src / models directory content structure
src/models
├── config
│   └── index.js
├── index.js
├── migrate
│   ├── 20200725045100-create-shop.js
│   ├── 20200727025727-create-session.js
│   └── 20200801120113-create-schedule-lock.js
├── scheduleLock.js
├── seed
│   └── 20200725050230-first-shop.js
└── shop.js

adjustmentsrc/models/scheduleLock.jsAndsrc/models/migrate/20200801120113-create-schedule-lock.js

// src/models/scheduleLock.js
const { Model } = require('sequelize');

module.exports = (sequelize, DataTypes) => {
  class scheduleLock extends Model {
    /**
     * Helper method for defining associations.
     * This method is not a part of Sequelize lifecycle.
     * The `models/index` file will call this method automatically.
     */
    static associate(models) {
      // define association here
    }
  }
  scheduleLock.init(
    {
      name: DataTypes.STRING,
      counter: DataTypes.INTEGER,
    },
    {
      sequelize,
      modelName: 'ScheduleLock',
      tableName: 'schedule_lock',
    }
  );
  return scheduleLock;
};
// src/models/migrate/20200801120113-create-schedule-lock.js
module.exports = {
  up: async (queryInterface, Sequelize) => {
    await queryInterface.createTable('schedule_lock', {
      id: {
        allowNull: false,
        autoIncrement: true,
        primaryKey: true,
        type: Sequelize.INTEGER,
      },
      name: {
        type: Sequelize.STRING,
      },
      counter: {
        type: Sequelize.INTEGER,
      },
      created_at: {
        allowNull: false,
        type: Sequelize.DATE,
      },
      updated_at: {
        allowNull: false,
        type: Sequelize.DATE,
      },
    });
  },
  down: async (queryInterface, Sequelize) => {
    await queryInterface.dropTable('schedule_lock');
  },
};

Then write the patrol logic:

$MKDIR Src / schedules # new Src / schedules to store scheduled tasks

$tree SRC - L 1 # show SRC directory content structure
src
├── config
├── controllers
├── middlewares
├── models
├── moulds
├── schedules
├── server.js
├── services
└── utils
// src/schedules/inspectAttack.js
const { basename } = require('path');
const schedule = require('node-schedule');
const { sequelize, ScheduleLock, Sequelize } = require('../models');
const mailService = require('../services/mail');
const shopService = require('../services/shop');
const escapeHtmlInObject = require('../utils/escape-html-in-object');
const logger = require('../utils/logger');
const { Op } = Sequelize;

//The lock name of the current task
const LOCK_NAME = basename(__dirname);
//Maximum lock occupancy time
const LOCK_TIMEOUT = 15 * 60 * 1000;
//Concurrent number of distributed tasks
const CONCURRENCY = 1;
//Alarm mail sending object
const MAIL_RECEIVER = '[email protected]';

class InspectAttack {
  mailService;
  shopService;

  async init() {
    this.mailService = await mailService();
    this.shopService = await shopService();

    //Patrol every 15 minutes
    schedule.scheduleJob('*/15 * * * *', this.findAttackedShopInfoAndSendMail);
  }

  findAttackedShopInfoAndSendMail = async () => {
    //Lock
    const lockUpT = await sequelize.transaction();
    try {
      const [lock] = await ScheduleLock.findOrCreate({
        where: { name: LOCK_NAME },
        defaults: { name: LOCK_NAME, counter: 0 },
        transaction: lockUpT,
      });

      if (lock.counter >= CONCURRENCY) {
        if (Date.now() - lock.updatedAt.valueOf() > LOCK_TIMEOUT) {
          lock.counter--;
          await lock.save({ transaction: lockUpT });
        }
        await lockUpT.commit();
        return;
      }

      lock.counter++;
      await lock.save({ transaction: lockUpT });
      await lockUpT.commit();
    } catch (err) {
      logger.error(err);
      await lockUpT.rollback();
      return;
    }

    try {
      //Looking for abnormal data
      const shops = await this.shopService.find({
        pageSize: 100,
        where: {
          name: { [Op.or]: [{ [Op.like]: '<%' }, { [Op.like]: '%>' }] },
        },
      });

      //Send alarm email
      if (shops.length) {
        Const subject ='security warning, suspicious store information found! ';
        const html = `
  < div > the following is the store information suspected to contain network attacks found by the server Patrol: < / div > the store information suspected to contain network attacks
  <pre>
  ${shops
    .map((shop) => JSON.stringify(escapeHtmlInObject(shop), null, 2))
    .join('\n')}
  </pre>`;
        await this.mailService.sendMail({ to: MAIL_RECEIVER, subject, html });
      }
    } catch {}

    //Unlocking
    const lockDownT = await sequelize.transaction();
    try {
      const lock = await ScheduleLock.findOne({
        where: { name: LOCK_NAME },
        transaction: lockDownT,
      });
      if (lock.counter > 0) {
        lock.counter--;
        await lock.save({ transaction: lockDownT });
      }
      await lockDownT.commit();
    } catch {
      await lockDownT.rollback();
    }
  };
}

module.exports = async () => {
  const s = new InspectAttack();
  await s.init();
};
// src/schedules/index.js
const inspectAttackSchedule = require('./inspectAttack');

module.exports = async function initSchedules() {
  await inspectAttackSchedule();
};
// src/server.js
const express = require('express');
const { resolve } = require('path');
const { promisify } = require('util');
const initMiddlewares = require('./middlewares');
const initControllers = require('./controllers');
+const initSchedules = require('./schedules');
const logger = require('./utils/logger');

const server = express();
const port = parseInt(process.env.PORT || '9000');
const publicDir = resolve('public');
const mouldsDir = resolve('src/moulds');

async function bootstrap() {
  server.use(await initMiddlewares());
  server.use(express.static(publicDir));
  server.use('/moulds', express.static(mouldsDir));
  server.use(await initControllers());
  server.use(errorHandler);
+  await initSchedules();
  await promisify(server.listen.bind(server, port))();
  logger.info(`> Started on port ${port}`);
}
// ...

View alarms

After adding two new store information containing network attacks, you can receive a warning email when the number of minutes is a multiple of 15:

Build from scratch Node.js  Enterprise web server (11): timed tasks

Build from scratch Node.js  Enterprise web server (11): timed tasks

Source code of this chapter

host1-tech/nodejs-server-examples – 11-schedule

Read more

Build from scratch Node.js Enterprise web server (zero): static services
Build from scratch Node.js Enterprise web server (1): interface and layering
Build from scratch Node.js Enterprise web server (2): Verification
Build from scratch Node.js Enterprise web server (3): Middleware
Build from scratch Node.js Enterprise web server (4): exception handling
Build from scratch Node.js Enterprise web server (5): database access
Build from scratch Node.js Enterprise web server (6): session
Build from scratch Node.js Enterprise web server (7): authentication login
Build from scratch Node.js Enterprise web server (8): network security
Build from scratch Node.js Enterprise web server (9): configuration items
Build from scratch Node.js Enterprise web server (x): log
Build from scratch Node.js Enterprise web server (11): timed tasks
Build from scratch Node.js Enterprise web server (12): remote call
Build from scratch Node.js Enterprise web server (XIII): breakpoint debugging and performance analysis
Build from scratch Node.js Enterprise web server (14): automated testing
Build from scratch Node.js Enterprise web server (XV): summary and Prospect

Recommended Today

Practice of query operation of database table (Experiment 3)

Following the previous two experiments, this experiment is to master the use of select statements for various query operations: single table query, multi table connection and query, nested query, set query, to consolidate the database query operation.Now follow Xiaobian to practice together!Based on the data table (student, course, SC, teacher, TC) created and inserted in […]