Forsun is a high-performance and high-precision timing service, which can easily manage tens of millions of timing tasks.

Project address:https://github.com/snower/forsun

Or lookHigh performance ten million level timed task management service forsun usage details

Forsun built-in supports shell, HTTP, redis, thrift, beanstalk and MYSQL to trigger callback executors at the appointed time. However, in many cases, its own project needs are various. A single built-in executor can not be well integrated in its own project, so forsun also supports the registration of its own trigger executor action through extension development.

So let’s easily and happily implement an extension integrating celery.

Example (implement a celery actuator extension)

Add code celery_ extension. py

# -*- coding: utf-8 -*-

import json
import logging
from concurrent.futures import ThreadPoolExecutor
from celery import Celery
from forsun.extension import Extension
from forsun.action.action import Action

app = Celery('hello', broker='amqp://[email protected]//')
executor = ThreadPoolExecutor()


@app.task
def hello():
    return 'hello world'


@app.task
def add(x, y):
    return x + y


class CeleryAction(Action):
    METHODS = {
        "hello": hello,
        "add": add,
    }

    async def execute(self, *args, **kwargs):
        method = self.params.get("method", '')
        args = json.loads(self.params.get('args', '[]'))
        kwargs = json.loads(self.params.get('kwargs', '{}'))
        if method not in self.METHODS:
            logging.info("celery action execute unknow method %s", method)
            return
        executor.submit(self.METHODS[method], *tuple(args), **kwargs)
        logging.info("celery action execute %s", method)


class CeleryExtension(Extension):
    name = "celery"

    def register(self):
        self.register_action("celery", CeleryAction)

It can be seen that implementing an extension is very simple. Define an extension class celeryextension, which inherits from forsun extension. Extension, add an executor, and celeryaction inherits from forsun extension. Action, name it, register the executor celeryaction in the register function of the extension class, and finish it.

When the created timed task expires and triggers, the execute function of celeryaction will be called automatically. The TS attribute of the current action instance stores the task trigger time. Params is the params parameter of the created timed task. Extract the parameter and continue to complete the parsing.

It should be noted that because the whole forsun service is implemented by tornado asynchronous IO, the execution of action will use asynchronous calls. If you need to make synchronous blocking calls, it is recommended to put the methods to be executed into ThreadPoolExecutor for better performance.

Add startup parameter load extension

forsund --bind=0.0.0.0 --port=6458 --http=0.0.0.0:8001 --log=/var/log/forsun.log --log-level=INFO --driver=mem --driver-mem-store-file=/var/lib/fousun/forsun.session --extension-path=./ --extension=celery_extension.CeleryExtension

If you use the conf file to configure, you can also add the extension loading parameter to the conf file.

#Extended configuration

At this time, check the log output and you will find that the extension has been successfully loaded.

2020-03-20 14:09:20,650 1022 INFO register extension path ./
2020-03-20 14:09:20,762 1022 INFO load extension celery_extension.CeleryExtension <class 'celery_extension.CeleryExtension'>
2020-03-20 14:09:20,762 1022 INFO action register celery <class 'celery_extension.CeleryAction'>

Check the current status information through the info name, and you can find that there is already celery support in the supported actions list. It’s great. Now start your project journey happily.

forsun info

python_version: 3.6.9 (default, Nov  7 2019, 10:44:02)  [GCC 8.3.0]
forsun_version: 0.1.3
start_time:     2020-03-20 14:31:27.538081+08:00
cpu_user:       0.18
cpu_system:     0.1
mem_rss:        28.06M
mem_vms:        122.46M
current_time:   2020-03-20 14:31:38+08:00

stores: mem;redis
current_store:  mem
actions:        shell;http;redis;thrift;celery
bind_port:      0.0.0.0:6458
http_bind_port:
extensions:     celery_extension.CeleryExtension

HTTP request test

curl -X PUT -H 'Content-Type: application/json' -d '{"key": "test", "seconds": 5, "minute": 0, "hour": 0, "day": 0, "month": 0, "count": 1, "action": "celery", "params": {"method": "hello"}}' http://127.0.0.1:8001/v1/plan

{"errcode": 0, "errmsg": "", "data": {"key": "test", "second": 5, "minute": 0, "hour": 0, "day": 0, "month": 0, "week": -1, "status": 0, "count": 0, "is_time_out": true, "next_time": 1584657610, "current_count": 0, "last_timeout": 0, "created_time": 1584657605.0, "action": "celery", "params": {"method": "hello"}}}

Wait for 5 seconds and you will see the successful call of celery.

last

In addition to adding custom executor actions through the extension, forsun can also customize the persistence storage through the extension. I’ll talk about this later.

In the actual project, we have a lot of large and programmable timing scheduling needs such as order payment timeout and distribution timeout. A convenient and high-performance and accurate timing task scheduling service can obviously greatly save our time. Why can’t we have more fun games.

This work adoptsCC agreementThe author, link and reprint of this article must be indicated