An important module in Python — asyncio

Time:2021-1-27

I’ve always been interested in asyncio. After all, it’s a highly concurrent module recommended by the official website. Python also introduces the concept of coroutine in Python 3.4. Also through this collation, we have a deeper understanding of the use of this module

What does asyncio do?

  • Asynchronous network operation
  • Concurrent
  • Xiecheng

In the python 3.0 era, the asynchronous network module in the standard library: Select (very low level) Python 3.0 era, the third-party asynchronous network library: Tornado Python 3.4 era, asyncio: support TCP, subprocess

Now asyncio has a lot of modules supported: aiohttp, aiodns, aioredis and so ongithub.com/aio-libsHere is a list of what has been supported and is being updated continuously

So far, of course, asyncio is not the only one to implement the coroutine. Tornado and gevent have implemented similar functions

Notes on some keywords of asyncio:

  • event_ Loop event loop: the program opens an infinite loop, registers some functions to the event loop, and calls the corresponding coroutine function when the event occurs

  • Coroutine: a coroutine object is a function defined with async keyword. Its call will not execute the function immediately, but will return a coroutine object. The coroutine object needs to be registered in the event loop and called by the event loop.

  • Task task: a coroutine object is a native function that can be suspended, and the task is to further encapsulate the coroutine, which contains various states of the task

  • Future: represents the result of tasks performed or not performed in the future. There is no essential difference between task and task

  • Async / await keyword: python3.5 is used to define the keyword of a coroutine, async defines a coroutine, and await is used to suspend the blocking asynchronous call interface.

After reading the above keywords, you may turn around and go. In fact, at the beginning of understanding and studying asyncio module, there is a kind of conflict, and you don’t know why. This also leads to a long period of time, this module has not been paid attention to and used. However, when you encounter various performance problems with Python in your work, you tell yourself to study hard There are two modules.

Define a coroutine

An important module in Python -- asyncio

import time import asyncio

now = lambda : time.time()

async def do_some_work(x): print(“waiting:”, x)

Start = now () # here is a coroutine object, do at this time_ some_ The work function is not executed

coroutine = do_ some_ Work (2) print (coroutine) ා create an event loop

loop = asyncio.get_ event_ Loop () # adds the coroutine to the event loop

loop.run_until_complete(coroutine) print(“Time:”,now()-start)

An important module in Python -- asyncio

In the above band, we define a coroutine by async keyword. Of course, coroutines cannot be run directly, and need to be added to the event loop

asyncio.get_ event_ Loop: create an event loop and use run_ until_ Complete registers the coroutine with the event loop and starts the event loop

Create a task

The coroutine object cannot be run directly. When registering an event loop, it is actually run_ until_ The complete method encapsulates the coroutine as a task object. The task object is a subclass of the future class, which stores the state of the coroutine after it runs and is used to obtain the results of the coroutine in the future

An important module in Python -- asyncio

import asyncio import time

now = lambda: time.time()

async def do_some_work(x): print(“waiting:”, x)

start = now()

coroutine = do_some_work(2)

loop = asyncio.get_event_loop()

task = loop.create_task(coroutine) print(task)

loop.run_until_complete(task) print(task) print(“Time:”,now()-start)

An important module in Python -- asyncio

The results were as follows

<Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex2.py:13>> waiting: 2

<Task finished coro=<do_some_work() done, defined at /app/py_code/study_asyncio/simple_ex2.py:13> result=None> Time: 0.0003514289855957031

After the task is created, it is in the pending state before the task is added to the event loop. When it is completed, it is in the finished state

About the above pass loop.create_ Task (coroutine) to create a task asyncio.ensure_ Future (coroutine) create task

Explanation on the official website of the two orders:docs.python.org/3/library/asyncio-…

asyncio.ensure_future(coro_or_future, *, loop=None)¶

Schedule the execution of a coroutine object: wrap it in a future. Return a Task object.

If the argument is a Future, it is returned directly.

docs.python.org/3/library/asyncio-…

An important module in Python -- asyncio

AbstractEventLoop.create_task(coro)

Schedule the execution of a coroutine object: wrap it in a future. Return a Task object.

Third-party event loops can use their own subclass of Task for interoperability. In this case, the result type is a subclass of Task.

This method was added in Python 3.4.2. Use the async() function to support also older Python versions.

An important module in Python -- asyncio

Binding callback

Binding callback can get the execution result when the task is finished. The last parameter of the callback is the future object, through which the return value of the coroutine can be obtained.

An important module in Python -- asyncio

import time import asyncio

now = lambda : time.time()

async def do_some_work(x): print(“waiting:”,x) return “Done after {}s”.format(x) def callback(future): print(“callback:”,future.result())

start = now()

coroutine = do_some_work(2)

loop = asyncio.get_event_loop()

task = asyncio.ensure_future(coroutine) print(task)

task.add_done_callback(callback) print(task)

loop.run_until_complete(task) print(“Time:”, now()-start)

An important module in Python -- asyncio

The results were as follows

<Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex3.py:13>>

<Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex3.py:13> cb=[callback() at /app/py_code/study_asyncio/simple_ex3.py:18]> waiting: 2 callback: Done after 2s

Time: 0.00039196014404296875

Through add_ done_ The callback method adds a callback function to the task. When the task (or coroutine) is completed, the callback function will be called. The future parameter is used to get the result of the implementation. Here, the task we create and the future object in the callback are actually the same object

Blocking and await

Async can be used to define the coroutine object, await can be used to suspend the time-consuming operation, just like yield in the generator, the function gives up control. When a coroutine encounters await, the event loop will suspend the coroutine and execute other coroutines until other coroutines are suspended or executed, and then the next coroutine will be executed

Time consuming operations are usually IO operations, such as network request, file reading, etc. We use asyncio.sleep Function to simulate IO operations. The purpose of the coroutine is to make these IO operations asynchronous.

An important module in Python -- asyncio

import asyncio import time

now = lambda :time.time()

async def do_ some_ Work (x): Print (“waiting:” x) # await is followed by calling time-consuming operation

await asyncio.sleep(x) return “Done after {}s”.format(x)

start = now()

coroutine = do_some_work(2)

loop = asyncio.get_event_loop()

task = asyncio.ensure_future(coroutine)

loop.run_until_complete(task) print(“Task ret:”, task.result()) print(“Time:”, now() – start)

An important module in Python -- asyncio

In await asyncio.sleep (x) Because sleep here simulates blocking or time-consuming operation. At this time, control will be given up. In other words, when a function is blocked, await method is used to give up the control of the coroutine so that the loop can call other coroutines.

Concurrency and parallelism

Concurrency refers to a system with multiple activities at the same time

Parallelism is worth using concurrency to make a system run faster. Parallelism can be used in many abstract levels of the operating system

So concurrency usually means that there are multiple tasks need to be carried out at the same time, while parallelism means that there are multiple tasks to be executed at the same time

The following example is very vivid:

Concurrent case is a teacher at the same time to assist different people homework. In parallel, several teachers assist multiple students in their homework at the same time. In short, if one person eats three steamed buns at the same time or three people eat one at the same time, eating one steamed bun is a task

An important module in Python -- asyncio

import asyncio import time

now = lambda :time.time()

async def do_some_work(x): print(“Waiting:”,x)

await asyncio.sleep(x) return “Done after {}s”.format(x)

start = now()

coroutine1 = do_some_work(1)

coroutine2 = do_some_work(2)

coroutine3 = do_some_work(4)

tasks = [

asyncio.ensure_future(coroutine1),

asyncio.ensure_future(coroutine2),

asyncio.ensure_future(coroutine3)

]

loop = asyncio.get_event_loop()

loop.run_until_complete(asyncio.wait(tasks)) for task in tasks: print(“Task ret:”,task.result()) print(“Time:”,now()-start)

An important module in Python -- asyncio

Results of operation:

An important module in Python -- asyncio

Waiting: 1 Waiting: 2 Waiting: 4 Task ret: Done after 1s

Task ret: Done after 2s

Task ret: Done after 4s

Time: 4.004154920578003

An important module in Python -- asyncio

The total time is about 4 seconds. The blocking time of 4 S is enough for the first two coroutines to complete. If it is a synchronous task, it will take at least 7 seconds. At this point, we use aysncio to achieve concurrency. asyncio.wait (tasks) can also be used asyncio.gather (* tasks), the former receives a list of tasks, and the latter receives a bunch of tasks.

about asyncio.gather and asyncio.wait Description of the official website:

docs.python.org/3/library/asyncio-…

Return a future aggregating results from the given coroutine objects or futures.

All futures must share the same event loop. If all the tasks are done successfully, the returned future’s result is the list of results (in the order of the original sequence, not necessarily the order of results arrival). If return_exceptions is true, exceptions in the tasks are treated the same as successful results, and gathered in the result list; otherwise, the first raised exception will be immediately propagated to the returned future.

docs.python.org/3/library/asyncio-…

An important module in Python -- asyncio

Wait for the Futures and coroutine objects given by the sequence futures to complete. Coroutines will be wrapped in Tasks. Returns two sets of Future: (done, pending).

The sequence futures must not be empty.

timeout can be used to control the maximum number of seconds to wait before returning. timeout can be an int or float. If timeout is not specified or None, there is no limit to the wait time.

return_when indicates when this function should return.

An important module in Python -- asyncio

Coroutine nesting

Using async, we can define coroutines. Coroutines are used for time-consuming IO operations. We can also encapsulate more IO operations. In this way, nested coroutines are realized, that is, one coroutine await another coroutine, which is connected in this way.

An important module in Python -- asyncio

import asyncio import time

now = lambda: time.time()

async def do_some_work(x): print(“waiting:”,x)

await asyncio.sleep(x) return “Done after {}s”.format(x)

async def main():

coroutine1 = do_some_work(1)

coroutine2 = do_some_work(2)

coroutine3 = do_some_work(4)

tasks = [

asyncio.ensure_future(coroutine1),

asyncio.ensure_future(coroutine2),

asyncio.ensure_future(coroutine3)

]

dones, pendings = await asyncio.wait(tasks) for task in dones: print("Task ret:", task.result()) # results = await asyncio.gather(*tasks)
# for result in results:
# print("Task ret:",result)

start = now()

loop = asyncio.get_event_loop()

loop.run_until_complete(main()) print(“Time:”, now()-start)

An important module in Python -- asyncio

If we put the following in the above code:

dones, pendings = await asyncio.wait(tasks) for task in dones: print("Task ret:", task.result())

Replace with:

results = await asyncio.gather(*tasks) for result in results: print("Task ret:",result)

The result is a list of results

If you do not process the result in the main coprocessor function and return the await directly, the outermost run_ until_ Complete will return the result of the main coroutine. Change the above code to:

An important module in Python -- asyncio

import asyncio import time

now = lambda: time.time()

async def do_some_work(x): print(“waiting:”,x)

await asyncio.sleep(x) return “Done after {}s”.format(x)

async def main():

coroutine1 = do_some_work(1)

coroutine2 = do_some_work(2)

coroutine3 = do_some_work(4)

tasks = [

asyncio.ensure_future(coroutine1),

asyncio.ensure_future(coroutine2),

asyncio.ensure_future(coroutine3)

] return await asyncio.gather(*tasks)

start = now()

loop = asyncio.get_event_loop()

results = loop.run_until_complete(main()) for result in results: print(“Task ret:”,result) print(“Time:”, now()-start)

An important module in Python -- asyncio

Or return to use asyncio.wait Method to suspend the coroutine.

Change the code to:

An important module in Python -- asyncio

import asyncio import time

now = lambda: time.time()

async def do_some_work(x): print(“waiting:”,x)

await asyncio.sleep(x) return “Done after {}s”.format(x)

async def main():

coroutine1 = do_some_work(1)

coroutine2 = do_some_work(2)

coroutine3 = do_some_work(4)

tasks = [

asyncio.ensure_future(coroutine1),

asyncio.ensure_future(coroutine2),

asyncio.ensure_future(coroutine3)

] return await asyncio.wait(tasks)

start = now()

loop = asyncio.get_event_loop()

done,pending = loop.run_until_complete(main()) for task in done: print(“Task ret:”,task.result()) print(“Time:”, now()-start)

An important module in Python -- asyncio

You can also use asyncio’s as_ Completed method

An important module in Python -- asyncio

import asyncio import time

now = lambda: time.time()

async def do_some_work(x): print(“waiting:”,x)

await asyncio.sleep(x) return “Done after {}s”.format(x)

async def main():

coroutine1 = do_some_work(1)

coroutine2 = do_some_work(2)

coroutine3 = do_some_work(4)

tasks = [

asyncio.ensure_future(coroutine1),

asyncio.ensure_future(coroutine2),

asyncio.ensure_future(coroutine3)

] for task in asyncio.as_completed(tasks):

result = await task print(“Task ret: {}”.format(result))

start = now()

loop = asyncio.get_event_loop()

loop.run_until_complete(main()) print(“Time:”, now()-start)

An important module in Python -- asyncio

It can also be seen from the above that the calling and combination of coroutines are very flexible, mainly reflected in the processing of results: how to return and how to suspend

Stop of coroutine

The future object has several states:

  • Pending
  • Running
  • Done
  • Cacelled

When creating future, the task is pending. Of course, when the event loop is called and executed, it is running. After the call, it is done. If you need to stop the event loop, you need to cancel the task first. have access to asyncio.Task Gets the task of the event loop

An important module in Python -- asyncio

import asyncio import time

now = lambda :time.time()

async def do_some_work(x): print(“Waiting:”,x)

await asyncio.sleep(x) return “Done after {}s”.format(x)

coroutine1 =do_some_work(1)

coroutine2 =do_some_work(2)

coroutine3 =do_some_work(2)

tasks = [

asyncio.ensure_future(coroutine1),

asyncio.ensure_future(coroutine2),

asyncio.ensure_future(coroutine3),

]

start = now()

loop = asyncio.get_event_loop() try:

loop.run_until_complete(asyncio.wait(tasks)) except KeyboardInterrupt as e: print(asyncio.Task.all_tasks()) for task in asyncio.Task.all_tasks(): print(task.cancel())

loop.stop()

loop.run_forever() finally:

loop.close() print(“Time:”,now()-start)

An important module in Python -- asyncio

After the event loop is started, Ctrl + C will trigger run immediately_ until_ The execution exception of complete is keyborardinterrupt. And then through the loop asyncio.Task Cancel future. You can see the output as follows:

An important module in Python -- asyncio

Waiting: 1 Waiting: 2 Waiting: 2

^C{<Task finished coro=<do_some_work() done, defined at /app/py_code/study_asyncio/simple_ex10.py:13> result=’Done after 1s’>, <Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex10.py:15> wait_for= cb=[_wait.._on_completion() at /usr/local/lib/python3.5/asyncio/tasks.py:428]>, <Task pending coro=<do_some_work() running at /app/py_code/study_asyncio/simple_ex10.py:15> wait_for= cb=[_wait.._on_completion() at /usr/local/lib/python3.5/asyncio/tasks.py:428]>, <Task pending coro=<wait() running at /usr/local/lib/python3.5/asyncio/tasks.py:361> wait_for=>}

False

True

True

True

Time: 1.0707225799560547

An important module in Python -- asyncio

True means cannel is successful. After loop stop, you need to open the event loop again, and finally close it. Otherwise, an exception will be thrown

It is a scheme to loop tasks and cancel them one by one. However, as mentioned above, we encapsulate the list of tasks in the main function, and call the event loop outside the main function. At this time, main is equivalent to the most outgoing task, so you can handle the main function of the wrapper.

Event loops in different threads

Many times, our event loop is used to register coroutines, and some coroutines need to be added to the event loop dynamically. A simple way is to use multithreading. The current thread creates an event loop, and then creates a new thread to start the event loop in the new thread. The current thread will not be blocked.

An important module in Python -- asyncio

import asyncio from threading import Thread import time

now = lambda :time.time() def start_loop(loop):

asyncio.set_event_loop(loop)

loop.run_forever() def more_work(x): print(‘More work {}’.format(x))

time.sleep(x) print(‘Finished more work {}’.format(x))

start = now()

new_loop = asyncio.new_event_loop()

t = Thread(target=start_loop, args=(new_loop,))

t.start() print(‘TIME: {}’.format(time.time() – start))

new_loop.call_soon_threadsafe(more_work, 6)

new_loop.call_soon_threadsafe(more_work, 3)

An important module in Python -- asyncio

After starting the above code, the current thread will not be blocked, and the new thread will execute calls in sequence_ soon_ More of threadsafe method registration_ The latter is because time.sleep The operation is synchronously blocked, so it is finished_ Work needs about 6 + 3

New thread coroutine

An important module in Python -- asyncio

import asyncio import time from threading import Thread

now = lambda :time.time() def start_loop(loop):

asyncio.set_event_loop(loop)

loop.run_forever()

async def do_some_work(x): print(‘Waiting {}’.format(x))

await asyncio.sleep(x) print(‘Done after {}s’.format(x)) def more_work(x): print(‘More work {}’.format(x))

time.sleep(x) print(‘Finished more work {}’.format(x))

start = now()

new_loop = asyncio.new_event_loop()

t = Thread(target=start_loop, args=(new_loop,))

t.start() print(‘TIME: {}’.format(time.time() – start))

asyncio.run_coroutine_threadsafe(do_some_work(6), new_loop)

asyncio.run_coroutine_threadsafe(do_some_work(4), new_loop)

An important module in Python -- asyncio

In the above example, a new is created in the main thread_ Loop, and then open an infinite event loop in another child thread. The main thread runs through run_ coroutine_ Threadsafe newly registers the helper object. In this way, the concurrent operation of event loop can be carried out in the sub thread, and the main thread will not be blocked. The total execution time is about 6 seconds.

All efforts are worth expecting, every dream should be irrigated!

This work adoptsCC agreementReprint must indicate the author and the link of this article