Usage Summary of Python collaborative process (I)

Time:2022-5-6

1、 Python collaboration keyword

Async: declare collaboration

Note: the function declared through async is not a function, and a separate call will not be executed

Await: it is used to wait for objects that can wait, including collaboration (that is, the collaboration declared by async), task (asyncio. Create_task(), asyncio ensure_ Future () is the task created) and future (it is a low-level waiting object, which represents the final result of an asynchronous operation. Generally, you don’t create it yourself)

Note: the current coordination process will wait for blocking at await (go back to execute other coordination processes) until the waiting coordination process or task is completed

2、 Waiting object

There are three main types of waiting objects: collaboration, task and future. The keyword await can be used to wait for the completion of events

Synergetic process

  • Synergetic function: a function defined in the form of async def;
  • Coroutine object: the object returned by calling the coroutine function.

task

  • Via asyncio create_ Task() will create a task with the collaboration function, which will be automatically scheduled for immediate operation

    Note: Python > = 3.7
  • python3. 6. Use asyncio ensure_ Future() create task

Future

  • Is a special low-level waiting object that represents the final result of an asynchronous operation.
  • When a future object is waiting, this means that the coroutine will remain waiting until the future object is operated elsewhere.
  • It is generally not necessary to create future objects in application level code

You can use the following methods

  • asyncio.gather(*coros_or_futures, loop=None, return_exceptions=False)

    1. If coros_ or_ A waiting object in futures * is a collaborative process, which will be automatically added to the schedule as a task
    2. If all the waiting objects complete successfully, the result will be a list of all the return values aggregated. The order of the result values is consistent with that of the objects that can be waited for in AWS.
    3. If return_ The first exception that will be thrown by the first task (exception) will be false by default. Other waiting objects in the AWS sequence will not be cancelled and will continue to run. If return_ If exceptions is true, exceptions will be handled as successful results and aggregated into the result list.
    4. If gather () is cancelled, all the pending objects that have been submitted (not yet completed) will also be cancelled.

      If any task or future object in the AWS sequence is cancelled, it will be treated as if cancelederror was raised – in this case, the gather() call will not be cancelled. This is to prevent a submitted task / future from being cancelled, resulting in the cancellation of other tasks / future.
  • asyncio.shield(arg, *, loop=None)

    1. Protect a waiting object from being cancelled.
    2. Arg is a collaborative process, which will be automatically added to the schedule as a task.
  • asyncio.wait_for(fut, timeout, *, loop=None)

    1. It is used to wait for a future and set timeout. Timeout can be none or the number of waiting seconds represented by float or int. If timeout is none, wait until it is complete.
    2. If a timeout occurs, the task will be canceled and asyncio.exe will be raised TimeoutError.
    3. To avoid task cancellation, you can add shield ().
  • asyncio.wait(fs, *, loop=None, timeout=None, return_when=ALL_COMPLETED)

    import asyncio
    
    async def wait1():
        print('in 1')
        await asyncio.sleep(1)
        return 1
    
    async def wait2():
        print('in 2')
        await asyncio.sleep(2)
        return 2
async def wait3():
      print('in 3')
      await asyncio.sleep(3)
      return 3


  async def main():
      done, pending = await asyncio.wait([wait1(), wait2(), wait3()], timeout=2.5)
      print(done,'\n', pending)

  asyncio.run(main())

result

in 1

in 3

in 2

{< task finished Coro = < wait1 () done, defined at / users / zhangjintao / desktop / essay / small test code / multitask / collaboration / 3.7 / test. Py: 3 > result = 1 >, < task finished Coro = < wait2 () done, defined at / users / zhangjintao / desktop / essay / small test code / multitask / collaboration / 3.7 / test. Py: 8 > result = 2 >}

{< task pending Coro = < wait3 () running at / users / zhangjintao / desktop / essay / small test code / multitask / collaboration / 3.7 / test. Py: 16 > wait_for = < future pending CB = [< taskwakeupmethwrapper object at 0x10acc2650 > ()] >}

  1. Timeout timeout will not cause exceptions. Timeout will return the status of uncompleted tasks. Done is the set of completed tasks and pending is the set of uncompleted tasks
  2. If timeout is not set, the result will be returned by default after all is completed, and pending returns an empty set

3、 Then execute the code in the thread pool

loop.run_in_executor`(executor, func, *args)

import asyncio
import concurrent.futures

def blocking_io():
    # File operations (such as logging) can block the
    # event loop: run them in a thread pool.
    with open('/dev/urandom', 'rb') as f:
        return f.read(100)

def cpu_bound():
    # CPU-bound operations will block the event loop:
    # in general it is preferable to run them in a
    # process pool.
    return sum(i * i for i in range(10 ** 7))

async def main():
    loop = asyncio.get_running_loop()

    ## Options:

    # 1. Run in the default loop's executor:
    result = await loop.run_in_executor(
        None, blocking_io)
    print('default thread pool', result)

    # 2. Run in a custom thread pool:
    with concurrent.futures.ThreadPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, blocking_io)
        print('custom thread pool', result)

    # 3. Run in a custom process pool:
    with concurrent.futures.ProcessPoolExecutor() as pool:
        result = await loop.run_in_executor(
            pool, cpu_bound)
        print('custom process pool', result)

asyncio.run(main())

See:https://docs.python.org/zh-cn/3.7/library/…

This work adoptsCC agreement, reprint must indicate the author and the link to this article