Requests, aiohttp, httpx comparison

Time:2021-8-22

Among Python’s many HTTP clients, the most famous isrequestsaiohttpandhttpx。 Without the help of other third-party libraries,requestsOnly synchronization requests can be sent;aiohttpOnly asynchronous requests can be sent;httpxIt can send both synchronous and asynchronous requests.

The so-called synchronization request means that in single process and single thread code, after a request is initiated, the next request cannot be initiated until the return result is received. The so-called asynchronous request means that in single process and single thread code, after initiating a request, you can continue to send more requests while waiting for the website to return results.

Today, let’s take a shallow evaluation to compare the performance of the three libraries from the perspective of sending get requests multiple times.

Of course, the test results are related to the network speed, but you can still see the problem when you test the same network at the same time.

Requests send requests

import requests

url = 'https://www.baidu.com/'
headers = headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}

def main():
    res = requests.get(url,headers=headers)
    print(res.status_code)

if __name__ == '__main__':
    main()

Httpx send request

To send a synchronization request using httpx:

import httpx

url = 'https://www.baidu.com/'
headers = headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}

def main():
    res = httpx.get(url,headers=headers)
    print(res.status_code)

if __name__ == '__main__':
      main()

The synchronization mode of httpx coincides with the requests code by 99%. You only need to change requests to httpx to run normally.

Send asynchronous requests using httpx:

import httpx
import asyncio

url = 'https://www.baidu.com/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}


async def main():
    async with httpx.AsyncClient() as client:
        resp = await client.get(url, headers=headers)
        print(resp.status_code)

if __name__ == '__main__':
    asyncio.run(main())

Aiohttp send request

import asyncio

import aiohttp

url = 'https://www.baidu.com/'
headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36'}

async def main():
    async with aiohttp.ClientSession() as client:
        async with client.get(url, headers=headers) as resp:
            print(resp.text)
            print(resp.status)

if __name__ == '__main__':
    asyncio.run(main())

The code of AIO HTTP coincides with that of httpx asynchronous mode by 90%, except that asyncclient is replaced by clientsession

Using requests.post creates a new connection each time, which is slow. If a session is initialized first, requests will remain connected, which greatly improves the request speed. Therefore, in this evaluation, we test the two cases respectively

requests

Requests do not remain connected

import time

Send 100 requests, time consuming: 10.29585409164287

Requests remain connected

import time

Sending 100 requests takes 4.679062128067017, which is obviously close to 6S

httpx

Httpx synchronization mode

import time

Send 100 requests, time consuming: 16.60569405555725

Httpx asynchronous mode: create httpx. Asyncclient() only once

import httpx

Send 100 requests, time consuming: 4.359861135482788

Httpx asynchronous mode: create httpx. Asyncclient() each time

import httpx

Send 100 requests, time consuming: 6.378381013870239

aiohttp

Aiohttp: create aiohttp. Clientsession() only once

import time

Send 100 requests, time consuming: 2.235464334487915

Aiohttp: aiohttp. Clientsession() is created each time

import time

Send 100 requests, time consuming: 2.6662471294403076

Request 100 speed rankings

Aiohttp (create client only once) > aiohttp (create client every time) > httpx asynchronous (create client only once) > requests.session > httpx asynchronous (create client every time) > requests > HTTP asynchronous

summary

  • If you only send a few requests. Then the code is the simplest to use the synchronization mode of requests or httpx.
  • Whether requests creates a session to keep the connection. The speed difference is relatively large. In the case of no anti climbing, only speed is pursued. It is recommended to use requests. Session()
  • If you need to send many requests, but some places need to send synchronous requests and some places need to send asynchronous requests, using httpx is the easiest.
  • If you want to send many requests as soon as possible, then AIO HTTP is the fastest.

This work adoptsCC agreement, reprint must indicate the author and the link to this article