Summarize several very useful Python libraries


Own library

1、 Datetime

Datetime is the standard processing date and time in Python.

1. Get current date and time

>>> from datetime import datetime

>>> now =

>>> print(now)

2021-06-14 09:33:10.460192

>>> print(type(now))

<class 'datetime.datetime'>

2. Gets the specified date and time

>>> from datetime import datetime

>>> dt = datetime(2021,6,10,12,0)

>>> print(dt)

2021-06-10 12:00:00

3. Datetime to timestamp

In a computer, time is actually expressed in numbers. We call the time of 00:00:00 UTC + 00:00 time zone on January 1, 1970 as epoch time0(the time timestamp before 1970 was negative). The current time is the number of seconds relative to epoch time, which is called timestamp.

>>> from datetime import datetime

>>> now =

>>> now

datetime.datetime(2021, 6, 14, 9, 38, 34, 969006)

>>> now. timestamp() 	# Convert datetime to timestamp


4. Timestamp to datetime

>>> from datetime import datetime

>>> timestamp = 1623634714.969006

>>> print(datetime.fromtimestamp(timestamp))

2021-06-14 09:38:34.969006

5. STR to datetime

>>> from datetime import datetime

>>> day = datetime.strptime('2021-6-10 12:12:12','%Y-%m-%d %H:%M:%S')

>>> print(day)

2021-06-10 12:12:12

6. Datetime to str

>>> from datetime import datetime

>>> now =

>>> print(now)

2021-06-14 09:49:02.281820

>>> print(type(now))

<class 'datetime.datetime'>

>>> str_day = now.strftime('%Y-%m-%d %H:%M:%S')

>>> print(str_day)

2021-06-14 09:49:02

>>> print(type(str_day))

<class 'str'>

2、 Collections

Collections is a collection module built in Python. It provides many useful collection classes, among which the statistical function is very practical.


CounterIs a simple counter, for example, counting the number of characters

>>> from collections import Counter

>>> c = Counter()

>>> str = 'jdkjefwnewnfjqbefbqbefqbferbb28934`83278784727'

>>> c.update(str)

>>> c

Counter({'b': 6, 'e': 5, 'f': 5, '8': 4, '7': 4, 'j': 3, 'q': 3, '2': 3, 'w': 2, 'n': 2, '3': 2, '4': 2, 'd': 1, 'k': 1, 'r': 1, '9': 1, '`': 1})

3、 Base64

Base64 is a method of representing arbitrary binary data with 64 characters.

Open with NotepadexejpgpdfWhen using these files, we will see a lot of garbled code, because binary files contain many characters that cannot be displayed and printed. Therefore, if text processing software such as Notepad can process binary data, we need a binary to string conversion method. Base64 is one of the most common binary encoding methods.

>>> import base64
>>> base64.b64encode(b'binary\x00string')
>>> base64.b64decode(b'YmluYXJ5AHN0cmluZw==')

4、 Hashlib

Python’s hashlib provides common summarization algorithms, such as MD5, SHA1 and so on.

What is a digest algorithm? The algorithm is also called hash algorithm and hash algorithm. It converts any length of data into a fixed length data string (usually represented by hexadecimal string) through a function.

Taking the common summary algorithm MD5 as an example, we calculate the MD5 value of a string:

>>> import hashlib

>>> md5 = hashlib.md5()

>>> md5. Update ("programmer Tang Ding". Encode ('utf-8 '))

>>> print(md5.hexdigest())


Where can the summary algorithm be applied? For example:

Any website that allows users to log in will store the user name and password of the user. How to store user name and password? The method is to save to the database table. If the user password is saved in clear text, if the database is leaked, the passwords of all users will fall into the hands of hackers. In addition, the website operation and maintenance personnel can access the database, that is, they can obtain the passwords of all users. The correct way to save the password is not to store the user’s plaintext password, but to store the summary of the account order, such as MD5. When the user logs in, first calculate the MD5 of the plaintext password entered by the user, and then compare it with the MD5 stored in the database. If it is consistent, it indicates that the password is entered correctly. If it is inconsistent, the password must be wrong.

Third party Library

1、 Requests

Requests is a python third-party library, which is particularly convenient for processing URL resources. In the previous “Introduction to reptile” article, we have a preliminary understanding of it.

1. Install requests

If anaconda is installed, requests is already available. Otherwise, you need to install through pip on the command line:

$ pip install requests

If you encounter permission denied installation failure, please add sudo and try again.

2. To access Douban homepage through get, you only need a few lines of code:

>>> import requests
>>> r = requests. get(' '# watercress Homepage
>>> r.status_code
>>> r.text
'<! DOCTYPE HTML > \ n < HTML > \ n < head > \ n < meta name = "description" content = "provide recommendations, comments and... '

3. For URLs with parameters, pass in a dict asparamsParameters:

>>> r = requests.get('', params={'q': 'python', 'cat': '1001'})
>>>R.url # actual requested URL

4. Requests automatically detects the code and can be usedencodingProperty view:

>>> r.encoding

5. Whether the response is text or binary content, we can usecontentAttribute acquisitionbytesObject:

>>> r.content
b'<!DOCTYPE html>\n<html>\n<head>\n<meta http-equiv="Content-Type" content="text/html; charset=utf-8">\n...'

6. The convenience of requests is that for specific types of responses, such as JSON, you can directly obtain:

>>> r = requests.get('*%20from%20weather.forecast%20where%20woeid%20%3D%202151330&format=json')
>>> r.json()
{'query': {'count': 1, 'created': '2017-11-17T07:14:12Z', ...

7. When the HTTP header needs to be passed in, we pass in a dict asheadersParameters:

>>> r = requests.get('', headers={'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 11_0 like Mac OS X) AppleWebKit'})
>>> r.text
'<! DOCTYPE HTML > \ n < HTML > \ n < head > \ n < meta charset = "UTF-8" > \ n < title > Douban (mobile version) < / Title >... '

8. To send a post request, you just need toget()Method becomespost(), and then pass indataParameter as data of post request:

>>> r ='', data={'form_email': '[email protected]', 'form_password': '123456'})

9. Requests is used by defaultapplication/x-www-form-urlencodedEncode post data. If you want to pass JSON data, you can directly pass in JSON parameters:

params = {'key': 'value'}
r = requests. Post (URL, JSON = params) # is automatically serialized into JSON

10. Similarly, uploading files requires a more complex encoding format, but requests simplifies it tofilesParameters:

>>> upload_files = {'file': open('report.xls', 'rb')}
>>> r =, files=upload_files)

When reading files, be sure to use'rb'That is, binary mode reading, which is obtained in this waybytesLength is the length of the file.

holdpost()Replace method withput()delete()And so on, you can request resources in the form of put or delete.

11. In addition to easily obtaining the response content, requests is also very simple to obtain other information of HTTP response. For example, get the response header:

>>> r.headers
{Content-Type': 'text/html; charset=utf-8', 'Transfer-Encoding': 'chunked', 'Content-Encoding': 'gzip', ...}
>>> r.headers['Content-Type']
'text/html; charset=utf-8'

12. We don’t have to parse cookies to get special requests easily:

>>> r.cookies['ts']

13. To pass in a cookie in a request, just prepare a dictcookiesParameters:

>>> cs = {'token': '12345', 'status': 'working'}
>>> r = requests.get(url, cookies=cs)

14. Finally, to specify the timeout, pass in the timeout parameter in seconds:

>>> r = requests. Get url = 2.5 # seconds after timeout

2、 Chardet

String encoding has always been a headache, especially when dealing with some non-standard third-party Web pages. Although Python provides Unicode representationstrandbytesTwo data types, and can beencode()anddecode()Method conversion, however, without knowing the encodingbytesdodecode()It’s hard to do.

For unknown encodedbytes, convert it intostr, you need to “guess” the code first. The way of guessing is to collect various coded characteristic characters first, and judge according to the characteristic characters, you can have a great probability of “guessing right”.

Of course, we certainly can’t write the function of detection coding from scratch, which is time-consuming and laborious. The third-party library, chardet, comes in handy. Use it to detect coding, simple and easy to use.

1. Install chardet

If anaconda is installed, chardet is already available. Otherwise, you need to install through pip on the command line:

$ pip install chardet

If you encounter permission denied installation failure, please add sudo and try again.

2. When we get onebytesIt can be detected and coded. To detect the code with chardet, only one line of code is required:

>>> chardet.detect(b'Hello, world!')
{'encoding': 'ascii', 'confidence': 1.0, 'language': ''}

3、 Psutil

Using Python to write scripts to simplify daily operation and maintenance work is an important use of Python. Under Linux, there are many system commands that allow us to monitor the running state of the system at any time, such aspstopfreewait. To obtain this system information, python cansubprocessThe module calls and gets the result. But it is very troublesome to do so, especially to write a lot of parsing code.

Another good way to get system information in Python is to usepsutilThis third-party module can not only realize system monitoring through one or two lines of code, but also be used across platforms. It supports Linux / Unix / OSX / windows and so on. It is an indispensable and necessary module for system administrators and operation and maintenance partners.

1. Install psutil

If anaconda is installed, psutil is already available. Otherwise, you need to install through pip on the command line:

$ pip install psutil

If you encounter permission denied installation failure, please add sudo and try again.

2. Get CPU Information

Let’s get the CPU information first:

>>> import psutil
>>> psutil. cpu_ Count() # CPU logical quantity
>>> psutil. cpu_ Count (logical = false) # CPU physical core
#2 indicates dual core hyper threading, and 4 indicates 4-core non hyper threading

3. Statistics of CPU user / system / idle time:

>>> psutil.cpu_times()
scputimes(user=10963.31, nice=0.0, system=5138.67, idle=356102.45)

4. Get memory information

Use psutil to obtain physical memory and exchange memory information, respectively:

>>> psutil.virtual_memory()
svmem(total=8589934592, available=2866520064, percent=66.6, used=7201386496, free=216178688, active=3342192640, inactive=2650341376, wired=1208852480)
>>> psutil.swap_memory()
sswap(total=1073741824, used=150732800, free=923009024, percent=14.0, sin=10705981440, sout=40353792)

The returned is an integer in bytes. You can see that the total memory size is 8589934592 = 8 GB. 7201386496 = 6.7 GB has been used, accounting for 66.6%.

The swap size is 1073741824 = 1 GB.

5. Get disk information

You can obtain disk partition, disk utilization and disk IO information through psutil:

>>> psutil. disk_ Partitions () # disk partition information
[sdiskpart(device='/dev/disk1', mountpoint='/', fstype='hfs', opts='rw,local,rootfs,dovolfs,journaled,multilabel')]
>>> psutil. disk_ Usage ('/') # disk usage
sdiskusage(total=998982549504, used=390880133120, free=607840272384, percent=39.1)
>>> psutil. disk_ io_ Counters() # disk IO
sdiskio(read_count=988513, write_count=274457, read_bytes=14856830464, write_bytes=17509420032, read_time=2228966, write_time=1618405)

Well, let’s introduce so many first. Tang Ding will introduce more practical Python libraries one by one later

This is the end of this article about summarizing several very practical Python libraries. For more information about Python libraries, please search the previous articles of developeppaer or continue to browse the relevant articles below. I hope you will support developeppaer in the future!