Python: performance analysis with pyinstrument

Time:2022-6-20

Guidance

In computing intensive computing or some web applications, we often need to do performance analysis on the code. In Python, the most primitive method is to usetimeIn packagetimeFunction in seconds:

from time import sleep, time

def func1():
    sleep(0.001)

def func2():
    sleep(2)
    
begin1 = time()
func1()
end1 = time()


begin2 = time()
func2()
end2 = time()
print("func1 consume: %f, func2 consume:%f, func3 consume: %f"\
    % (end1-begin1, end2-begin2, end2-begin1))

The console outputs are as follows:

func1 consume: 0.001271, func2 consume:2.000421, func3 consume: 2.001692

Basic usage of pyinstrument

But once there are more functions, this method is obviously too complicated. Similar to cprofile in C language, python also has special performance analysis toolspyinstrument(the library is not built-in and needs to be installed by conda/pip). We can use it in complex projects to replace the simpletime.time()

First, let’s take a look at the basic usage. Its framework is as follows:

from pyinstrument import Profiler
from time import sleep

def func1():
    sleep(0.1)

def func2():
    sleep(2)

profiler = Profiler()
profiler.start()

#Here is the code you want to analyze. Here we analyze func1 and func2 functions
func1()
func2()

profiler.stop()

profiler.print()

It can be seen that the tool also successfully analyzed the running time of functions, and pointed out that the func2 function running for 2S is a performance bottleneck:

NLP多任务学习

If we further reduce the running time of the function:

def func3():
    sleep(0.0001)

profiler = Profiler()
profiler.start()

func3()

profiler.stop()

profiler.print()

No samples were recorded is displayed as follows:
NLP多任务学习

This is because the running time of your code is less than 1ms. If you still want to analyze this code, you can choose to adjust the interval value to be smaller than the default 0.001 (1ms). For example:

profiler = Profiler(interval=0.0001)

At this point, you will find that func3 can also be detected:

NLP多任务学习

In addition, if you want to view the analysis results in the browser, you can use theprofiler.open_in_browser()replaceprofiler.print()Console printout for:

NLP多任务学习

You can also useprofiler.output_html()Output the profile as HTML.

Analyzing Web response performance in flask

We can also perform performance analysis on flask applications. The specific usage is as follows:

from flask import Flask, g, make_response, request
app = Flask(__name__)

@app.before_request
def before_request():
    if "profile" in request.args:
        g.profiler = Profiler()
        g.profiler.start()


@app.after_request
def after_request(response):
    if not hasattr(g, "profiler"):
        return response
    g.profiler.stop()
    output_html = g.profiler.output_html()
    return make_response(output_html)

In this way, the program will detect the?profileParameter, if detected, the analysis will start. After running the request of the profiler, it will generate an HTML output instead of the real response and return.

reference resources