Technology sharing | using perf and flame analysis software


Author: Agust í n
Translator: mengwick
Original text:…

In this blog post, we’ll explore how to use perf and flame chart together. They are used to generate a graph of the function that our selected software is calling. Here we use the percona branch version, but it can be extended to any software that can parse stack traces.

Before continuing, note that, like any analysis tool, do not run in a production environment unless you know what you are doing.

Install required packages

For simplicity, use the CentOS 7 version, but for Debian based distributions, they should be the same (the only difference in the steps is to use theapt-get install linux-tools-$(uname -r)Instead of the yum command).

Install perf

SHELL> sudo yum install -y perf

Get flame chart software package

SHELL> mkdir -p ~/src
SHELL> cd ~/src
SHELL> git clone

All installed! Let’s move on

Grab and collect samples

Flame diagram is a way to visualize data, so we need some samples that can be used as the basis. There are three ways to do this (note that we use the-pOption to grab only the data of the processes we are interested in, but if necessary, we can grab the data of all the running processes)

1. Only grab the set time (here is 10 seconds)

SHELL> sudo perf record -a -F 99 -g -p $(pgrep -x mysqld) -- sleep 10

2. Grab until we send out the interrupt signal (Ctrl + C)

SHELL> sudo perf record -a -F 99 -g -p $(pgrep -x mysqld)

3. Capture the life cycle of the whole process

#Note that if we interrupt this variant, we also kill the child process above
SHELL> sudo perf record -a -F 99 -g -- /sbin/mysqld \
--defaults-file=/etc/percona-server.conf.d/mysqld.cnf --user=mysql


SHELL> sudo perf record -a -F 99 -g -p $(pgrep -x mysqld) -- mysql -e "SELECT * FROM db.table"

In the first case of the third scenario, we were forced to grab the data of all the processes, because the implementation was impossible until the process ID number (PID) (by executing the command, we were actually starting the MySQL service). This type of command is very useful when you want to get data from the start of the process, otherwise it is impossible.

In the second scenario, we execute a query on the running MySQL service, so we can use the-pOption to grab data for this process. This is convenient, for example, if you want to grab data while the job is running.

Prepare samples

After the initialization grab is completed, we need to “read” the collected data. This is necessary because of the adoption ofperf recordThe generated format is binary. Therefore, we will implement:

SHELL> sudo perf script > perf.script

By default, it readsperf.dataperf recordAlso output to this file by default. It can be used separately-iOptions and-oOption to override the write.

Now we can read the generated text file because it is already in a readable form. However, when you do that, you will soon realize why we need to aggregate this data into a more understandable form.

Generate flame diagram

We can take the output of the first command as the output of the second command and do the following in a single command line. Because we did not add the GIT folder of the flame graph to the path variable, we used the full path.

SHELL> ~/src/FlameGraph/ perf.script | ~/src/FlameGraph/ > flamegraph.svg

Now we open the. SVG file in any browser and start to analyze the information rich graphics.

How does it look

As an example, I’ll post a screenshot of the full command, output, and flame generation using the second method of grabbing data as follows. We executeINSERT INTO...SELECTStatement, we can analyze the execution process.

SHELL> time sudo perf record -a -F 99 -g \
-p $(pgrep -x mysqld) \
-- mysql test -e "INSERT INTO joinit SELECT NULL, uuid(), time(now()),  (FLOOR( 1 + RAND( ) *60 )) FROM joinit;"
PID/TID switch overriding SYSTEM
[ perf record: Woken up 7 times to write data ]
[ perf record: Captured and wrote 1.909 MB (8214 samples) ]

real 1m24.366s
user 0m0.133s
sys 0m0.378s
SHELL> sudo perf script | \ 
~/src/FlameGraph/ perf.script | \
~/src/FlameGraph/ > mysql_select_into_flamegraph.svg

The keen reader will notice that we go one step further here, merging steps 2 and 3 through a pipe (|) to avoidperf.scriptFile write and read data. In addition, there is time output. We can estimate the amount of data generated by the tool (about 2MB data is generated in 1 minute and 25 seconds); of course, this depends on many factors, so we should treat it carefully and test it in our own environment.

The generated flame diagram is as follows:

Technology sharing | using perf and flame analysis software

An obvious candidate for optimization iswrite_record: if we make this function faster, there is a great potential to reduce the overall execution time (the blue box in the lower left corner indicates that we can see that 60% of the samples are obtained in this code path). In the last chapter below, we’ll provide a blog that explains in detail how to interpret the flame diagram, but for now, you only need to know that moving the mouse over each function name will dynamically change the information displayed in the lower left corner. You can also better understand it through the following guidelines:

Technology sharing | using perf and flame analysis software


For support engineers, in many cases, we use this tool to gain insight into what MySQL is doing and when. In this way, we can better understand the operation behind the specific load and take corresponding measures. This software is used for optimization or troubleshooting, it is a very powerful tool in our toolbox! As we all know, humans are better at processing images than text, and in my opinion, the tool is excellent.

Related connections

Interpreting Flame Graphs (scroll down to the “Flame Graph Interpretation” section)….

Flame graphs 201, percona sharing session…

Brendan Gregg, flame chart author…