The Method of Deadlock Analysis on Linux Platform

Time:2019-9-12

Deadlocks: refers to the phenomenon that two or more processes (threads) are waiting for each other because of competing for resources in the execution process. Without external force, they will not be able to advance. At this time, it is said that the system is in a deadlock state or that the system has a deadlock. These processes (threads) that are always waiting for each other are called deadlock processes (threads). Because resource occupancy is mutually exclusive, when a process requests resources, the process (thread) can never allocate the necessary resources without external assistance and can not continue to run, which results in a special phenomenon of deadlock.

A situation where two or more threads in an execution program are permanently blocked (waiting), and each thread is waiting for resources occupied and blocked by other threads. For example, if thread 1 locks record A and waits for record B, and thread 2 locks record B and waits for record A, then deadlock occurs in both threads. In computer systems, if the system’s resource allocation strategy is inappropriate, it is more common that programmers write programs with errors, which will lead to process deadlock due to inappropriate competition for resources.

Four Necessary Conditions for Deadlock Generation

(1) Mutual exclusion condition: a resource can only be used by one process (thread) at a time.

(2) Request and retention conditions: When a process (thread) is blocked by requesting resources, it keeps the acquired resources intact.

(3) Non-deprivation conditions: the process (thread) acquired resources, before the end of use, can not be forcibly deprived.

(4) Loop waiting conditions: a head-to-tail relationship between multiple processes (threads) is formed.

Figure 1. Deadlock schematic diagram of cross-locking:

Note: After func 2 and func4 are executed, sub-thread 1 acquires lock A and is trying to acquire lock B, but sub-thread 2 acquires lock B at this time and is trying to acquire lock A, so sub-thread 1 and sub-thread 2 will not be able to acquire lock A and lock B because they are occupied by each other and will never be released. Deadlock was born.

Using pstack and GDB tools to analyze deadlock programs

A Brief Introduction of pstack on Linux Platform

Pstack is the next useful tool for Linux (such as Red Hat Linux system, Ubuntu Linux system, etc.). Its function is to print out the stack information of this process. The call relationship stack of all threads can be output.

A Brief Introduction of GDB on Linux Platform

GDB is a powerful UNIX debugging tool released by GNU Open Source Organization. Linux system contains GNU debugger gdb, which is a debugger for debugging C and C++ programs. Program developers can observe the internal structure and memory usage of programs while they are running.

Some of the main functions provided by GDB are as follows:

1. Running the program, setting parameters and environment that can affect the operation of the program;

2. The control program stops running under specified conditions.

When the program stops, the state of the program can be checked.

When the program crash, you can check the core file.

Fifth, we can modify the errors of the program and re-run the program.

6. The value of variables in program can be monitored dynamically.

7. The code can be executed step by step to observe the running state of the program.

The object of debugging GDB program is executable file or process, not source code file of program. However, not all executable files can be debugged with gdb. If the generated executable file can be used for debugging, it is necessary to add – g parameter when compiling the g++ (gcc) instruction to specify that the program contains debugging information at compiling time. Debugging information includes the type of each variable in the program, the address mapping in the executable file, and the line number of the source code. GDB uses this information to associate source code with machine code. There are many basic commands of gdb, and no detailed introduction is given. If you need further understanding, please refer to the GDB manual.

Listing 1. Test program

#include  
 #include  
 #include  

 pthread_mutex_t mutex1 = PTHREAD_MUTEX_INITIALIZER; 
 pthread_mutex_t mutex2 = PTHREAD_MUTEX_INITIALIZER; 
 pthread_mutex_t mutex3 = PTHREAD_MUTEX_INITIALIZER; 
 pthread_mutex_t mutex4 = PTHREAD_MUTEX_INITIALIZER; 

 static int sequence1 = 0; 
 static int sequence2 = 0; 

 int func1() 
 { 
    pthread_mutex_lock(&mutex1); 
    ++sequence1; 
    sleep(1); 
    pthread_mutex_lock(&mutex2); 
    ++sequence2; 
    pthread_mutex_unlock(&mutex2); 
    pthread_mutex_unlock(&mutex1); 

    return sequence1; 
 } 

 int func2() 
 { 
    pthread_mutex_lock(&mutex2); 
    ++sequence2; 
    sleep(1); 
    pthread_mutex_lock(&mutex1); 
    ++sequence1; 
    pthread_mutex_unlock(&mutex1); 
    pthread_mutex_unlock(&mutex2); 

    return sequence2; 
 } 

 void* thread1(void* arg) 
 { 
    while (1) 
    { 
        int iRetValue = func1(); 

        if (iRetValue == 100000) 
        { 
            pthread_exit(NULL); 
        } 
    } 
 } 

 void* thread2(void* arg) 
 { 
    while (1) 
    { 
        int iRetValue = func2(); 

        if (iRetValue == 100000) 
        { 
            pthread_exit(NULL); 
        } 
    } 
 } 

 void* thread3(void* arg) 
 { 
    while (1) 
    { 
        sleep(1); 
        char szBuf[128]; 
        memset(szBuf, 0, sizeof(szBuf)); 
        strcpy(szBuf, "thread3"); 
    } 
 } 

 void* thread4(void* arg) 
 { 
    while (1) 
    { 
        sleep(1); 
        char szBuf[128]; 
        memset(szBuf, 0, sizeof(szBuf)); 
        strcpy(szBuf, "thread3"); 
    } 
 } 

 int main() 
 { 
    pthread_t tid[4]; 
    if (pthread_create(&tid[0], NULL, &thread1, NULL) != 0) 
    { 
        _exit(1); 
    } 
    if (pthread_create(&tid[1], NULL, &thread2, NULL) != 0) 
    { 
        _exit(1); 
    } 
    if (pthread_create(&tid[2], NULL, &thread3, NULL) != 0) 
    { 
        _exit(1); 
    } 
    if (pthread_create(&tid[3], NULL, &thread4, NULL) != 0) 
    { 
        _exit(1); 
    } 

    sleep(5); 
    //pthread_cancel(tid[0]); 

    pthread_join(tid[0], NULL); 
    pthread_join(tid[1], NULL); 
    pthread_join(tid[2], NULL); 
    pthread_join(tid[3], NULL); 

    pthread_mutex_destroy(&mutex1); 
    pthread_mutex_destroy(&mutex2); 
    pthread_mutex_destroy(&mutex3); 
    pthread_mutex_destroy(&mutex4); 

    return 0; 
 } 

Listing 2. Compile the test program

 [[email protected]linuxbldsrv purify]$ g++ -g lock.cpp -o lock -lpthread 

Listing 3. Find the process number of the test program

 [[email protected] purify]$ ps -ef|grep lock 
 dyu       6721  5751  0 15:21 pts/3    00:00:00 ./lock 

Listing 4. Output of the first execution of pstack (pstack – process number) on deadlocked processes

 [[email protected] purify]$ pstack 6721 
 Thread 5 (Thread 0x41e37940 (LWP 6722)): 
 #0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 
 #1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 
 #2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0 
 #3  0x0000000000400a9b in func1() () 
 #4  0x0000000000400ad7 in thread1(void*) () 
 #5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 4 (Thread 0x42838940 (LWP 6723)): 
 #0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 
 #1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 
 #2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0 
 #3  0x0000000000400a17 in func2() () 
 #4  0x0000000000400a53 in thread2(void*) () 
 #5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 3 (Thread 0x43239940 (LWP 6724)): 
 #0  0x0000003d19c9a541 in nanosleep () from /lib64/libc.so.6 
 #1  0x0000003d19c9a364 in sleep () from /lib64/libc.so.6 
 #2  0x00000000004009bc in thread3(void*) () 
 #3  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #4  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 2 (Thread 0x43c3a940 (LWP 6725)): 
 #0  0x0000003d19c9a541 in nanosleep () from /lib64/libc.so.6 
 #1  0x0000003d19c9a364 in sleep () from /lib64/libc.so.6 
 #2  0x0000000000400976 in thread4(void*) () 
 #3  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #4  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 1 (Thread 0x2b984ecabd90 (LWP 6721)): 
 #0  0x0000003d1a807b35 in pthread_join () from /lib64/libpthread.so.0 
 #1  0x0000000000400900 in main ()    

Listing 5. Output of the second execution of pstack (pstack – process number) for deadlock processes

 [[email protected] purify]$ pstack 6721 
 Thread 5 (Thread 0x40bd6940 (LWP 6722)): 
 #0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 
 #1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 
 #2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0 
 #3  0x0000000000400a87 in func1() () 
 #4  0x0000000000400ac3 in thread1(void*) () 
 #5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 4 (Thread 0x415d7940 (LWP 6723)): 
 #0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 
 #1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 
 #2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0 
 #3  0x0000000000400a03 in func2() () 
 #4  0x0000000000400a3f in thread2(void*) () 
 #5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 3 (Thread 0x41fd8940 (LWP 6724)): 
 #0  0x0000003d19c7aec2 in memset () from /lib64/libc.so.6 
 #1  0x00000000004009be in thread3(void*) () 
 #2  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #3  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 2 (Thread 0x429d9940 (LWP 6725)): 
 #0  0x0000003d19c7ae0d in memset () from /lib64/libc.so.6 
 #1  0x0000000000400982 in thread4(void*) () 
 #2  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #3  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 
 Thread 1 (Thread 0x2af906fd9d90 (LWP 6721)): 
 #0  0x0000003d1a807b35 in pthread_join () from /lib64/libpthread.so.0 
 #1  0x0000000000400900 in main () 

Viewing the function call relation stack of this process several times in succession for analysis: when the process is hanging, using pstack to view the function call stack of the process many times, deadlock threads will always be in the state of equal lock. Comparing the output results of the function call stack many times, we can determine which two threads (or several threads) have never been in the state of equal lock. Changing and always in a state of equal lock (there may be two threads that remain unchanged).

Output analysis:

According to the above output comparison, thread 1 and thread 2 change from the sleep function where the first pstack output is to the memset function where the second pstack output is. But threads 4 and 5 are always in the state of equal lock (pthread_mutex_lock), and there is no change in the output of pstack information for two consecutive times, so we can infer that threads 4 and 5 are deadlocked.

Gdb into thread output:

Listing 6. Then attach to the deadlock process through GDB

   (gdb) info thread 
  5 Thread 0x41e37940 (LWP 6722)  0x0000003d1a80d4c4 in __lll_lock_wait () 
  from /lib64/libpthread.so.0 
  4 Thread 0x42838940 (LWP 6723)  0x0000003d1a80d4c4 in __lll_lock_wait () 
  from /lib64/libpthread.so.0 
  3 Thread 0x43239940 (LWP 6724)  0x0000003d19c9a541 in nanosleep () 
 from /lib64/libc.so.6 
  2 Thread 0x43c3a940 (LWP 6725)  0x0000003d19c9a541 in nanosleep () 
 from /lib64/libc.so.6 
 * 1 Thread 0x2b984ecabd90 (LWP 6721)  0x0000003d1a807b35 in pthread_join () 
 from /lib64/libpthread.so.0 

Listing 7. Output switched to thread 5

 (gdb) thread 5 
 [Switching to thread 5 (Thread 0x41e37940 (LWP 6722))]#0  0x0000003d1a80d4c4 in 
 __lll_lock_wait () from /lib64/libpthread.so.0 
 (gdb) where 
 #0  0x0000003d1a80d4c4 in __lll_lock_wait () from /lib64/libpthread.so.0 
 #1  0x0000003d1a808e1a in _L_lock_1034 () from /lib64/libpthread.so.0 
 #2  0x0000003d1a808cdc in pthread_mutex_lock () from /lib64/libpthread.so.0 
 #3  0x0000000000400a9b in func1 () at lock.cpp:18 
 #4  0x0000000000400ad7 in thread1 (arg=0x0) at lock.cpp:43 
 #5  0x0000003d1a80673d in start_thread () from /lib64/libpthread.so.0 
 #6  0x0000003d19cd40cd in clone () from /lib64/libc.so.6 

Listing 8. Output of Thread 4 and Thread 5

 (gdb) f 3 
 #3  0x0000000000400a9b in func1 () at lock.cpp:18 
 18          pthread_mutex_lock(&mutex2); 
 (gdb) thread 4 
 [Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
 __lll_lock_wait () from /lib64/libpthread.so.0 
 (gdb) f 3 
 #3  0x0000000000400a17 in func2 () at lock.cpp:31 
 31          pthread_mutex_lock(&mutex1); 
 (gdb) p mutex1 
 $1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
 __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
  __size = "
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
2
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0B2
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
1", '
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0' , __align = 2} (gdb) p mutex3 $2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = '
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0' , __align = 0} (gdb) p mutex2 $3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, __kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, __size = "
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
2
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0C2
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
1", '
 (gdb) f 3 
#3  0x0000000000400a9b in func1 () at lock.cpp:18 
18          pthread_mutex_lock(&mutex2); 
(gdb) thread 4 
[Switching to thread 4 (Thread 0x42838940 (LWP 6723))]#0  0x0000003d1a80d4c4 in 
__lll_lock_wait () from /lib64/libpthread.so.0 
(gdb) f 3 
#3  0x0000000000400a17 in func2 () at lock.cpp:31 
31          pthread_mutex_lock(&mutex1); 
(gdb) p mutex1 
$1 = {__data = {__lock = 2, __count = 0, __owner = 6722, __nusers = 1, __kind = 0, 
__spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000B\032\000\000\001", '\000'
, __align = 2} 
(gdb) p mutex3 
$2 = {__data = {__lock = 0, __count = 0, __owner = 0, __nusers = 0, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = '\000' , __align = 0} 
(gdb) p mutex2 
$3 = {__data = {__lock = 2, __count = 0, __owner = 6723, __nusers = 1, 
__kind = 0, __spins = 0, __list = {__prev = 0x0, __next = 0x0}}, 
__size = "\002\000\000\000\000\000\000\000C\032\000\000\001", '\000'
, __align = 2} 
(gdb) 
0' , __align = 2} (gdb)

As can be seen above, thread 4 is trying to get lock mutex1, but lock mutex1 has been obtained by thread LWP 6722 (_owner = 6722), thread 5 is trying to get lock mutex2, but lock mutex2 has been obtained by LWP 6723 (_owner = 6723), as can be seen from the output of pstack, LWP 6722 and line Cheng 5 corresponds, while LWP 6723 corresponds to thread 4. So we can conclude that thread 4 and thread 5 have a deadlock phenomenon of cross-locking. Looking at the source code of threads, we find that threads 4 and 5 use both mutex1 and mutex2, and the application sequence is unreasonable.

summary

This paper briefly introduces a method of analyzing deadlock problem on Linux platform, which has certain effect on the analysis of some deadlock problems. I hope it will be helpful to you all. Understanding the causes of deadlocks, especially the four necessary conditions for deadlocks, can maximize the possibility of avoiding, preventing and removing deadlocks. Therefore, in the aspects of system design and process scheduling, attention should be paid to how to prevent these four necessary conditions from being established, how to determine the reasonable allocation algorithm of resources, and how to avoid the process occupying the system resources permanently. In addition, it is necessary to prevent the process from occupying resources while it is in a waiting state. In the process of system operation, it is necessary to dynamically check the resource requests that can be satisfied by each system issued by the process, and decide whether to allocate resources according to the results of the inspection. If the system may be deadlocked after allocation, it will not be allocated, otherwise it will be given. Distribution. Therefore, the allocation of resources should be planned reasonably, and the use of orderly resource allocation method and banker algorithm is an effective way to avoid deadlock.