Linux virtual address space

Time:2020-2-13

background

Today, when learning Linux memory related knowledge, I saw the content related to virtual address, so I recorded the knowledge related to virtual address space, heap and stack. In the article I read, the kernel version is relatively old, so I just recorded it, and the knowledge about the new version of kernel will be sorted out later

Linux virtual address space

In IA-32, the virtual address space is usually a 4GB address block, Generally, it is divided into user space and kernel space by 3:1. 3:1 is not the only option. Because the boundary definition is defined as a constant in the source code, there is little work to choose another partition method. In some cases, it is better to divide by symmetry (1:1). It can be defined by \
However, this does not mean that the kernel has only so much physical memory available. It only means that it can control this part of address space and map it to physical memory as needed

Virtual memory segment mapping

The virtual address is mapped to the physical memory through the table and maintained by the operating system
Kernel space has higher privilege level in page table. When user state program accesses these pages, it will cause page fault
Kernel space is persistent and can be mapped to the same physical memory in all processes
Kernel code and data are always addressable. On the contrary, user mode address space mapping changes with process switching

Linux virtual address space

User process segment

The user process is segmented as follows

Name Storage content
Stack Local variable, function parameter, return address, etc
heap Dynamically allocated memory [brk()]
BSS segment Global and static local variables that are not initialized or have an initial value of 0
Data segment Global and static local variables initialized with an initial value other than 0
Code segment Executable code, string literal, read-only variable

The heap is managed by the programmer himself, displaying application and release;
BSS segment, data segment and code segment are the segments of executable program at compile time
Heap and stack are runtime segments

Detailed explanation of segments

Kernel space

The kernel always resides in memory and is part of the operating system. Applications are not allowed to read or write this area or directly call functions defined by kernel code

Stack

Stack, also known as stack, is automatically allocated and released by compiler and behaves like stack in data structure (first in first out). The stack has three main uses:

1. Provide storage space for non static local variables (called “automatic variables” in C language) declared inside functions.
2. Record the maintenance information related to function call process, which is called stack frame or procedure activation record. It includes the function return address, the function parameters that are not suitable for loading registers and the saving of some register values. The stack is not required except for recursive calls. Because local variables, parameters, and return address space are known at compile time and allocated to the BSS segment.
3. Temporary storage area, which is used to temporarily store the long arithmetic expression part calculation results or the stack memory allocated by the alloca() function.

Reusing stack space continuously helps keep active stack memory in the CPU cache, thus speeding up access. Each thread in the process has its own stack. When pushing data into the stack, if it exceeds its capacity, the memory area corresponding to the stack will be exhausted, thus triggering a page error. At this time, if the stack size is lower than the maximum stack size, rlimit? Stack (usually 8m), the stack will grow dynamically and the program will continue to run. When the mapped stack area is expanded to the required size, it is no longer shrunk.

The ulimit-s command in Linux can view and set the maximum stack value. When the stack used by the program exceeds this value, a stack overflow occurs, and the program receives a segmentation fault. Note that increasing the stack size may increase memory overhead and start-up time.

The stack can grow both down (to low memory address) and up, depending on the implementation. The stack described in this article grows down.

The size of the stack is dynamically adjusted by the kernel at runtime.

Memory map segment (MMAP)

The kernel directly maps the contents of hard disk files to memory, which can be requested by any application through mmap() system call of Linux or createfilemapping() / mapviewofile() of windows. Memory mapping is a convenient and efficient way of file I / O, so it is used to load dynamic shared libraries.
The user can also create an anonymous memory map, which has no corresponding file and can be used to store program data. In Linux, if a large block of memory is requested through malloc(), the C runtime will create an anonymous memory map instead of using heap memory. ” Large “means larger than the threshold value mmap_threshold, which is 128KB by default and can be adjusted through mallopt().

This area is used to map the dynamic link library used by the executable. In Linux version 2.4, if the executable depends on the shared library, the system will allocate the corresponding space for these dynamic libraries at the address starting from 0x40000000, and load them into the space when the program is loaded. In the Linux 2.6 kernel, the starting address of the shared library is moved up to a location closer to the stack area.

From the layout of the process address space, we can see that in the case of shared libraries, there are two places left for the heap: one is from the. BSS segment to 0x40000000, less than 1GB; the other is from the shared library to the stack, less than 2GB. The size of these two spaces depends on the size and number of stacks and shared libraries. In this way, is the maximum heap space that an application can request only 2GB? In fact, this is related to the Linux kernel version. In the classic layout of the process address space given above, the load address of the shared library is 0x40000000, which is actually before Linux kernel 2.6. In version 2.6, the load address of the shared library has been moved to a location close to the stack, i.e. near 0xbfxxxxx. Therefore, the heap range at this time will not be divided into two “fragments” by the shared library, so kernel 2 In the 32-bit Linux system of 6, the theoretical maximum memory requested by malloc is about 2.9GB.

Heap (heap)

Heap is used to store the memory segments allocated dynamically when the process is running, which can be expanded or reduced dynamically. The content in the heap is anonymous and cannot be accessed directly by name, but only indirectly by pointer. When a process calls functions such as malloc (c) / new (c + +) to allocate memory, the newly allocated memory is dynamically added to the heap (expanded); when a function such as free (c) / delete (c + +) is called to release memory, the released memory is removed (reduced) from the heap.

Allocated heap memory is byte aligned space to accommodate atomic operations. The heap manager manages the memory of each request through a linked list. Because heap requests and releases are unordered, memory fragmentation will eventually occur. Heap memory is generally allocated and released by the application, and the reclaimed memory can be reused. If the programmer does not release it, the operating system may recycle it automatically at the end of the program.

The end of the heap is identified by the break pointer. When the heap manager needs more memory, it can move the break pointer to expand the heap by calling brk() and sbrk(), which are usually called automatically by the system.

There are two common problems when using heap: 1) freeing or overwriting memory that is still in use (“memory corruption”); 2) not freeing memory that is no longer in use (“memory leak”). When the number of releases is less than the number of requests, a memory leak may have been caused. The leaked memory is often larger than the data structure forgotten to release, because the allocated memory is usually rounded to the next power of 2 greater than the number of applications (for example, for 212B, it will be rounded to 256b).

Note that the heap is different from the “heap” in the data structure and behaves like a linked list.

Difference between heap and stack

① Management mode: the stack is automatically managed by the compiler; the heap is controlled by the programmer, easy to use, but prone to memory leakage.

② Growth direction: stack expansion to low address (i.e. “down growth”) is a continuous memory area; heap expansion to high address (i.e. “up growth”) is a discontinuous memory area. This is because the system uses linked list to store the free memory address, which is naturally discontinuous, and the linked list traverses from low address to high address.

③ Space size: the address at the top of the stack and the maximum capacity of the stack are predetermined by the system (usually 2m or 10m by default); the size of the heap is limited by the effective virtual memory in the computer system, and the heap memory in the 32-bit Linux system can reach 2.9g space.

④ Storage content: when the stack calls a function, it first presses the address of the next instruction (the next executable statement of the function call statement) in the main function, then the function argument, and then the local variable of the called function. At the end of this call, the local variables first come out of the stack, then the parameters, and finally the top of the stack pointer points to the address of the first stored instruction, from which the program continues to run the next executable statement. The heap usually uses a byte in the header to store its size. The heap is used to store data that has no relation with function call in its lifetime. The specific content is arranged by the programmer.

⑤ Allocation method: the stack can be allocated statically or dynamically. Static allocation is done by the compiler, such as the allocation of local variables. In dynamic allocation, the alloca function applies for space on the stack and releases it automatically when it is used up. The heap can only be allocated dynamically and released manually.

⑥ Allocation efficiency: the stack is supported by the bottom layer of the computer: special register storage stack address is allocated, and stack pressing is executed by special instructions, so the efficiency is high. Heap is provided by function library, which has complex mechanism and much lower efficiency than stack. In Windows system, virtualalloc can allocate a piece of memory directly in the process address space, which is fast and flexible.

⑦ System response after allocation: as long as the remaining space of the stack is larger than the applied space, the system will provide memory for the program, otherwise an exception will be reported to indicate stack overflow.

The operating system maintains a list of free memory addresses for the heap. When the system receives the program’s memory allocation request, it will traverse the list to find the first heap node whose space is larger than the requested space, then delete the node from the free node list and allocate the node space to the program. If there is not enough space (probably due to too much memory fragmentation), it is possible to call system functions to increase the memory space of program data segments, so as to have the opportunity to allocate enough memory, and then return. Most systems will record the allocated memory size at the first address of the memory space for subsequent release functions (such as free / delete) to correctly release the memory space.

In addition, because the size of the heap node found is not exactly equal to the size of the application, the system will automatically put the redundant part back into the free list.

⑧ Fragmentation problem: there will be no fragmentation problem in the stack, because the stack is the first in and last out queue. Before the memory block pops up the stack, the last in stack content on it has popped up. Frequent requests for release operations will result in the discontinuity of heap memory space, resulting in a large number of fragments, reducing the efficiency of the program.

It can be seen that the heap is easy to cause memory fragmentation; because there is no special system support, the efficiency is very low; because it may cause user state and kernel state switching, the cost of memory application is more expensive. So the stack is the most widely used in the program, and the function call is also completed by the stack. The parameters, return address, stack base pointer and local variables in the call process are stored by the stack. Therefore, it is recommended to use the stack as much as possible and only use the heap when allocating a large amount or a large amount of memory space.

When using stack and heap, we should avoid crossing the boundary, otherwise the program may crash or destroy the structure of stack and heap, resulting in unexpected consequences.

BSS segment

The BSS (block started by symbol) section usually stores the following symbols in the program:

  • Uninitialized global and static local variables
  • Global and static local variables with an initial value of 0 (compiler implementation dependent)
  • Undefined symbol with an initial value other than 0 (the initial value is the size of the common block)

    In C, statically allocated variables that are not explicitly initialized are initialized to 0 (arithmetic type) or null pointer (pointer type). Because BSS will be cleared by the operating system when the program is loaded, global variables with no initial value or initial value of 0 are all in BSS. BSS segment only reserves space for uninitialized static allocation variables and does not occupy space in the target file, which can reduce the volume of the target file. However, when the program is running, it needs to allocate memory space for variables, so the target file must record the sum of all uninitialized static allocated variable sizes (write the machine code through the start ﹣ BSS and end ﹣ BSS addresses). Initializes the memory allocated for the BSS segment to 0 when the loader loads the program. In embedded software, before entering the main() function, the BSS segment is mapped by the C runtime system to the memory initialized to zero (high efficiency).

    Note that although both are placed in the BSS segment, global variables with an initial value of 0 are strong symbols, while uninitialized global variables are weak symbols. If a strong symbol with the same name has been defined elsewhere (the initial value may not be 0), the weak symbol will not cause redefinition error when it is linked with it, but the initial value at runtime may not be the expected value (it will be overwritten by the strong symbol). Therefore, when defining a global variable, if it is only used in this file, try to use the static keyword to decorate it; otherwise, you need to assign an initial value (even 0 value) to the global variable definition to ensure that the variable is a strong symbol, so as to find the variable name conflict when linking, rather than being covered by an unknown value.

    Some compilers store uninitialized global variables in the common section and put them in the BSS section when linking. The – fno common option can be used during compilation to prevent uninitialized global variables from being put into the common section.

    In addition, because the target file does not contain BSS segment, the address space content of BSS segment is unknown after the program is burned into memory (flash). In the process of u-boot startup, after you move (copy) the u-boot stage2 code (usually located in the file lib XXXX / board. C) to SDRAM space, you must manually add the code of BSS segment, instead of relying on the value of 0 when the variables in stage2 code are defined.

Data segment

Data segments are usually used to store global and static local variables initialized in a program with an initial value of not 0. Data segment belongs to static memory allocation (static storage area), which is readable and writable.

The data segment is stored in the target file (usually solidified in the image file in the embedded system), and its content is initialized by the program. For example, for the global variable int gvar = 10, the data of 10 must be saved in the data segment of the target file, and then copied to the corresponding memory when the program is loaded.

The difference between data segment and BSS segment is as follows: 
 1) BSS segment does not occupy physical file size, but occupies memory space; data segment occupies physical file, but also occupies memory space.
 For large arrays such as int ar0 [10000] = {1, 2, 3,...} and int AR1 [10000], AR1 is placed in the BSS segment, only 10000 * 4 bytes of records need to be initialized to 0, instead of recording each data 1, 2, 3... Like ar0. At this time, BSS saves considerable disk space for the target file.
 2) When the program reads the data of the data segment, the system will start the page missing fault and allocate the corresponding physical memory; when the program reads the data of the BSS segment, the kernel will turn it to a zero page without page missing fault or allocating the corresponding physical memory for it.
 The whole segment of the runtime data segment and the BSS segment is commonly referred to as the data segment. In some data, "data segment" refers to data segment + BSS segment + heap.

Code snippet (text)

Code segment is also called body segment or text segment, which is usually used to store program execution code (i.e. machine instructions executed by CPU). Generally, C language execution statements are compiled into machine code and saved in code snippets. Code segments are usually shareable, so frequently executed programs only need to have a copy in memory. Code snippets are usually read-only to prevent other programs from accidentally modifying their instructions (a write to that snippet will result in a snippet error). Some architectures also allow code snippets to be writable, that is, to modify programs.

The code segment instructions are executed in sequence according to the program design process. For the sequential instructions, they will only be executed once (each process); if there are repetitions, the jump instruction is needed; if there is recursion, the stack is needed.

The code segment instructions include the opcode and the operand (or object address reference). If the operation object is an immediate number (specific value), it will be directly included in the code; if it is local data, it will allocate space in the stack area, and then reference the data address; if it is in BSS segment and data segment, it will also reference the data address.

Code snippets are most susceptible to optimization measures.

Reference resources:
https://www.cnblogs.com/clove…

Recommended Today

Cookies and session

Session overview Just like a phone call, multiple conversations are requests and responses. Once disconnected, they end The problem to be solved in the session, the problem of keeping the data of multiple requests Client technology cookie 1. Header of HTTP protocol Request header: Cookie client delivers information to server Response header: set cookie server […]