Details of multi-path and large capacity hard disk mount under CentOS


I. application environment and requirementsBlade servers connect HP storage through fiber switches, forming a 2×2 link The storage capacity of the operating system for CentOS 6.4 64 bit mount is 2.5t

Based on this application environment, two problems need to be solved:

In order to ensure the stability and transmission performance of the link, multi-path technology can be used; the attached storage hard disk exceeds 2T, and the MBR partition format cannot be supported, so the GPT partition format is needed

Because CentOS 6.4 has its own HP storage driver, the attached storage hard disk will be automatically identified. Otherwise, you need to install the storage driver first.

What is multi-path

Ordinary computer hosts are all hard disks attached to a bus. Here is a one-to-one relationship. When it comes to a San environment consisting of optical fiber or an IPSAN environment consisting of iSCSI, because the host and storage are connected by optical fiber switches or multiple network cards and IP, then a many to many relationship is formed. In other words, there are multiple paths from host to storage. There are multiple paths for IO from host to storage. Each host can go through several different paths to the corresponding storage. If it is used at the same time, how is the I / O traffic allocated? One of the paths is broken. How to deal with it? From the perspective of operating system, each path is considered as a physical disk, but in fact, it is only a different path to the same physical disk, which brings confusion to users when using. Multipath software is to solve the above problems.

The main function of multipath is to cooperate with storage devices to achieve the following functions:

1. Failover and recovery

2. Load balancing of IO traffic

3. Disk virtualization

Because multipath software needs to be used together with storage, different manufacturers provide different versions based on different operating systems. And some manufacturers, software and hardware are not sold together. If you want to use multipath software, you may need to purchase license from the manufacturer. For example, EMC multi-path software based on Linux needs to purchase license separately. Fortunately, both the red hat and SUSE 2.6 kernels come with free multi-path software package, which can be used for free. At the same time, it is a relatively general package, which can support most of the storage manufacturers’ devices. Even some not well-known manufacturers can support and run well by slightly modifying the configuration file.

The intuitive feeling is that when fdisk-l command is executed in Linux system, hard disks like / dev / sda1, / dev / sdb1, / dev / SDC1, / dev / SDD1 will appear. Because there are four combined paths, the Linux system will consider each link hop as a hard disk.

3. Introduction to multipath under Linux

In CentOS 6.4, multipath is installed by default:

[[email protected] ~]# rpm -qa|grep mapper device-mapper-multipath-0.4.9-64.el6.x86_64 device-mapper-event-libs-1.02.77-9.el6.x86_64 device-mapper-multipath-libs-0.4.9-64.el6.x86_64 device-mapper-persistent-data-0.1.4-1.el6.x86_64 device-mapper-libs-1.02.77-9.el6.x86_64 device-mapper-event-1.02.77-9.el6.x86_64 device-mapper-1.02.77-9.el6.x86_64

Device mapper multipath: multipath tools. It mainly provides tools such as multipathd and multipath and configuration files such as multipath.conf. These tools create and configure multipath devices through the ioctr interface of device mapper (call the user space Library of device mapper, and the multipath devices created will be in / dev / mapper).

Device mapper: it mainly includes two parts: kernel part and user part.

The kernel consists of device mapper core (DM. KO) and some target drivers (md-multipath. KO). The core completes the device mapping, while target processes the I / O from the mapped device according to the mapping relationship and its own characteristics. At the same time, in the core part, an interface is provided. Users can communicate with the kernel part through ioctr to guide the behavior of kernel driver, such as how to create mapped device, the attributes of these dive, etc. The user space part of Linux device mapper mainly includes the device mapper package. These include the dmsetup tool and libraries to help create and configure mapped devices. These libraries are mainly abstract and encapsulate the interface for communication with ioctr, so as to facilitate the creation and configuration of mapped devices. These libraries need to be called in the program of multipath tool.

Dm-multipath.ko and dm.ko: dm.ko are device mapper drivers. It is the foundation of multipath. DM multipath is actually a target driver of DM.

Scsi_id: included in udev package, it can be configured in multipath.conf to get the serial number of SCSI device. Through the serial number, you can judge that multiple paths correspond to the same device. This is the key to multi-path implementation. Scsi_id is to send the inquery command of EVPD page80 or page83 to the device to query the identification of the SCSI device through the SG driver. However, some devices do not support EVPD’s inquery command, so they cannot be used to generate multipath devices. However, it is possible to override the scsi_id to provide a virtual identifier for devices that cannot provide the SCSI device identity and output it to the standard output. When the multipath program creates a multipath device, it calls the SCSI? ID to obtain the SCSI ID of the device from its standard output. When rewriting, you need to modify the return value of the SCSI ID program to 0. Because in the multipath program, this line is checked to determine if the SCSI ID has been successfully obtained.

4. Configure multipath

The basic configuration script is as follows:

[[email protected] ~]# cat /etc/multipath.conf defaults { polling_interval 30 failback immediate no_path_retry queue rr_min_io 100 path_checker tur user_friendly_names yes } # SVC device { vendor “IBM” product “2145” path_grouping_policy group_by_prio prio_callout “/sbin/mpath_prio_alua /dev/%n” }

Multipath basic operation command

#/ etc / init.d/multipathd start? Start the mulitipath service? Multipath – f? Delete the existing path? Multipath – V2? Format path? Multipath – ll? View multipath

If the configuration is correct, there will be more mpathbp1 and other devices in the / dev / mapper / directory. With fdisk – L command, you can see the disks created by multipath software, such as: / dev / mapper / mpathbp1

V. format hard disk

By executing fdisk – L, you can see that the storage has been identified successfully and the multipath configuration is correct. The information is as follows:

[[email protected] ~]# fdisk -l …… Disk /dev/mapper/mpathb: 2684.4 GB, 2684354560000 bytes 255 heads, 63 sectors/track, 326354 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk identifier: 0x00000000 Device Boot Start End Blocks Id System /dev/mapper/mpathbp1 1 267350 2147483647+ ee GPT ……

Through the above information, it can be found that the partition format of GPT is already available. Next, the hard disk needs to be formatted. If not, perform the following steps first:

1. Create a new partition

[root @ localhost ~] (pvcreate / dev / mapper / mpathb [root @ localhost ~] (parted / dev / mapper / mapper / mpathb GNU parted 2.1 using / dev / mapper / mapper / mpathbp1 welcome to GNU parted! Type ‘help’ to view a list of commands. (parted) mklabel GPT. Set the partition type to GPT (parted) mkpart extended 0% 100% toexpand the partition, and use the entire hard disk (the whole hard disk) to expand the partition, and use the entire hard disk (the whole hard disk).Parted) quit n: You may need to update /etc/fstab.

2. Format & Mount hard disk

[[email protected] ~]# mkfs.ext4 /dev/mapper/mpathbp1 [[email protected] ~]# mount /dev/mapper/mpathbp1 /test

After the mount is successful, you can use it.

3. Dynamic mount area

When a new partition is created in the system, because mount mount fails after system restart, you need to write the partition information to the / etc / fstab file for permanent mount.

[[email protected] ~]# vi /etc/fstab /dev/mapper/mpathbp1 /test ext4 defaults 1 2

Save and exit. After reboot, / dev / mapper / mpathbp1 will automatically mount to the / test directory