How to use disk multipath under Linux

Time:2021-10-19

Multipath configuration under Linux

1、 Basic configuration of multipath in Linux:

  1. Pass the command: lsmod | grep DM_ Multipath checks whether the installation is normal and successful. If there is no output indicating that it is not installed, install the software package through the yum function: Yum – y install device mapper device mapper multipath

Then, check the multipath status through the command: multipath – ll to see whether the module is loaded successfully

[ [email protected] ~]#Multipath – ll view multipath status

Mar 10 19:18:28 | /etc/multipath.conf does not exist, blacklisting all devices.

Mar 10 19:18:28 | A sample multipath.conf file is located at

Mar 10 19:18:28 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf

Mar 10 19:18:28 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf

Mar 10 19:18:28 – DM multipath kernel driver not loaded – DM module not loaded

If the module is not loaded successfully, please use the following command to initialize DM or restart the system
—Use the following commands to initialize and start DM for the first time:

  • modprobe dm-multipath
    modprobe dm-round-robin
    service multipathd start
    multipath –v2

After initialization, check whether the loading is successful through the multipath – ll command

[[email protected] ~]# multipath -ll

Mar 10 19:21:14 | /etc/multipath.conf does not exist, blacklisting all devices.

Mar 10 19:21:14 | A sample multipath.conf file is located at

Mar 10 19:21:14 | /usr/share/doc/device-mapper-multipath-0.4.9/multipath.conf

Mar 10 19:21:14 | You can run /sbin/mpathconf to create or modify /etc/multipath.conf

DM multipath kernel driver not loaded — if this prompt is missing, it indicates that the DM module has been loaded successfully.

As can be seen from the above prompt, the DM module is loaded successfully, but there is no multipath.conf configuration file under / etc /. The next step is how to configure the multipath.conf file.

How Disk Multipath works on Linux

2. Configure multipath:

Create a multipath configuration file via the VI command. The path is / etc / multipath.conf. Add the simplest configuration for multipath to work normally in the configuration file as follows:

vi /etc/multipath.conf

blacklist {
devnode “^sda”

}

defaults {
user_friendly_names yes

path_grouping_policy multibus

failback immediate

no_path_retry fail

}

After editing, save the configuration and use the command:

/Etc / init.d/multipathd start # starts the mulitipath service

If the service cannot be started and you are not prompted with OK, the following is true:

[[email protected] mapper]# service multipathd start

Starting multipathd daemon: no prompt OK

Just switch on and off the service again.

[[email protected] mapper]# /etc/init.d/multipathd stop

Stopping multipathd daemon: [ OK ]

[[email protected] mapper]# /etc/init.d/multipathd start

Starting multipathd daemon: [OK] – prompt OK to start the service normally

View via command:

[[email protected] mapper]# multipath -ll

Mpatha (360a980064665072443469563477396c) dm-0 NetApp, Lun – creates a Lun

size=3.5G features=’0′ hwhandler=’0′ wp=rw

`-+- policy=’round-robin 0′ prio=4 status=active

|-1:0:0:0 SDB 8:16 active ready running — two drive letters SDB and SDE under multipath

`- 2:0:0:0 sde 8:64 active ready running

There are two more folders mpatha and mpathap1 in the directory / dev / mapper /.

[[email protected] mapper]# cd /dev/mapper/

[[email protected] mapper]# ls

control mpatha mpathap1

At the same time, two device IDs are added under the command of fdisk – L:

When multipathing is not configured:

[[email protected]~]# fdisk -l

Disk /dev/sda: 146.8 GB, 146815733760 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000a6cdd

Device Boot Start End Blocks Id System

/dev/sda1 * 1 26 204800 83 Linux

Partition 1 does not end on cylinder boundary.

/dev/sda2 26 287 2097152 82 Linux swap / Solaris

Partition 2 does not end on cylinder boundary.

/dev/sda3 287 17850 141071360 83 Linux

Disk /dev/sdb: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

Device Boot Start End Blocks Id System

/dev/sdb1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/sde: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

Device Boot Start End Blocks Id System

/dev/sde1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Two can network cards get the same drive letter:

/Dev / SDE and / dev / SDB

After configuration, / dev / mapper / mpatha and / dev / mapper / mpathap1 are added:

[[email protected] mapper]# fdisk -l

Disk /dev/sda: 146.8 GB, 146815733760 bytes

255 heads, 63 sectors/track, 17849 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk identifier: 0x000a6cdd

Device Boot Start End Blocks Id System

/dev/sda1 * 1 26 204800 83 Linux

Partition 1 does not end on cylinder boundary.

/dev/sda2 26 287 2097152 82 Linux swap / Solaris

Partition 2 does not end on cylinder boundary.

/dev/sda3 287 17850 141071360 83 Linux

Disk /dev/sdb: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

Device Boot Start End Blocks Id System

/dev/sdb1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/sde: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

Device Boot Start End Blocks Id System

/dev/sde1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/mapper/mpatha: 3774 MB, 3774873600 bytes

117 heads, 62 sectors/track, 1016 cylinders

Units = cylinders of 7254 * 512 = 3714048 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Disk identifier: 0xac956c3a

Device Boot Start End Blocks Id System

/dev/mapper/mpathap1 1 1016 3685001 83 Linux

Partition 1 does not start on physical sector boundary.

Disk /dev/mapper/mpathap1: 3773 MB, 3773441024 bytes

255 heads, 63 sectors/track, 458 cylinders

Units = cylinders of 16065 * 512 = 8225280 bytes

Sector size (logical/physical): 512 bytes / 512 bytes

I/O size (minimum/optimal): 4096 bytes / 65536 bytes

Alignment offset: 1024 bytes

Disk identifier: 0x00000000

Disk /dev/mapper/mpathap1 doesn’t contain a valid partition table

Multipath – F # delete the existing path and two new paths will be deleted

Multipath – V2 # format appears again after the path is formatted

15 Linux multipath examples third part
15 Linux multipath examples for Linux admin

3. Basic operation of multipath disk

To operate the disk generated by multipath software, directly operate the disk in the / dev / mapper / directory

Before partitioning the disk generated by multipath software, it is best to run the pvcreate command:

pvcreate /dev/mapper/mpatha

This directory is used for fdisk / dev / mapper / mpatha partition

When fdisk is used to partition and save the disk generated by multipath software, there will be an error, which is ignored

ls -l /dev/mapper/

[[email protected] mnt]# ls -l /dev/mapper/

total 0

crw-rw—-. 1 root root 10, 58 Mar 10 19:10 control

lrwxrwxrwx. 1 root root 7 Mar 10 20:28 mpatha -> ../dm-0

lrwxrwxrwx. 1 root root 7 Mar 10 20:33 mpathap1 -> ../dm-1

Mpathap1 is the partition of the multipath disk

Mkfs.ext4/dev/mapper/mpathap1 # formats the mpath1p1 partition into an ext4 file system

Mount / dev / mapper / mpathap1 / MNT / # mount mpathap1 partition

Use / dev / mapper / mpathap1 for formatting and mounting

How to Check if a Disk is Busy in Linux

4. Partition disk:

As mentioned above, the directory / dev / mapper / mpatha is used when partitioning

[[email protected]~]# fdisk /dev/mapper/mpatha

Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel

Building a new DOS disklabel with disk identifier 0xac956c3a.

Changes will remain in memory only, until you decide to write them.

After that, of course, the previous content won’t be recoverable.

Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)

WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to

switch off the mode (command ‘c’) and change display units to

sectors (command ‘u’).

Command (m for help): n ——————————- new partition

Command action

e extended

p primary partition (1-4)

P —————————————- primary partition

Partition number (1-4): 1

First cylinder (1-1016, default 1):

Using default value 1

Last cylinder, +cylinders or +size{K,M,G} (1-1016, default 1016):

Using default value 1016

Command (m for help): W ———————– writing a list is equivalent to saving

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

Note: if two nodes of the same device have the same drive letter, the other drive letter needs to be written w again. No n more.

5. Format:

[[email protected] ~]# mkfs.ext4 /dev/mapper/mpathap1

mke2fs 1.41.12 (17-May-2010)

/dev/sdd1 alignment is offset by 1024 bytes.

This may result in very poor performance, (re)-partitioning suggested.

Filesystem label=

OS type: Linux

Block size=4096 (log=2)

Fragment size=4096 (log=2)

Stride=1 blocks, Stripe width=16 blocks

230608 inodes, 921250 blocks

46062 blocks (5.00%) reserved for the super user

First data block=0

Maximum filesystem blocks=943718400

29 block groups

32768 blocks per group, 32768 fragments per group

7952 inodes per group

Superblock backups stored on blocks:

32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done

Creating journal (16384 blocks): done

Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 33 mounts or

180 days, whichever comes first. Use tune2fs -c or -i to override.

Using Linux Iotop to check disk IO usage Per process
Sort Disk Read Write in Linux Iotop Command

  1. Mount / dev / mapper / mpathap1 to / MNT

[[email protected] ~]# mount /dev/mapper/mpathap1 /mnt

3、 The advanced configuration of multipath. The previous configuration uses the default configuration of multipath to complete multipath, such as mapping device names and multipath load balancing methods. Is multipath configured according to our own defined method? The answer is OK.

1. Configuration of multipath.conf file

The next job is to edit the configuration file of / etc / multipath.conf

Multipath.conf mainly includes the configuration of blacklist, multipaths and devices

Blacklist configuration

blacklist {
devnode “^sda”

}

The multipaths section configures the configuration of multipaths and devices.

multipaths {
multipath {
wwid **# this value can be seen in multipath – v3

Alias iscsi-dm0 # mapped alias. You can choose any alias

path_ grouping_ Policy multibus # path group policy

path_ Checker tur # determines the path state

path_ Selector “round robin 0” # select the path for the next IO operation

}

}

Devices section configuration

devices {
device {
Vendor “iSCSI enterprise” # vendor name

Product “virtual disk” # product model

path_ grouping_ Policy multibus # default path group policy

getuid_ Callout “/ SBIN / scsi_id – G – U – S / block /% n” # get the default program used by the unique device number

prio_ Callout “/ SBIN / acs_prio_alua% d” # get the default program used for finite level values

path_ Checker readsector0 # determines the path status

path_ Selector “round robin 0” # select the path for the next IO operation

Failback immediate # failure recovery mode

no_ path_ Retry queue # the number of times the system tries to use the invalid path before disable queue

rr_ min_ IO 100 # the number of IO requests in the current user group before switching to another path

}

}

A complete advanced configuration in my local is as follows:

[[email protected] ~]# vi /etc/multipath.conf

blacklist {
devnode “^sda”

}

multipaths {
multipath {
wwid 360a98000646650724434697454546156

alias mpathb_fcoe

path_grouping_policy multibus

path_checker “directio”

prio “random”

path_selector “round-robin 0”

}

}

devices {
device {
vendor “NETAPP”

product “LUN”

getuid_callout “/lib/udev/scsi_id –whitelisted –device=/dev/%n”

path_checker “directio”

path_selector “round-robin 0”

failback immediate

no_path_retry fail

}

}

Where wwid, vendor, product, getuid_ Callout these parameters can be obtained through the: multipath – V3 command. If each wwid alias is set in / etc / multipath.conf, the alias will override this setting.

15 Linux multipath command examples

4、 Load balancing test:

You can use the DD command to read and write to the device, and view the I / 0 status through iostat. Which path does the traffic go from:

DD command: DD if = / dev / zero of = / MNT / 1gfile BS = 8K count = 131072 above, we have mounted the disk in the / MNT folder, so we can directly read and write to the / MNT folder when reading and writing the disk.

If you want to repeatedly read and write to the disk, you can use the following statement:

[ [email protected] ~]# for ((i=1;i<=5;i++)); do dd if=/dev/zero of=/mnt/1Gfile bs=8k count=131072 2>&1|grep MB; done; — Repeat reading and writing for 5 times. This value can be modified according to your test requirements.

How to Find disk SCSI ID WWIDS In Linux

On the other console, enter iostat 2 10 to view the IO read / write status:

You can see that SDC and SDD are two multi-path drive letters. The traffic is uniform. In the two paths, the load balancing is very successful.

5、 Path redundancy backup test

When the port of one path is down, all traffic will be directly switched to the other path.

Troubleshooting Disk Space Usage and Disk IO Issues In Linux
How to Check Disk space usage Disk utilization in Linux

Recommended Today

SQL exercise 20 – Modeling & Reporting

This blog is used to review and sort out the common topic modeling architecture, analysis oriented architecture and integration topic reports in data warehouse. I have uploaded these reports to GitHub. If you are interested, you can have a lookAddress:https://github.com/nino-laiqiu/TiTanI recorded a relatively complete development process in my hexo blog deployed on GitHub. You can […]