Migration root disk into mirror in LVM

I’ve found several manuals describing how to add second disk to Volume Group and transform it into mirror. This does not looks complex, and everyone can do it, when everything is working like described in documentation. The problems start when something goes wrong, and one of steps failed.
In my environment there is virtual machine with CentOS 7 hosted by VMWare ESXi. This VM has system disk on one datastore. To provide redundation on OS level I decided to add second disk with the same capacity from another datastore and create RAID-1 (mirror) on them.

Current situation looks as follows:

[root@prod ~]# lvs
  LV   VG     Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  root centos -wi-ao---- 11.60g
  swap centos -wi-ao----  3.91g
  srv  data   rwi-aor--- 99.99g                               100.00

[root@prod ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  centos lvm2 a--   15.51g    0
  /dev/sdb   data   lvm2 a--  100.00g    0
  /dev/sdc   data   lvm2 a--  100.00g    0

[root@prod ~]# pvscan
  PV /dev/sda2   VG centos   lvm2 [15.51 GiB / 0    free]
  PV /dev/sdb    VG data     lvm2 [100.00 GiB / 0    free]
  PV /dev/sdc    VG data     lvm2 [100.00 GiB / 0    free]
  Total: 3 [215.50 GiB] / in use: 3 [215.50 GiB] / in no VG: 0 [0   ]

[root@prod ~]# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "centos" using metadata type lvm2
  Found volume group "data" using metadata type lvm2

There is already VG “centos” created on physical volumene /dev/sda2, which contains two logical volumens: root and swap. There is also second VG named “data”, but this one is already mirrored, and we have nothing to do with that. Let’s take a look on /dev/sda2 volume:

[root@prod ~]# pvdisplay
  --- Physical volume ---
  PV Name               /dev/sda2
  VG Name               centos
  PV Size               15.51 GiB / not usable 3.00 MiB
  Allocatable           yes (but full)
  PE Size               4.00 MiB
  Total PE              3970
  Free PE               0
  Allocated PE          3970
  PV UUID               RJaYrc-pV0r-qfNf-S92B-NSJc-mLpF-A5GOkf

As I already wrote, this is VMWare environment, so I created volume with the same capacity on second datastore and I connected it to Virtual Machine with CentOS. Operating system does not know yet about it, so we need to perform bus scan. The simpliest way is to reboot VM, but this requires downtime, and this can be inadvisable. Let’s do it without restart, we need to force bus scan in OS. But at first we take a look what we have:

[root@prod ~]# fdisk -l

Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00015ff8

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048    33554431    16264192   8e  Linux LVM

Disk /dev/mapper/centos-swap: 4194 MB, 4194304000 bytes, 8192000 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/centos-root: 12.5 GB, 12457082880 bytes, 24330240 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/data-srv_rmeta_0: 4 MB, 4194304 bytes, 8192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/data-srv_rimage_0: 107.4 GB, 107365793792 bytes, 209698816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/data-srv_rmeta_1: 4 MB, 4194304 bytes, 8192 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/data-srv_rimage_1: 107.4 GB, 107365793792 bytes, 209698816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/mapper/data-srv: 107.4 GB, 107365793792 bytes, 209698816 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Please notice additional metadata in mirrored data-srv volume in opposition to centos-root and centos-swap.
Let’s scan the bus:

[root@prod ~]# echo "- - -" > /sys/class/scsi_host/host0/scan
[root@prod ~]# echo "- - -" > /sys/class/scsi_host/host1/scan

It is good to know to which bus is new disc connected, but there should be no problems when you scan all of them. It is worthwhile to check system logs if anything new appeared.

[root@prod ~]# less /var/log/messages
[root@prod ~]# echo "- - -" > /sys/class/scsi_host/host2/scan
[root@prod ~]# less /var/log/messages

Let’s take a look once more what we have:

[root@prod ~]# fdisk -l

Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Disk /dev/sda: 17.2 GB, 17179869184 bytes, 33554432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00015ff8

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *        2048     1026047      512000   83  Linux
/dev/sda2         1026048    33554431    16264192   8e  Linux LVM

Disk /dev/mapper/centos-swap: 4194 MB, 4194304000 bytes, 8192000 sectors
[...]

Disk /dev/sdd: 17.2 GB, 17179869184 bytes, 33554432 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

I snipped output, because it looks like before, the only change is that after logical volumes list there is new physical device /dev/sdd.
Now we need to create the same partition layout like on primary system disk:

[root@prod ~]# sfdisk -d /dev/sda
# partition table of /dev/sda
unit: sectors

/dev/sda1 : start=     2048, size=  1024000, Id=83, bootable
/dev/sda2 : start=  1026048, size= 32528384, Id=8e
/dev/sda3 : start=        0, size=        0, Id= 0
/dev/sda4 : start=        0, size=        0, Id= 0

And this partition layout will be saved to new disk:

[root@prod ~]# sfdisk -d /dev/sda  |sfdisk /dev/sdd
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdd: 2088 cylinders, 255 heads, 63 sectors/track
sfdisk:  /dev/sdd: unrecognized partition table type

Old situation:
sfdisk: No partitions found

New situation:
Units: sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdd1   *      2048   1026047    1024000  83  Linux
/dev/sdd2       1026048  33554431   32528384  8e  Linux LVM
/dev/sdd3             0         -          0   0  Empty
/dev/sdd4             0         -          0   0  Empty
Warning: partition 1 does not end at a cylinder boundary
Warning: partition 2 does not start at a cylinder boundary
Warning: partition 2 does not end at a cylinder boundary
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)

Let’s take a look how it looks now on /dev/sdd:

[root@prod ~]# sfdisk -d /dev/sdd
# partition table of /dev/sdd
unit: sectors

/dev/sdd1 : start=     2048, size=  1024000, Id=83, bootable
/dev/sdd2 : start=  1026048, size= 32528384, Id=8e
/dev/sdd3 : start=        0, size=        0, Id= 0
/dev/sdd4 : start=        0, size=        0, Id= 0

Looks good, let’s create phisycal volume:

[root@prod ~]# pvcreate /dev/sdd2
  Physical volume "/dev/sdd2" successfully created

[root@prod ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  centos lvm2 a--   15.51g     0
  /dev/sdb   data   lvm2 a--  100.00g     0
  /dev/sdc   data   lvm2 a--  100.00g     0
  /dev/sdd2         lvm2 a--   15.51g 15.51g

And expand “centos” Volume Group:

[root@prod ~]# vgextend centos /dev/sdd2
  Volume group "centos" successfully extended

[root@prod ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  centos lvm2 a--   15.51g     0
  /dev/sdb   data   lvm2 a--  100.00g     0
  /dev/sdc   data   lvm2 a--  100.00g     0
  /dev/sdd2  centos lvm2 a--   15.51g 15.51g

Done, now transform it into mirror:

[root@prod ~]# lvconvert -m 1 --corelog centos/root
  Insufficient free space: 1 extents needed, but only 0 available
[root@prod ~]# lvconvert -m 1 --alloc anywhere centos/root
  Insufficient free space: 1 extents needed, but only 0 available

Ooops…

And here all of mentioned at beginning manuals becomes useless. Those which I found didn’t predict this situation or didn’t provide resolution for this. There was some tries with allocation of space for metadata in different places or shrinking partition, but it ended without success or total disaster with data consistency. During my research I didn’t found the solution which I applied, that’s why I’m describing this. I decided to switch off swap space and reduce swap partition, to provide some space for metadata. Let’s begin from checking:

[root@prod ~]# swapon
NAME      TYPE      SIZE USED PRIO
/dev/dm-0 partition 3.9G   0B   -1

Let’s switch off swap space (if there is only one, we can use -a):

[root@prod ~]# swapoff -av
swapoff /dev/dm-0

Let’s check again:

[root@prod ~]# swapon

Now it’s time to reduce logical volume (here is my mistake, where I put “512M” instead of “-512M”, and volume was reduced to 512M, so remember about minus sign):

[root@prod ~]# lvreduce centos/swap -L 512M
  WARNING: Reducing active logical volume to 512.00 MiB
  THIS MAY DESTROY YOUR DATA (filesystem etc.)
Do you really want to reduce swap? [y/n]: y
  Reducing logical volume swap to 512.00 MiB
  Logical volume swap successfully resized

Let’s check:

[root@prod ~]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  root centos -wi-ao----  11.60g
  swap centos -wi-a----- 512.00m
  srv  data   rwi-aor---  99.99g                               100.00

Some space are freed, so let’s try to transform root filesystem into mirrored fs:

[root@prod ~]# lvconvert -m 1 centos/root
[root@prod ~]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  root centos rwi-aor---  11.60g                                 0.00
  swap centos -wi-a----- 512.00m
  srv  data   rwi-aor---  99.99g                               100.00

Voila! Operation completed successfully, let’s do the same with swap:

[root@prod ~]# lvconvert -m 1 centos/swap

And check:

[root@prod ~]# lvs
  LV   VG     Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert
  root centos rwi-aor---  11.60g                                55.66
  swap centos rwi-a-r--- 512.00m                               100.00
  srv  data   rwi-aor---  99.99g                               100.00

Mirrors are done, root filesystem is syncing, but we can expand swap to maximal available capacity:

[root@prod ~]# lvextend -l 100%FREE centos/swap
  Extending 2 mirror images.
  Extending logical volume swap to 6.80 GiB
  device-mapper: resume ioctl on  failed: Invalid argument
  Unable to resume centos-swap (253:0)
  Problem reactivating swap
  libdevmapper exiting with 1 device(s) still suspended.

Unfortunatelly system didn’t activate swap automatically, because swap signature was destroyed and needs to be restored:

[root@prod ~]# mkswap /dev/centos/swap

Now we can activate swap in system:

[root@prod ~]# swapon -av
swapon /dev/mapper/centos-swap
swapon: /dev/mapper/centos-swap: found swap signature: version 1, page-size 4, same byte order
swapon: /dev/mapper/centos-swap: pagesize=4096, swapsize=4185915392, devsize=4185915392

Let’s check if it works:

[root@prod ~]# swapon
NAME      TYPE      SIZE USED PRIO
/dev/dm-4 partition 3.9G   0B   -1

This way I went a crash course of LVM logic, because I didn’t have opportunity to manage LVM before. I hope this manual will help those which encountered similar problem.

You may also like...

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.