Archive for May, 2010

04
May

Linux LVM: logical volume migration

I was recently confronted with the task of migrating logical volume holding MySQL databases to a separate physical volume. Due to important application running on top of MySQL any downtime was out of the question.

Easiest way to perform this operation would be using pvmove command.

[root@r2d2 ~]# pvmove -n test /dev/sda5 /dev/sdb1
/dev/sda5: Moved: 6.2%
/dev/sda5: Moved: 18.8%
/dev/sda5: Moved: 28.1%
/dev/sda5: Moved: 37.5%
/dev/sda5: Moved: 46.9%
/dev/sda5: Moved: 56.2%
/dev/sda5: Moved: 65.6%
/dev/sda5: Moved: 75.0%
/dev/sda5: Moved: 84.4%
/dev/sda5: Moved: 93.8%
/dev/sda5: Moved: 100.0%

In the example we moved logical volume 'test' from physical volume /dev/sda5 to /dev/sdb1. While pvmove is running, we can get the necessary information including the progress by issuing command lvs.

[root@r2d2 ~]# lvs -a -o+devices
LV        VG  Attr   LSize  Move     Copy% Devices
home      sys -wi-ao 20.00G                /dev/sda5(544)
mysql     sys -wi-a- 1.00G                 /dev/sda5(1888)
[pvmove0] sys p-C-ao 1.00G /dev/sda5 56.25 /dev/sda5(1920),/dev/sdb1(0)
root      sys -wi-ao 7.00G                 /dev/sda5(256)
swap      sys -wi-ao 2.00G                 /dev/sda5(480)
test      sys -wI-ao 1.00G                 pvmove0(0)
usr       sys -wi-ao 8.00G                 /dev/sda5(0)
usr       sys -wi-ao 8.00G                 /dev/sda5(1856)
var       sys -wi-ao 2.00G                 /dev/sda5(224)
var       sys -wi-ao 2.00G                 /dev/sda5(1824)

Another, and somewhat more tricky way of doing this is to use LV mirroring. I say more tricky because it requires more work and the last part of the operation is not really intuitive so it may be prone to errors.

We start with this:

[root@r2d2 ~]# pvs
PV        VG  Fmt  Attr PSize   PFree
/dev/sda5 sys lvm2 a-   220.75G 179.75G
/dev/sdb1 sys lvm2 a-   3.72G   3.72G
 
[root@r2d2 ~]# lvs -a -o+devices
LV   VG  Attr   LSize Devices
[---]
test sys -wi-a- 1.00G /dev/sda5(1920)
[---]

Two PVs, the source - sda5, and unused, target PV - sdb1. Logical volume test is the LV we want to move to sdb1.

In order to verify that the content of the LV is consistent on the end, we will create a sample file and compare its content before and after.

[root@r2d2 ~]# dd if=/dev/zero of=/mnt/random bs=512 count=20000
20000+0 records in
20000+0 records out
10240000 bytes (10 MB) copied, 0.122046 s, 83.9 MB/s
 
[root@r2d2 ~]# md5sum /mnt/random
596c35b949baf46b721744a13f76a258 /mnt/random

We start by adding another copy to 'test' LV and placing it on a new PV.

[root@r2d2 ~]# lvconvert -m 1 sys/test /dev/sdb1
Not enough PVs with free space available for parallel allocation.
Consider --alloc anywhere if desperate.
Unable to allocate extents for mirror(s).

Hm, confusing error. :) The reason for this is that Linux LVM implementation requires mirror to use a log volume, and this log volume needs to reside on a PV of its own. In order to go around this we can instruct LVM to place the log in the memory. This is not recommended for situations when mirroring is needed on the long run, since mirror needs to be rebuilt every time the LV is activated. But this is the perfect solution for our needs as our mirror will not last for long.

[root@r2d2 ~]# lvconvert -m 1 sys/test /dev/sdb1 --mirrorlog core
sys/test: Converted: 6.2%
sys/test: Converted: 15.6%
sys/test: Converted: 28.1%
sys/test: Converted: 37.5%
sys/test: Converted: 46.9%
sys/test: Converted: 56.2%
sys/test: Converted: 65.6%
sys/test: Converted: 75.0%
sys/test: Converted: 84.4%
sys/test: Converted: 93.8%
sys/test: Converted: 100.0%
Logical volume test converted.

Again, we can get all information with lvs command.

[root@r2d2 ~]# lvs -a -o+devices
LV              VG  Attr   LSize Copy%  Devices
home            sys -wi-ao 20.00G       /dev/sda5(544)
mysql           sys -wi-a- 1.00G        /dev/sda5(1888)
root            sys -wi-ao 7.00G        /dev/sda5(256)
swap            sys -wi-ao 2.00G        /dev/sda5(480)
test            sys mwi-a- 1.00G 15.62  test_mimage_0(0),test_mimage_1(0)
[test_mimage_0] sys Iwi-ao 1.00G        /dev/sda5(1920)
[test_mimage_1] sys Iwi-ao 1.00G        /dev/sdb1(0)
usr             sys -wi-ao 8.00G        /dev/sda5(0)
usr             sys -wi-ao 8.00G        /dev/sda5(1856)
var             sys -wi-ao 2.00G        /dev/sda5(224)
var             sys -wi-ao 2.00G        /dev/sda5(1824)

So now we have our LV mirrored with copies on both PVs. Just for test, we can verify the content of the file.

[root@r2d2 ~]# md5sum /mnt/random
596c35b949baf46b721744a13f76a258 /mnt/random

Seems fine. :)

Last step, the one where we have to be more careful, is removing mirror copy from old PV. We have to be more careful because the PV name in this case is the name of a PV we want to remove from mirroring, which is the opposite from what we had when establishing the mirror. If we use wrong PV name we will end up on the begining, with LV on a wrong PV and without mirroring.

[root@r2d2 ~]# lvconvert -m 0 sys/test /dev/sda5
Logical volume test converted.

Again we use lvs to see the status.

[root@r2d2 ~]# lvs -a -o+devices
LV   VG  Attr   LSize Devices
[---]
test sys -wi-a- 1.00G /dev/sdb1(0)
[---]

On the end, to verify that we have a clean situation, we can remount the LV and compare the content.

[root@r2d2 ~]# umount /mnt/
[root@r2d2 ~]# lvchange -an /dev/sys/test
[root@r2d2 ~]# lvchange -ay /dev/sys/test
[root@r2d2 ~]# mount /dev/sys/test /mnt
 
[root@r2d2 ~]# md5sum /mnt/random
596c35b949baf46b721744a13f76a258 /mnt/random

Everything fine. :)

Just a small note for the end. In case you try to create a mirror on a logical volume that was not yet activated, sync of the mirror will start after the activation.

[root@r2d2 ~]# lvconvert -m 1 sys/test /dev/sdb1 --mirrorlog core
Conversion starts after activation