[one-users] iSCSI multipath

Tue Jan 22 03:10:35 PST 2013

Hi,

thank you both for such a high quality and educational thread. I wasn't
aware of the possibility of configuring the LVM metadata in a host as
read-only, but it fits beautifully with OpenNebula.

Mihály, would you like to share your experiences somewhere in the
OpenNebula community? wiki.opennebula.org or blog.opennebula.org? I think
many people can benefit from your setup.

cheers,
Jaime

On Mon, Jan 21, 2013 at 6:58 PM, Miloš Kozák <milos.kozak at lejmr.com> wrote:

> Oh snap, that sounds great I didn't know about that.. it makes all easier.
> In this scenario only frontend can work with LVM, so no issues of
> concurrent change. Only one last think to make it really safe against that.
> Is there any way to suppress LVM changes from hosts, make it read only? And
> let it RW at frontend?
>
> Thanks
>
>
> Dne 21.1.2013 18:50, Mihály Héder napsal(a):
>
>> Hi,
>>
>> no, you don't have to do any of that. Also, nebula doesn't have to
>> care about LVM metadata at all and therefore there is no corresponding
>> function in it. At /etc/lvm there is no metadata, only configuration
>> files.
>>
>> Lvm metadata simply sits somewhere at the beginning of your
>> iscsi-shared disk, like a partition table. So it is on the storage
>> that is accessed by all your hosts, and no distribution is necessary.
>> Nebula frontend simply issues lvcreate, lvchange, etc, on this shared
>> disk and those commands will manipulate the metadata.
>>
>> It is really LVM's internal business, many layers below opennebula.
>> All you have to make sure that you don't run these commands
>> concurrently  from multiple hosts on the same iscsi-attached disk,
>> because then they could interfere with each other. This setting is
>> what you have to indicate in /etc/lvm on the server hosts.
>>
>> Cheers
>> Mihály
>>
>>
>> On 21 January 2013 18:37, Miloš Kozák <milos.kozak at lejmr.com> wrote:
>>
>>> Thank you. does it mean, that I can distribute metadata files located in
>>> /etc/lvm on frontend onto other hosts and these hosts will see my logical
>>> volumes? Is there any code in nebula which would provide it? Or I need to
>>> update DS scripts to update/distribute LVM metadata among servers?
>>>
>>> Thanks, Milos
>>>
>>> Dne 21.1.2013 18:29, Mihály Héder napsal(a):
>>>
>>>  Hi,
>>>>
>>>> lvm metadata[1] is simply stored on the disk. In the setup we are
>>>> discussing this happens to be a  shared virtual disk on the storage,
>>>> so any other hosts that are attaching the same virtual disk should see
>>>> the changes as they happen, provided that they re-read the disk. This
>>>> re-reading step is what you can trigger with lvscan, but nowadays that
>>>> seems to be unnecessary. For us it works with Centos 6.3 so I guess Sc
>>>> Linux should be fine as well.
>>>>
>>>> Cheers
>>>> Mihály
>>>>
>>>>
>>>> [1]
>>>> http://www.centos.org/docs/5/**html/Cluster_Logical_Volume_**
>>>> Manager/lvm_metadata.html<http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html>
>>>>
>>>> On 21 January 2013 12:53, Miloš Kozák <milos.kozak at lejmr.com> wrote:
>>>>
>>>>> Hi,
>>>>> thank you for great answer. As I wrote my objective is to avoid as much
>>>>> of
>>>>> clustering sw (pacemaker,..) as possible, so clvm is one of these
>>>>> things
>>>>> I
>>>>> feel bad about them in my configuration.. Therefore I would rather let
>>>>> nebula manage LVM metadata in the first place as I you wrote. Only one
>>>>> last
>>>>> thing I dont understand is a way nebula distributes LVM metadata?
>>>>>
>>>>> Is kernel in Scientific Linux 6.3 new enought to LVM issue you
>>>>> mentioned?
>>>>>
>>>>> Thanks Milos
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Dne 21.1.2013 12:34, Mihály Héder napsal(a):
>>>>>
>>>>>  Hi!
>>>>>>
>>>>>> Last time we could test an Equalogic it did not have option for
>>>>>> create/configure Virtual Disks inside in it by an API, so I think the
>>>>>> iSCSI driver is not an alternative, as it would require a
>>>>>> configuration step per virtual machine on the storage.
>>>>>>
>>>>>> However, you can use your storage just fine in a shared LVM scenario.
>>>>>> You need to consider two different things:
>>>>>> -the LVM metadata, and the actual VM data on the partitions. It is
>>>>>> true, that the concurrent modification of the metadata should be
>>>>>> avoided as in theory it can damage the whole virtual group. You could
>>>>>> use clvm which avoids that by clustered locking, and then every
>>>>>> participating machine can safely create/modify/delete LV-s. However,
>>>>>> in a nebula setup this is not necessary in every case: you can make
>>>>>> the LVM metadata read only on your host servers, and let only the
>>>>>> frontend modify it. Then it can use local locking that does not
>>>>>> require clvm.
>>>>>> -of course the host servers can write the data inside the partitions
>>>>>> regardless that the metadata is read-only for them. It should work
>>>>>> just fine as long as you don't start two VMs for one partition.
>>>>>>
>>>>>> We are running this setup with a dual controller Dell MD3600 storage
>>>>>> without issues so far. Before that, we used to do the same with XEN
>>>>>> machines for years on an older EMC (that was before nebula). Now with
>>>>>> nebula we have been using a home-grown module for doing that, which I
>>>>>> can send you any time - we plan to submit that as a feature
>>>>>> enhancement anyway. Also, there seems to be a similar shared LVM
>>>>>> module in the nebula upstream which we could not get to work yet, but
>>>>>> did not try much.
>>>>>>
>>>>>> The plus side of this setup is that you can make live migration work
>>>>>> nicely. There are two points to consider however: once you set the LVM
>>>>>> metadata read-only you wont be able to modify the local LVMs in your
>>>>>> servers, if there are any. Also, in older kernels, when you modified
>>>>>> the LVM on one machine the others did not get notified about the
>>>>>> changes, so you had to issue an lvs command. However in new kernels
>>>>>> this issue seems to be solved, the LVs get instantly updated. I don't
>>>>>> know when and what exactly changed though.
>>>>>>
>>>>>> Cheers
>>>>>> Mihály Héder
>>>>>> MTA SZTAKI ITAK
>>>>>>
>>>>>> On 18 January 2013 08:57, Miloš Kozák <milos.kozak at lejmr.com> wrote:
>>>>>>
>>>>>>> Hi, I am setting up a small installation of opennebula with
>>>>>>> sharedstorage
>>>>>>> using iSCSI. THe storage is Equilogic EMC with two controllers.
>>>>>>> Nowadays
>>>>>>> we
>>>>>>> have only two host servers so we use backed direct connection between
>>>>>>> storage and each server, see attachment. For this purpose we set up
>>>>>>> dm-multipath. Cause in the future we want to add other servers and
>>>>>>> some
>>>>>>> other technology will be necessary in the network segment. Thesedays
>>>>>>> we
>>>>>>> try
>>>>>>> to make it as same as possible with future topology from protocols
>>>>>>> point
>>>>>>> of
>>>>>>> view.
>>>>>>>
>>>>>>> My question is related to the way how to define datastore, which
>>>>>>> driver
>>>>>>> and
>>>>>>> TM is the best and which?
>>>>>>>
>>>>>>> My primal objective is to avoid GFS2 or any other cluster filesystem
>>>>>>> I
>>>>>>> would
>>>>>>> prefer to keep datastore as block devices. Only option I see is to
>>>>>>> use
>>>>>>> LVM
>>>>>>> but I worry about concurent writes isn't it a problem? I was
>>>>>>> googling a
>>>>>>> bit
>>>>>>> and I found I would need to set up clvm - is it really necessary?
>>>>>>>
>>>>>>> Or is better to use iSCSI driver, drop the dm-multipath and hope?
>>>>>>>
>>>>>>> Thanks, Milos
>>>>>>>
>>>>>>> ______________________________**_________________
>>>>>>> Users mailing list
>>>>>>> Users at lists.opennebula.org
>>>>>>> http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**org<http://lists.opennebula.org/listinfo.cgi/users-opennebula.org>
>>>>>>>
>>>>>>>  ______________________________**_________________
>>> Users mailing list
>>> Users at lists.opennebula.org
>>> http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**org<http://lists.opennebula.org/listinfo.cgi/users-opennebula.org>
>>>
>>
> ______________________________**_________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/**listinfo.cgi/users-opennebula.**org<http://lists.opennebula.org/listinfo.cgi/users-opennebula.org>
>

-- 
Jaime Melis
Project Engineer
OpenNebula - The Open Source Toolkit for Cloud Computing
www.OpenNebula.org | jmelis at opennebula.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20130122/0da4686d/attachment-0001.htm>