Hi,<br><br>We have a custom datastore, and transfer manager driver, which runs the lvchange command when it is needed.<br>In order to work, you have to enable it in oned.conf.<br><br>for example:<br><span style="font-family:courier new,monospace"><br>
DATASTORE_MAD = [<br> executable = "one_datastore",<br> arguments = "-t 10 -d fs,vmware,iscsi,lvm,shared_lvm"]<br><br>TM_MAD = [<br> executable = "one_tm",<br> arguments = "-t 10 -d dummy,lvm,shared,qcow2,ssh,vmware,iscsi,shared_lvm" ]</span><br clear="all">
<div><br>After that, you can create a datastore, with the shared_lvm tm and datastore driver.<br><br>The only limitation is that you can't live migrate VM-s. We have a working solution for that as well, but it is still untested.I can send you that too, if you want to help us testing it.<br>
<br>Anyway, here are the drivers, feel free to use or modify it.<br><a href="https://dl.dropbox.com/u/140123/shared_lvm.tar.gz">https://dl.dropbox.com/u/140123/shared_lvm.tar.gz</a><br><br>--<br>Cheers,<br>Marlok Tamas<br>
MTA Sztaki<br><br></div>
<br><br><div class="gmail_quote">On Thu, Jan 24, 2013 at 11:32 PM, Mihály Héder <span dir="ltr"><<a href="mailto:mihaly.heder@sztaki.mta.hu" target="_blank">mihaly.heder@sztaki.mta.hu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
Well, if you can run the lvs or lvscan on at least one server<br>
successfully, then the metadata is probably fine.<br>
We had similar issues before we learned how to exclude unnecessary<br>
block devices in the lvm config.<br>
<br>
The thing is that lvscan and lvs will try to check _every_ potential<br>
block device by default for LVM partitions. If you are lucky, this is<br>
only annoying, because it will throw 'can't read /dev/sdX' or similar<br>
messages. However, if you are using dm-multipath, you will have one<br>
device for each path, like /dev/sdr _plus_ the aggregated device with<br>
the name you have configured in multipath.conf (/dev/mapper/yourname)<br>
what you actually need. LVM did not quite understand this situation<br>
and got stuck on the individual path devices, so we have configured to<br>
look for lvm only on the right place. In man page of lvm.conf look for<br>
the devices / scan and filter options. Also there are quite good<br>
examples in the comments there.<br>
<br>
Also, there could be a much simpler explanation to the issue:<br>
something with the iSCSI connection or multipath that are one layer<br>
below.<br>
<br>
I hope this helps.<br>
<br>
Cheers<br>
Mihály<br>
<br>
On 24 January 2013 23:18, Miloš Kozák <<a href="mailto:milos.kozak@lejmr.com">milos.kozak@lejmr.com</a>> wrote:<br>
> Hi, thank you. I tried to update TM ln script, which works but it is not<br>
> clean solution. So I will try to write hook code and then we can discuss it.<br>
><br>
> I deployed a few VM and now on the other server lvs command freezes. I have<br>
> not set up clvm, do you think it could be caused by lvm metadata corruption?<br>
> The thing is I can not longer start a VM on the other server.<br>
><br>
> Miloš<br>
><br>
> Dne 24.1.2013 23:10, Mihály Héder napsal(a):<br>
<div class="HOEnZb"><div class="h5">><br>
>> Hi!<br>
>><br>
>> We solve this problem via hooks that are activating the LV-s for us<br>
>> when we start/migrate a VM. Unfortunately I will be out of office<br>
>> until early next week but then I will consult with my colleague who<br>
>> did the actual coding of this part and we will share the code.<br>
>><br>
>> Cheers<br>
>> Mihály<br>
>><br>
>> On 24 January 2013 20:15, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com">milos.kozak@lejmr.com</a>> wrote:<br>
>>><br>
>>> Hi, I have just set it up having two hosts with shared blockdevice. On<br>
>>> top<br>
>>> of that LVM, as discussed earlier. Triggering lvs I can see all logical<br>
>>> volumes. When I create a new LV on the other server, I can see the LV<br>
>>> being<br>
>>> inactive, so I have to run lvchange -ay VG/LV enable it then this LV can<br>
>>> be<br>
>>> used for VM..<br>
>>><br>
>>> Is there any trick howto auto enable newly created LV on every host?<br>
>>><br>
>>> Thanks Milos<br>
>>><br>
>>> Dne 22.1.2013 18:22, Mihály Héder napsal(a):<br>
>>><br>
>>>> Hi!<br>
>>>><br>
>>>> You need to look at locking_type in the lvm.conf manual [1]. The<br>
>>>> default - locking in a local directory - is ok for the frontend, and<br>
>>>> type 4 is read-only. However, you should not forget that this only<br>
>>>> prevents damaging thing by the lvm commands. If you start to write<br>
>>>> zeros to your disk with the dd command for example, that will kill<br>
>>>> your partition regardless the lvm setting. So this is against user or<br>
>>>> middleware errors mainly, not against malicious attacks.<br>
>>>><br>
>>>> Cheers<br>
>>>> Mihály Héder<br>
>>>> MTA SZTAKI<br>
>>>><br>
>>>> [1] <a href="http://linux.die.net/man/5/lvm.conf" target="_blank">http://linux.die.net/man/5/lvm.conf</a><br>
>>>><br>
>>>> On 21 January 2013 18:58, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com">milos.kozak@lejmr.com</a>> wrote:<br>
>>>>><br>
>>>>> Oh snap, that sounds great I didn't know about that.. it makes all<br>
>>>>> easier.<br>
>>>>> In this scenario only frontend can work with LVM, so no issues of<br>
>>>>> concurrent<br>
>>>>> change. Only one last think to make it really safe against that. Is<br>
>>>>> there<br>
>>>>> any way to suppress LVM changes from hosts, make it read only? And let<br>
>>>>> it<br>
>>>>> RW<br>
>>>>> at frontend?<br>
>>>>><br>
>>>>> Thanks<br>
>>>>><br>
>>>>><br>
>>>>> Dne 21.1.2013 18:50, Mihály Héder napsal(a):<br>
>>>>><br>
>>>>>> Hi,<br>
>>>>>><br>
>>>>>> no, you don't have to do any of that. Also, nebula doesn't have to<br>
>>>>>> care about LVM metadata at all and therefore there is no corresponding<br>
>>>>>> function in it. At /etc/lvm there is no metadata, only configuration<br>
>>>>>> files.<br>
>>>>>><br>
>>>>>> Lvm metadata simply sits somewhere at the beginning of your<br>
>>>>>> iscsi-shared disk, like a partition table. So it is on the storage<br>
>>>>>> that is accessed by all your hosts, and no distribution is necessary.<br>
>>>>>> Nebula frontend simply issues lvcreate, lvchange, etc, on this shared<br>
>>>>>> disk and those commands will manipulate the metadata.<br>
>>>>>><br>
>>>>>> It is really LVM's internal business, many layers below opennebula.<br>
>>>>>> All you have to make sure that you don't run these commands<br>
>>>>>> concurrently from multiple hosts on the same iscsi-attached disk,<br>
>>>>>> because then they could interfere with each other. This setting is<br>
>>>>>> what you have to indicate in /etc/lvm on the server hosts.<br>
>>>>>><br>
>>>>>> Cheers<br>
>>>>>> Mihály<br>
>>>>>><br>
>>>>>> On 21 January 2013 18:37, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com">milos.kozak@lejmr.com</a>> wrote:<br>
>>>>>>><br>
>>>>>>> Thank you. does it mean, that I can distribute metadata files located<br>
>>>>>>> in<br>
>>>>>>> /etc/lvm on frontend onto other hosts and these hosts will see my<br>
>>>>>>> logical<br>
>>>>>>> volumes? Is there any code in nebula which would provide it? Or I<br>
>>>>>>> need<br>
>>>>>>> to<br>
>>>>>>> update DS scripts to update/distribute LVM metadata among servers?<br>
>>>>>>><br>
>>>>>>> Thanks, Milos<br>
>>>>>>><br>
>>>>>>> Dne 21.1.2013 18:29, Mihály Héder napsal(a):<br>
>>>>>>><br>
>>>>>>>> Hi,<br>
>>>>>>>><br>
>>>>>>>> lvm metadata[1] is simply stored on the disk. In the setup we are<br>
>>>>>>>> discussing this happens to be a shared virtual disk on the storage,<br>
>>>>>>>> so any other hosts that are attaching the same virtual disk should<br>
>>>>>>>> see<br>
>>>>>>>> the changes as they happen, provided that they re-read the disk.<br>
>>>>>>>> This<br>
>>>>>>>> re-reading step is what you can trigger with lvscan, but nowadays<br>
>>>>>>>> that<br>
>>>>>>>> seems to be unnecessary. For us it works with Centos 6.3 so I guess<br>
>>>>>>>> Sc<br>
>>>>>>>> Linux should be fine as well.<br>
>>>>>>>><br>
>>>>>>>> Cheers<br>
>>>>>>>> Mihály<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> [1]<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> <a href="http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html" target="_blank">http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html</a><br>
>>>>>>>><br>
>>>>>>>> On 21 January 2013 12:53, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com">milos.kozak@lejmr.com</a>><br>
>>>>>>>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> Hi,<br>
>>>>>>>>> thank you for great answer. As I wrote my objective is to avoid as<br>
>>>>>>>>> much<br>
>>>>>>>>> of<br>
>>>>>>>>> clustering sw (pacemaker,..) as possible, so clvm is one of these<br>
>>>>>>>>> things<br>
>>>>>>>>> I<br>
>>>>>>>>> feel bad about them in my configuration.. Therefore I would rather<br>
>>>>>>>>> let<br>
>>>>>>>>> nebula manage LVM metadata in the first place as I you wrote. Only<br>
>>>>>>>>> one<br>
>>>>>>>>> last<br>
>>>>>>>>> thing I dont understand is a way nebula distributes LVM metadata?<br>
>>>>>>>>><br>
>>>>>>>>> Is kernel in Scientific Linux 6.3 new enought to LVM issue you<br>
>>>>>>>>> mentioned?<br>
>>>>>>>>><br>
>>>>>>>>> Thanks Milos<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> Dne 21.1.2013 12:34, Mihály Héder napsal(a):<br>
>>>>>>>>><br>
>>>>>>>>>> Hi!<br>
>>>>>>>>>><br>
>>>>>>>>>> Last time we could test an Equalogic it did not have option for<br>
>>>>>>>>>> create/configure Virtual Disks inside in it by an API, so I think<br>
>>>>>>>>>> the<br>
>>>>>>>>>> iSCSI driver is not an alternative, as it would require a<br>
>>>>>>>>>> configuration step per virtual machine on the storage.<br>
>>>>>>>>>><br>
>>>>>>>>>> However, you can use your storage just fine in a shared LVM<br>
>>>>>>>>>> scenario.<br>
>>>>>>>>>> You need to consider two different things:<br>
>>>>>>>>>> -the LVM metadata, and the actual VM data on the partitions. It is<br>
>>>>>>>>>> true, that the concurrent modification of the metadata should be<br>
>>>>>>>>>> avoided as in theory it can damage the whole virtual group. You<br>
>>>>>>>>>> could<br>
>>>>>>>>>> use clvm which avoids that by clustered locking, and then every<br>
>>>>>>>>>> participating machine can safely create/modify/delete LV-s.<br>
>>>>>>>>>> However,<br>
>>>>>>>>>> in a nebula setup this is not necessary in every case: you can<br>
>>>>>>>>>> make<br>
>>>>>>>>>> the LVM metadata read only on your host servers, and let only the<br>
>>>>>>>>>> frontend modify it. Then it can use local locking that does not<br>
>>>>>>>>>> require clvm.<br>
>>>>>>>>>> -of course the host servers can write the data inside the<br>
>>>>>>>>>> partitions<br>
>>>>>>>>>> regardless that the metadata is read-only for them. It should work<br>
>>>>>>>>>> just fine as long as you don't start two VMs for one partition.<br>
>>>>>>>>>><br>
>>>>>>>>>> We are running this setup with a dual controller Dell MD3600<br>
>>>>>>>>>> storage<br>
>>>>>>>>>> without issues so far. Before that, we used to do the same with<br>
>>>>>>>>>> XEN<br>
>>>>>>>>>> machines for years on an older EMC (that was before nebula). Now<br>
>>>>>>>>>> with<br>
>>>>>>>>>> nebula we have been using a home-grown module for doing that,<br>
>>>>>>>>>> which<br>
>>>>>>>>>> I<br>
>>>>>>>>>> can send you any time - we plan to submit that as a feature<br>
>>>>>>>>>> enhancement anyway. Also, there seems to be a similar shared LVM<br>
>>>>>>>>>> module in the nebula upstream which we could not get to work yet,<br>
>>>>>>>>>> but<br>
>>>>>>>>>> did not try much.<br>
>>>>>>>>>><br>
>>>>>>>>>> The plus side of this setup is that you can make live migration<br>
>>>>>>>>>> work<br>
>>>>>>>>>> nicely. There are two points to consider however: once you set the<br>
>>>>>>>>>> LVM<br>
>>>>>>>>>> metadata read-only you wont be able to modify the local LVMs in<br>
>>>>>>>>>> your<br>
>>>>>>>>>> servers, if there are any. Also, in older kernels, when you<br>
>>>>>>>>>> modified<br>
>>>>>>>>>> the LVM on one machine the others did not get notified about the<br>
>>>>>>>>>> changes, so you had to issue an lvs command. However in new<br>
>>>>>>>>>> kernels<br>
>>>>>>>>>> this issue seems to be solved, the LVs get instantly updated. I<br>
>>>>>>>>>> don't<br>
>>>>>>>>>> know when and what exactly changed though.<br>
>>>>>>>>>><br>
>>>>>>>>>> Cheers<br>
>>>>>>>>>> Mihály Héder<br>
>>>>>>>>>> MTA SZTAKI ITAK<br>
>>>>>>>>>><br>
>>>>>>>>>> On 18 January 2013 08:57, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com">milos.kozak@lejmr.com</a>><br>
>>>>>>>>>> wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi, I am setting up a small installation of opennebula with<br>
>>>>>>>>>>> sharedstorage<br>
>>>>>>>>>>> using iSCSI. THe storage is Equilogic EMC with two controllers.<br>
>>>>>>>>>>> Nowadays<br>
>>>>>>>>>>> we<br>
>>>>>>>>>>> have only two host servers so we use backed direct connection<br>
>>>>>>>>>>> between<br>
>>>>>>>>>>> storage and each server, see attachment. For this purpose we set<br>
>>>>>>>>>>> up<br>
>>>>>>>>>>> dm-multipath. Cause in the future we want to add other servers<br>
>>>>>>>>>>> and<br>
>>>>>>>>>>> some<br>
>>>>>>>>>>> other technology will be necessary in the network segment.<br>
>>>>>>>>>>> Thesedays<br>
>>>>>>>>>>> we<br>
>>>>>>>>>>> try<br>
>>>>>>>>>>> to make it as same as possible with future topology from<br>
>>>>>>>>>>> protocols<br>
>>>>>>>>>>> point<br>
>>>>>>>>>>> of<br>
>>>>>>>>>>> view.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> My question is related to the way how to define datastore, which<br>
>>>>>>>>>>> driver<br>
>>>>>>>>>>> and<br>
>>>>>>>>>>> TM is the best and which?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> My primal objective is to avoid GFS2 or any other cluster<br>
>>>>>>>>>>> filesystem<br>
>>>>>>>>>>> I<br>
>>>>>>>>>>> would<br>
>>>>>>>>>>> prefer to keep datastore as block devices. Only option I see is<br>
>>>>>>>>>>> to<br>
>>>>>>>>>>> use<br>
>>>>>>>>>>> LVM<br>
>>>>>>>>>>> but I worry about concurent writes isn't it a problem? I was<br>
>>>>>>>>>>> googling<br>
>>>>>>>>>>> a<br>
>>>>>>>>>>> bit<br>
>>>>>>>>>>> and I found I would need to set up clvm - is it really necessary?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Or is better to use iSCSI driver, drop the dm-multipath and hope?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Thanks, Milos<br>
>>>>>>>>>>><br>
>>>>>>>>>>> _______________________________________________<br>
>>>>>>>>>>> Users mailing list<br>
>>>>>>>>>>> <a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
>>>>>>>>>>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>>>>>>>>>>><br>
>>>>>>> _______________________________________________<br>
>>>>>>> Users mailing list<br>
>>>>>>> <a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
>>>>>>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>>>>><br>
>>>>><br>
><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</div></div></blockquote></div><br>