Hi,<br><br>We are running it without CLVM.<br>If you examine the ONE/lvm driver (the tm/clone script for example), you can see, that the lvcreate command runs on the destination host. In the shared LVM driver, all the LVM commands are running on the frontend, hence there is no possibility of parralel changes (assuming that you are using only 1 frontend), because local locking is in effect on the frontend.<br clear="all">
<div><br>The other thing with the ONE/lvm driver is that it makes a snapshot in the clone script, while our driver makes a new clone LV. I tried to use the original LVM driver, and every time, I deployed a new VM, I got this error message:<br>
<pre>lv-one-50 must be active exclusively to create snapshot<br></pre>If you (or everyone else) knows, how to avoid this error, please let me know.<br>Besides that snapshots are much slower in write operations (as far as I know).<br>
<br>Hope this helps!<br>--<br>Cheers,<br>tmarlok<br></div>
<br><br><div class="gmail_quote">On Wed, Jan 30, 2013 at 1:37 PM, Miloš Kozák <span dir="ltr"><<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
Hi, thank you. I checked source codes and I found it is very similar
to LVM TM/Datastore drivers which is facilitated in ONE already only
you added lvchange -ay DEV. Do you run CLVM along that or not? <br>
<br>
I worry about parallel changes of LVM metadata which might destroy
them. From sequential behaviour it is probably not an issues can you
prove it to me? Or is it highly dangerous to run lvm_shared without
CLVM?<br>
<br>
Thanks, Milos<br>
<br>
<br>
<div>Dne 30.1.2013 10:09, Marlok Tamás
napsal(a):<br>
</div><div><div class="h5">
<blockquote type="cite">Hi,<br>
<br>
We have a custom datastore, and transfer manager driver, which
runs the lvchange command when it is needed.<br>
In order to work, you have to enable it in oned.conf.<br>
<br>
for example:<br>
<span style="font-family:courier new,monospace"><br>
DATASTORE_MAD = [<br>
executable = "one_datastore",<br>
arguments = "-t 10 -d fs,vmware,iscsi,lvm,shared_lvm"]<br>
<br>
TM_MAD = [<br>
executable = "one_tm",<br>
arguments = "-t 10 -d
dummy,lvm,shared,qcow2,ssh,vmware,iscsi,shared_lvm" ]</span><br clear="all">
<div><br>
After that, you can create a datastore, with the shared_lvm tm
and datastore driver.<br>
<br>
The only limitation is that you can't live migrate VM-s. We have
a working solution for that as well, but it is still untested.I
can send you that too, if you want to help us testing it.<br>
<br>
Anyway, here are the drivers, feel free to use or modify it.<br>
<a href="https://dl.dropbox.com/u/140123/shared_lvm.tar.gz" target="_blank">https://dl.dropbox.com/u/140123/shared_lvm.tar.gz</a><br>
<br>
--<br>
Cheers,<br>
Marlok Tamas<br>
MTA Sztaki<br>
<br>
</div>
<br>
<br>
<div class="gmail_quote">On Thu, Jan 24, 2013 at 11:32 PM, Mihály
Héder <span dir="ltr"><<a href="mailto:mihaly.heder@sztaki.mta.hu" target="_blank">mihaly.heder@sztaki.mta.hu</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi,<br>
<br>
Well, if you can run the lvs or lvscan on at least one server<br>
successfully, then the metadata is probably fine.<br>
We had similar issues before we learned how to exclude
unnecessary<br>
block devices in the lvm config.<br>
<br>
The thing is that lvscan and lvs will try to check _every_
potential<br>
block device by default for LVM partitions. If you are lucky,
this is<br>
only annoying, because it will throw 'can't read /dev/sdX' or
similar<br>
messages. However, if you are using dm-multipath, you will
have one<br>
device for each path, like /dev/sdr _plus_ the aggregated
device with<br>
the name you have configured in multipath.conf
(/dev/mapper/yourname)<br>
what you actually need. LVM did not quite understand this
situation<br>
and got stuck on the individual path devices, so we have
configured to<br>
look for lvm only on the right place. In man page of lvm.conf
look for<br>
the devices / scan and filter options. Also there are quite
good<br>
examples in the comments there.<br>
<br>
Also, there could be a much simpler explanation to the issue:<br>
something with the iSCSI connection or multipath that are one
layer<br>
below.<br>
<br>
I hope this helps.<br>
<br>
Cheers<br>
Mihály<br>
<br>
On 24 January 2013 23:18, Miloš Kozák <<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>>
wrote:<br>
> Hi, thank you. I tried to update TM ln script, which
works but it is not<br>
> clean solution. So I will try to write hook code and then
we can discuss it.<br>
><br>
> I deployed a few VM and now on the other server lvs
command freezes. I have<br>
> not set up clvm, do you think it could be caused by lvm
metadata corruption?<br>
> The thing is I can not longer start a VM on the other
server.<br>
><br>
> Miloš<br>
><br>
> Dne 24.1.2013 23:10, Mihály Héder napsal(a):<br>
<div>
<div>><br>
>> Hi!<br>
>><br>
>> We solve this problem via hooks that are
activating the LV-s for us<br>
>> when we start/migrate a VM. Unfortunately I will
be out of office<br>
>> until early next week but then I will consult
with my colleague who<br>
>> did the actual coding of this part and we will
share the code.<br>
>><br>
>> Cheers<br>
>> Mihály<br>
>><br>
>> On 24 January 2013 20:15, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>>
wrote:<br>
>>><br>
>>> Hi, I have just set it up having two hosts
with shared blockdevice. On<br>
>>> top<br>
>>> of that LVM, as discussed earlier. Triggering
lvs I can see all logical<br>
>>> volumes. When I create a new LV on the other
server, I can see the LV<br>
>>> being<br>
>>> inactive, so I have to run lvchange -ay VG/LV
enable it then this LV can<br>
>>> be<br>
>>> used for VM..<br>
>>><br>
>>> Is there any trick howto auto enable newly
created LV on every host?<br>
>>><br>
>>> Thanks Milos<br>
>>><br>
>>> Dne 22.1.2013 18:22, Mihály Héder napsal(a):<br>
>>><br>
>>>> Hi!<br>
>>>><br>
>>>> You need to look at locking_type in the
lvm.conf manual [1]. The<br>
>>>> default - locking in a local directory -
is ok for the frontend, and<br>
>>>> type 4 is read-only. However, you should
not forget that this only<br>
>>>> prevents damaging thing by the lvm
commands. If you start to write<br>
>>>> zeros to your disk with the dd command
for example, that will kill<br>
>>>> your partition regardless the lvm
setting. So this is against user or<br>
>>>> middleware errors mainly, not against
malicious attacks.<br>
>>>><br>
>>>> Cheers<br>
>>>> Mihály Héder<br>
>>>> MTA SZTAKI<br>
>>>><br>
>>>> [1] <a href="http://linux.die.net/man/5/lvm.conf" target="_blank">http://linux.die.net/man/5/lvm.conf</a><br>
>>>><br>
>>>> On 21 January 2013 18:58, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>>
wrote:<br>
>>>>><br>
>>>>> Oh snap, that sounds great I didn't
know about that.. it makes all<br>
>>>>> easier.<br>
>>>>> In this scenario only frontend can
work with LVM, so no issues of<br>
>>>>> concurrent<br>
>>>>> change. Only one last think to make
it really safe against that. Is<br>
>>>>> there<br>
>>>>> any way to suppress LVM changes from
hosts, make it read only? And let<br>
>>>>> it<br>
>>>>> RW<br>
>>>>> at frontend?<br>
>>>>><br>
>>>>> Thanks<br>
>>>>><br>
>>>>><br>
>>>>> Dne 21.1.2013 18:50, Mihály Héder
napsal(a):<br>
>>>>><br>
>>>>>> Hi,<br>
>>>>>><br>
>>>>>> no, you don't have to do any of
that. Also, nebula doesn't have to<br>
>>>>>> care about LVM metadata at all
and therefore there is no corresponding<br>
>>>>>> function in it. At /etc/lvm there
is no metadata, only configuration<br>
>>>>>> files.<br>
>>>>>><br>
>>>>>> Lvm metadata simply sits
somewhere at the beginning of your<br>
>>>>>> iscsi-shared disk, like a
partition table. So it is on the storage<br>
>>>>>> that is accessed by all your
hosts, and no distribution is necessary.<br>
>>>>>> Nebula frontend simply issues
lvcreate, lvchange, etc, on this shared<br>
>>>>>> disk and those commands will
manipulate the metadata.<br>
>>>>>><br>
>>>>>> It is really LVM's internal
business, many layers below opennebula.<br>
>>>>>> All you have to make sure that
you don't run these commands<br>
>>>>>> concurrently from multiple hosts
on the same iscsi-attached disk,<br>
>>>>>> because then they could interfere
with each other. This setting is<br>
>>>>>> what you have to indicate in
/etc/lvm on the server hosts.<br>
>>>>>><br>
>>>>>> Cheers<br>
>>>>>> Mihály<br>
>>>>>><br>
>>>>>> On 21 January 2013 18:37, Miloš
Kozák<<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>>
wrote:<br>
>>>>>>><br>
>>>>>>> Thank you. does it mean, that
I can distribute metadata files located<br>
>>>>>>> in<br>
>>>>>>> /etc/lvm on frontend onto
other hosts and these hosts will see my<br>
>>>>>>> logical<br>
>>>>>>> volumes? Is there any code in
nebula which would provide it? Or I<br>
>>>>>>> need<br>
>>>>>>> to<br>
>>>>>>> update DS scripts to
update/distribute LVM metadata among servers?<br>
>>>>>>><br>
>>>>>>> Thanks, Milos<br>
>>>>>>><br>
>>>>>>> Dne 21.1.2013 18:29, Mihály
Héder napsal(a):<br>
>>>>>>><br>
>>>>>>>> Hi,<br>
>>>>>>>><br>
>>>>>>>> lvm metadata[1] is simply
stored on the disk. In the setup we are<br>
>>>>>>>> discussing this happens
to be a shared virtual disk on the storage,<br>
>>>>>>>> so any other hosts that
are attaching the same virtual disk should<br>
>>>>>>>> see<br>
>>>>>>>> the changes as they
happen, provided that they re-read the disk.<br>
>>>>>>>> This<br>
>>>>>>>> re-reading step is what
you can trigger with lvscan, but nowadays<br>
>>>>>>>> that<br>
>>>>>>>> seems to be unnecessary.
For us it works with Centos 6.3 so I guess<br>
>>>>>>>> Sc<br>
>>>>>>>> Linux should be fine as
well.<br>
>>>>>>>><br>
>>>>>>>> Cheers<br>
>>>>>>>> Mihály<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> [1]<br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>><br>
>>>>>>>> <a href="http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html" target="_blank">http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html</a><br>
>>>>>>>><br>
>>>>>>>> On 21 January 2013 12:53,
Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>><br>
>>>>>>>> wrote:<br>
>>>>>>>>><br>
>>>>>>>>> Hi,<br>
>>>>>>>>> thank you for great
answer. As I wrote my objective is to avoid as<br>
>>>>>>>>> much<br>
>>>>>>>>> of<br>
>>>>>>>>> clustering sw
(pacemaker,..) as possible, so clvm is one of these<br>
>>>>>>>>> things<br>
>>>>>>>>> I<br>
>>>>>>>>> feel bad about them
in my configuration.. Therefore I would rather<br>
>>>>>>>>> let<br>
>>>>>>>>> nebula manage LVM
metadata in the first place as I you wrote. Only<br>
>>>>>>>>> one<br>
>>>>>>>>> last<br>
>>>>>>>>> thing I dont
understand is a way nebula distributes LVM metadata?<br>
>>>>>>>>><br>
>>>>>>>>> Is kernel in
Scientific Linux 6.3 new enought to LVM issue you<br>
>>>>>>>>> mentioned?<br>
>>>>>>>>><br>
>>>>>>>>> Thanks Milos<br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>><br>
>>>>>>>>> Dne 21.1.2013 12:34,
Mihály Héder napsal(a):<br>
>>>>>>>>><br>
>>>>>>>>>> Hi!<br>
>>>>>>>>>><br>
>>>>>>>>>> Last time we
could test an Equalogic it did not have option for<br>
>>>>>>>>>> create/configure
Virtual Disks inside in it by an API, so I think<br>
>>>>>>>>>> the<br>
>>>>>>>>>> iSCSI driver is
not an alternative, as it would require a<br>
>>>>>>>>>> configuration
step per virtual machine on the storage.<br>
>>>>>>>>>><br>
>>>>>>>>>> However, you can
use your storage just fine in a shared LVM<br>
>>>>>>>>>> scenario.<br>
>>>>>>>>>> You need to
consider two different things:<br>
>>>>>>>>>> -the LVM
metadata, and the actual VM data on the partitions. It is<br>
>>>>>>>>>> true, that the
concurrent modification of the metadata should be<br>
>>>>>>>>>> avoided as in
theory it can damage the whole virtual group. You<br>
>>>>>>>>>> could<br>
>>>>>>>>>> use clvm which
avoids that by clustered locking, and then every<br>
>>>>>>>>>> participating
machine can safely create/modify/delete LV-s.<br>
>>>>>>>>>> However,<br>
>>>>>>>>>> in a nebula setup
this is not necessary in every case: you can<br>
>>>>>>>>>> make<br>
>>>>>>>>>> the LVM metadata
read only on your host servers, and let only the<br>
>>>>>>>>>> frontend modify
it. Then it can use local locking that does not<br>
>>>>>>>>>> require clvm.<br>
>>>>>>>>>> -of course the
host servers can write the data inside the<br>
>>>>>>>>>> partitions<br>
>>>>>>>>>> regardless that
the metadata is read-only for them. It should work<br>
>>>>>>>>>> just fine as long
as you don't start two VMs for one partition.<br>
>>>>>>>>>><br>
>>>>>>>>>> We are running
this setup with a dual controller Dell MD3600<br>
>>>>>>>>>> storage<br>
>>>>>>>>>> without issues so
far. Before that, we used to do the same with<br>
>>>>>>>>>> XEN<br>
>>>>>>>>>> machines for
years on an older EMC (that was before nebula). Now<br>
>>>>>>>>>> with<br>
>>>>>>>>>> nebula we have
been using a home-grown module for doing that,<br>
>>>>>>>>>> which<br>
>>>>>>>>>> I<br>
>>>>>>>>>> can send you any
time - we plan to submit that as a feature<br>
>>>>>>>>>> enhancement
anyway. Also, there seems to be a similar shared LVM<br>
>>>>>>>>>> module in the
nebula upstream which we could not get to work yet,<br>
>>>>>>>>>> but<br>
>>>>>>>>>> did not try much.<br>
>>>>>>>>>><br>
>>>>>>>>>> The plus side of
this setup is that you can make live migration<br>
>>>>>>>>>> work<br>
>>>>>>>>>> nicely. There are
two points to consider however: once you set the<br>
>>>>>>>>>> LVM<br>
>>>>>>>>>> metadata
read-only you wont be able to modify the local LVMs in<br>
>>>>>>>>>> your<br>
>>>>>>>>>> servers, if there
are any. Also, in older kernels, when you<br>
>>>>>>>>>> modified<br>
>>>>>>>>>> the LVM on one
machine the others did not get notified about the<br>
>>>>>>>>>> changes, so you
had to issue an lvs command. However in new<br>
>>>>>>>>>> kernels<br>
>>>>>>>>>> this issue seems
to be solved, the LVs get instantly updated. I<br>
>>>>>>>>>> don't<br>
>>>>>>>>>> know when and
what exactly changed though.<br>
>>>>>>>>>><br>
>>>>>>>>>> Cheers<br>
>>>>>>>>>> Mihály Héder<br>
>>>>>>>>>> MTA SZTAKI ITAK<br>
>>>>>>>>>><br>
>>>>>>>>>> On 18 January
2013 08:57, Miloš Kozák<<a href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>><br>
>>>>>>>>>> wrote:<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Hi, I am
setting up a small installation of opennebula with<br>
>>>>>>>>>>> sharedstorage<br>
>>>>>>>>>>> using iSCSI.
THe storage is Equilogic EMC with two controllers.<br>
>>>>>>>>>>> Nowadays<br>
>>>>>>>>>>> we<br>
>>>>>>>>>>> have only two
host servers so we use backed direct connection<br>
>>>>>>>>>>> between<br>
>>>>>>>>>>> storage and
each server, see attachment. For this purpose we set<br>
>>>>>>>>>>> up<br>
>>>>>>>>>>> dm-multipath.
Cause in the future we want to add other servers<br>
>>>>>>>>>>> and<br>
>>>>>>>>>>> some<br>
>>>>>>>>>>> other
technology will be necessary in the network segment.<br>
>>>>>>>>>>> Thesedays<br>
>>>>>>>>>>> we<br>
>>>>>>>>>>> try<br>
>>>>>>>>>>> to make it as
same as possible with future topology from<br>
>>>>>>>>>>> protocols<br>
>>>>>>>>>>> point<br>
>>>>>>>>>>> of<br>
>>>>>>>>>>> view.<br>
>>>>>>>>>>><br>
>>>>>>>>>>> My question
is related to the way how to define datastore, which<br>
>>>>>>>>>>> driver<br>
>>>>>>>>>>> and<br>
>>>>>>>>>>> TM is the
best and which?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> My primal
objective is to avoid GFS2 or any other cluster<br>
>>>>>>>>>>> filesystem<br>
>>>>>>>>>>> I<br>
>>>>>>>>>>> would<br>
>>>>>>>>>>> prefer to
keep datastore as block devices. Only option I see is<br>
>>>>>>>>>>> to<br>
>>>>>>>>>>> use<br>
>>>>>>>>>>> LVM<br>
>>>>>>>>>>> but I worry
about concurent writes isn't it a problem? I was<br>
>>>>>>>>>>> googling<br>
>>>>>>>>>>> a<br>
>>>>>>>>>>> bit<br>
>>>>>>>>>>> and I found I
would need to set up clvm - is it really necessary?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Or is better
to use iSCSI driver, drop the dm-multipath and hope?<br>
>>>>>>>>>>><br>
>>>>>>>>>>> Thanks, Milos<br>
>>>>>>>>>>><br>
>>>>>>>>>>>
_______________________________________________<br>
>>>>>>>>>>> Users mailing
list<br>
>>>>>>>>>>> <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
>>>>>>>>>>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>>>>>>>>>>><br>
>>>>>>>
_______________________________________________<br>
>>>>>>> Users mailing list<br>
>>>>>>> <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
>>>>>>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>>>>><br>
>>>>><br>
><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</blockquote>
<br>
</div></div></div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br>