<html>
  <head>
    <meta content="text/html; charset=UTF-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    Hi, it sounds interesting, I think I am going to give it a try. I
    still struggle whether to use or not CLVM. For how long you have
    been running it like that? Have you ever had any serious issues
    related to LVM?<br>
    <br>
    Thank you, Milos <br>
    <br>
    <br>
    <div class="moz-cite-prefix">Dne 30.1.2013 13:59, Marlok Tamás
      napsal(a):<br>
    </div>
    <blockquote
cite="mid:CANifAPJw+VA3=0SOOGAK3UyYEfuDmHaKObOXpTnrjyy-5DkGnQ@mail.gmail.com"
      type="cite">Hi,<br>
      <br>
      We are running it without CLVM.<br>
      If you examine the ONE/lvm driver (the tm/clone script for
      example), you can see, that the lvcreate command runs on the
      destination host. In the shared LVM driver, all the LVM commands
      are running on the frontend, hence there is no possibility of
      parralel changes (assuming that you are using only 1 frontend),
      because local locking is in effect on the frontend.<br clear="all">
      <div><br>
        The other thing with the ONE/lvm driver is that it makes a
        snapshot in the clone script, while our driver makes a new clone
        LV. I tried to use the original LVM driver, and every time, I
        deployed a new VM, I got this error message:<br>
        <pre>lv-one-50 must be active exclusively to create snapshot
</pre>
        If you (or everyone else) knows, how to avoid this error, please
        let me know.<br>
        Besides that snapshots are much slower in write operations (as
        far as I know).<br>
        <br>
        Hope this helps!<br>
        --<br>
        Cheers,<br>
        tmarlok<br>
      </div>
      <br>
      <br>
      <div class="gmail_quote">On Wed, Jan 30, 2013 at 1:37 PM, Miloš
        Kozák <span dir="ltr"><<a moz-do-not-send="true"
            href="mailto:milos.kozak@lejmr.com" target="_blank">milos.kozak@lejmr.com</a>></span>
        wrote:<br>
        <blockquote class="gmail_quote" style="margin:0 0 0
          .8ex;border-left:1px #ccc solid;padding-left:1ex">
          <div text="#000000" bgcolor="#FFFFFF"> Hi, thank you. I
            checked source codes and I found it is very similar to LVM
            TM/Datastore drivers which is facilitated in ONE already
            only you added lvchange -ay DEV. Do you run CLVM along that
            or not? <br>
            <br>
            I worry about parallel changes of LVM metadata which might
            destroy them. From sequential behaviour it is probably not
            an issues can you prove it to me? Or  is it highly dangerous
            to run lvm_shared without CLVM?<br>
            <br>
            Thanks, Milos<br>
            <br>
            <br>
            <div>Dne 30.1.2013 10:09, Marlok Tamás napsal(a):<br>
            </div>
            <div>
              <div class="h5">
                <blockquote type="cite">Hi,<br>
                  <br>
                  We have a custom datastore, and transfer manager
                  driver, which runs the lvchange command when it is
                  needed.<br>
                  In order to work, you have to enable it in oned.conf.<br>
                  <br>
                  for example:<br>
                  <span style="font-family:courier new,monospace"><br>
                    DATASTORE_MAD = [<br>
                        executable = "one_datastore",<br>
                        arguments  = "-t 10 -d
                    fs,vmware,iscsi,lvm,shared_lvm"]<br>
                    <br>
                    TM_MAD = [<br>
                        executable = "one_tm",<br>
                        arguments  = "-t 10 -d
                    dummy,lvm,shared,qcow2,ssh,vmware,iscsi,shared_lvm"
                    ]</span><br clear="all">
                  <div><br>
                    After that, you can create a datastore, with the
                    shared_lvm tm and datastore driver.<br>
                    <br>
                    The only limitation is that you can't live migrate
                    VM-s. We have a working solution for that as well,
                    but it is still untested.I can send you that too, if
                    you want to help us testing it.<br>
                    <br>
                    Anyway, here are the drivers, feel free to use or
                    modify it.<br>
                    <a moz-do-not-send="true"
                      href="https://dl.dropbox.com/u/140123/shared_lvm.tar.gz"
                      target="_blank">https://dl.dropbox.com/u/140123/shared_lvm.tar.gz</a><br>
                    <br>
                    --<br>
                    Cheers,<br>
                    Marlok Tamas<br>
                    MTA Sztaki<br>
                    <br>
                  </div>
                  <br>
                  <br>
                  <div class="gmail_quote">On Thu, Jan 24, 2013 at 11:32
                    PM, Mihály Héder <span dir="ltr"><<a
                        moz-do-not-send="true"
                        href="mailto:mihaly.heder@sztaki.mta.hu"
                        target="_blank">mihaly.heder@sztaki.mta.hu</a>></span>
                    wrote:<br>
                    <blockquote class="gmail_quote" style="margin:0 0 0
                      .8ex;border-left:1px #ccc solid;padding-left:1ex">
                      Hi,<br>
                      <br>
                      Well, if you can run the lvs or lvscan on at least
                      one server<br>
                      successfully, then the metadata is probably fine.<br>
                      We had similar issues before we learned how to
                      exclude unnecessary<br>
                      block devices in the lvm config.<br>
                      <br>
                      The thing is that lvscan and lvs will try to check
                      _every_ potential<br>
                      block device by default for LVM partitions. If you
                      are lucky, this is<br>
                      only annoying, because it will throw 'can't read
                      /dev/sdX' or similar<br>
                      messages. However, if you are using dm-multipath,
                      you will have one<br>
                      device for each path, like /dev/sdr _plus_ the
                      aggregated device with<br>
                      the name you have configured in multipath.conf
                      (/dev/mapper/yourname)<br>
                      what you actually need. LVM did not quite
                      understand this situation<br>
                      and got stuck on the individual path devices, so
                      we have configured to<br>
                      look for lvm only on the right place. In man page
                      of lvm.conf look for<br>
                      the devices / scan and filter options. Also there
                      are quite good<br>
                      examples in the comments there.<br>
                      <br>
                      Also, there could be a much simpler explanation to
                      the issue:<br>
                      something with the iSCSI connection or multipath
                      that are one layer<br>
                      below.<br>
                      <br>
                      I hope this helps.<br>
                      <br>
                      Cheers<br>
                      Mihály<br>
                      <br>
                      On 24 January 2013 23:18, Miloš Kozák <<a
                        moz-do-not-send="true"
                        href="mailto:milos.kozak@lejmr.com"
                        target="_blank">milos.kozak@lejmr.com</a>>
                      wrote:<br>
                      > Hi, thank you. I tried to update TM ln
                      script, which works but it is not<br>
                      > clean solution. So I will try to write hook
                      code and then we can discuss it.<br>
                      ><br>
                      > I deployed a few VM and now on the other
                      server lvs command freezes. I have<br>
                      > not set up clvm, do you think it could be
                      caused by lvm metadata corruption?<br>
                      > The thing is I can not longer start a VM on
                      the other server.<br>
                      ><br>
                      > Miloš<br>
                      ><br>
                      > Dne 24.1.2013 23:10, Mihály Héder napsal(a):<br>
                      <div>
                        <div>><br>
                          >> Hi!<br>
                          >><br>
                          >> We solve this problem via hooks that
                          are activating the LV-s for us<br>
                          >> when we start/migrate a VM.
                          Unfortunately I will be out of office<br>
                          >> until early next week but then I will
                          consult with my colleague who<br>
                          >> did the actual coding of this part
                          and we will share the code.<br>
                          >><br>
                          >> Cheers<br>
                          >> Mihály<br>
                          >><br>
                          >> On 24 January 2013 20:15, Miloš
                          Kozák<<a moz-do-not-send="true"
                            href="mailto:milos.kozak@lejmr.com"
                            target="_blank">milos.kozak@lejmr.com</a>>

                           wrote:<br>
                          >>><br>
                          >>> Hi, I have just set it up having
                          two hosts with shared blockdevice. On<br>
                          >>> top<br>
                          >>> of that LVM, as discussed
                          earlier. Triggering lvs I can see all logical<br>
                          >>> volumes. When I create a new LV
                           on the other server, I can see the LV<br>
                          >>> being<br>
                          >>> inactive, so I have to run
                          lvchange -ay VG/LV enable it then this LV can<br>
                          >>> be<br>
                          >>> used for VM..<br>
                          >>><br>
                          >>> Is there any trick howto auto
                          enable newly created LV on every host?<br>
                          >>><br>
                          >>> Thanks Milos<br>
                          >>><br>
                          >>> Dne 22.1.2013 18:22, Mihály Héder
                          napsal(a):<br>
                          >>><br>
                          >>>> Hi!<br>
                          >>>><br>
                          >>>> You need to look at
                          locking_type in the lvm.conf manual [1]. The<br>
                          >>>> default - locking in a local
                          directory - is ok for the frontend, and<br>
                          >>>> type 4 is read-only. However,
                          you should not forget that this only<br>
                          >>>> prevents damaging thing by
                          the lvm commands. If you start to write<br>
                          >>>> zeros to your disk with the
                          dd command for example, that will kill<br>
                          >>>> your partition regardless the
                          lvm setting. So this is against user or<br>
                          >>>> middleware errors mainly, not
                          against malicious attacks.<br>
                          >>>><br>
                          >>>> Cheers<br>
                          >>>> Mihály Héder<br>
                          >>>> MTA SZTAKI<br>
                          >>>><br>
                          >>>> [1] <a
                            moz-do-not-send="true"
                            href="http://linux.die.net/man/5/lvm.conf"
                            target="_blank">http://linux.die.net/man/5/lvm.conf</a><br>
                          >>>><br>
                          >>>> On 21 January 2013 18:58,
                          Miloš Kozák<<a moz-do-not-send="true"
                            href="mailto:milos.kozak@lejmr.com"
                            target="_blank">milos.kozak@lejmr.com</a>>

                            wrote:<br>
                          >>>>><br>
                          >>>>> Oh snap, that sounds
                          great I didn't know about that.. it makes all<br>
                          >>>>> easier.<br>
                          >>>>> In this scenario only
                          frontend can work with LVM, so no issues of<br>
                          >>>>> concurrent<br>
                          >>>>> change. Only one last
                          think to make it really safe against that. Is<br>
                          >>>>> there<br>
                          >>>>> any way to suppress LVM
                          changes from hosts, make it read only? And let<br>
                          >>>>> it<br>
                          >>>>> RW<br>
                          >>>>> at frontend?<br>
                          >>>>><br>
                          >>>>> Thanks<br>
                          >>>>><br>
                          >>>>><br>
                          >>>>> Dne 21.1.2013 18:50,
                          Mihály Héder napsal(a):<br>
                          >>>>><br>
                          >>>>>> Hi,<br>
                          >>>>>><br>
                          >>>>>> no, you don't have to
                          do any of that. Also, nebula doesn't have to<br>
                          >>>>>> care about LVM
                          metadata at all and therefore there is no
                          corresponding<br>
                          >>>>>> function in it. At
                          /etc/lvm there is no metadata, only
                          configuration<br>
                          >>>>>> files.<br>
                          >>>>>><br>
                          >>>>>> Lvm metadata simply
                          sits somewhere at the beginning of your<br>
                          >>>>>> iscsi-shared disk,
                          like a partition table. So it is on the
                          storage<br>
                          >>>>>> that is accessed by
                          all your hosts, and no distribution is
                          necessary.<br>
                          >>>>>> Nebula frontend
                          simply issues lvcreate, lvchange, etc, on this
                          shared<br>
                          >>>>>> disk and those
                          commands will manipulate the metadata.<br>
                          >>>>>><br>
                          >>>>>> It is really LVM's
                          internal business, many layers below
                          opennebula.<br>
                          >>>>>> All you have to make
                          sure that you don't run these commands<br>
                          >>>>>> concurrently  from
                          multiple hosts on the same iscsi-attached
                          disk,<br>
                          >>>>>> because then they
                          could interfere with each other. This setting
                          is<br>
                          >>>>>> what you have to
                          indicate in /etc/lvm on the server hosts.<br>
                          >>>>>><br>
                          >>>>>> Cheers<br>
                          >>>>>> Mihály<br>
                          >>>>>><br>
                          >>>>>> On 21 January 2013
                          18:37, Miloš Kozák<<a
                            moz-do-not-send="true"
                            href="mailto:milos.kozak@lejmr.com"
                            target="_blank">milos.kozak@lejmr.com</a>>

                            wrote:<br>
                          >>>>>>><br>
                          >>>>>>> Thank you. does
                          it mean, that I can distribute metadata files
                          located<br>
                          >>>>>>> in<br>
                          >>>>>>> /etc/lvm on
                          frontend onto other hosts and these hosts will
                          see my<br>
                          >>>>>>> logical<br>
                          >>>>>>> volumes? Is there
                          any code in nebula which would provide it? Or
                          I<br>
                          >>>>>>> need<br>
                          >>>>>>> to<br>
                          >>>>>>> update DS scripts
                          to update/distribute LVM metadata among
                          servers?<br>
                          >>>>>>><br>
                          >>>>>>> Thanks, Milos<br>
                          >>>>>>><br>
                          >>>>>>> Dne 21.1.2013
                          18:29, Mihály Héder napsal(a):<br>
                          >>>>>>><br>
                          >>>>>>>> Hi,<br>
                          >>>>>>>><br>
                          >>>>>>>> lvm
                          metadata[1] is simply stored on the disk. In
                          the setup we are<br>
                          >>>>>>>> discussing
                          this happens to be a  shared virtual disk on
                          the storage,<br>
                          >>>>>>>> so any other
                          hosts that are attaching the same virtual disk
                          should<br>
                          >>>>>>>> see<br>
                          >>>>>>>> the changes
                          as they happen, provided that they re-read the
                          disk.<br>
                          >>>>>>>> This<br>
                          >>>>>>>> re-reading
                          step is what you can trigger with lvscan, but
                          nowadays<br>
                          >>>>>>>> that<br>
                          >>>>>>>> seems to be
                          unnecessary. For us it works with Centos 6.3
                          so I guess<br>
                          >>>>>>>> Sc<br>
                          >>>>>>>> Linux should
                          be fine as well.<br>
                          >>>>>>>><br>
                          >>>>>>>> Cheers<br>
                          >>>>>>>> Mihály<br>
                          >>>>>>>><br>
                          >>>>>>>><br>
                          >>>>>>>> [1]<br>
                          >>>>>>>><br>
                          >>>>>>>><br>
                          >>>>>>>><br>
                          >>>>>>>> <a
                            moz-do-not-send="true"
href="http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html"
                            target="_blank">http://www.centos.org/docs/5/html/Cluster_Logical_Volume_Manager/lvm_metadata.html</a><br>
                          >>>>>>>><br>
                          >>>>>>>> On 21 January
                          2013 12:53, Miloš Kozák<<a
                            moz-do-not-send="true"
                            href="mailto:milos.kozak@lejmr.com"
                            target="_blank">milos.kozak@lejmr.com</a>><br>
                          >>>>>>>> wrote:<br>
                          >>>>>>>>><br>
                          >>>>>>>>> Hi,<br>
                          >>>>>>>>> thank you
                          for great answer. As I wrote my objective is
                          to avoid as<br>
                          >>>>>>>>> much<br>
                          >>>>>>>>> of<br>
                          >>>>>>>>>
                          clustering sw (pacemaker,..) as possible, so
                          clvm is one of these<br>
                          >>>>>>>>> things<br>
                          >>>>>>>>> I<br>
                          >>>>>>>>> feel bad
                          about them in my configuration.. Therefore I
                          would rather<br>
                          >>>>>>>>> let<br>
                          >>>>>>>>> nebula
                          manage LVM metadata in the first place as I
                          you wrote. Only<br>
                          >>>>>>>>> one<br>
                          >>>>>>>>> last<br>
                          >>>>>>>>> thing I
                          dont understand is a way nebula distributes
                          LVM metadata?<br>
                          >>>>>>>>><br>
                          >>>>>>>>> Is kernel
                          in Scientific Linux 6.3 new enought to LVM
                          issue you<br>
                          >>>>>>>>>
                          mentioned?<br>
                          >>>>>>>>><br>
                          >>>>>>>>> Thanks
                          Milos<br>
                          >>>>>>>>><br>
                          >>>>>>>>><br>
                          >>>>>>>>><br>
                          >>>>>>>>><br>
                          >>>>>>>>> Dne
                          21.1.2013 12:34, Mihály Héder napsal(a):<br>
                          >>>>>>>>><br>
                          >>>>>>>>>> Hi!<br>
                          >>>>>>>>>><br>
                          >>>>>>>>>> Last
                          time we could test an Equalogic it did not
                          have option for<br>
                          >>>>>>>>>>
                          create/configure Virtual Disks inside in it by
                          an API, so I think<br>
                          >>>>>>>>>> the<br>
                          >>>>>>>>>> iSCSI
                          driver is not an alternative, as it would
                          require a<br>
                          >>>>>>>>>>
                          configuration step per virtual machine on the
                          storage.<br>
                          >>>>>>>>>><br>
                          >>>>>>>>>>
                          However, you can use your storage just fine in
                          a shared LVM<br>
                          >>>>>>>>>>
                          scenario.<br>
                          >>>>>>>>>> You
                          need to consider two different things:<br>
                          >>>>>>>>>> -the
                          LVM metadata, and the actual VM data on the
                          partitions. It is<br>
                          >>>>>>>>>> true,
                          that the concurrent modification of the
                          metadata should be<br>
                          >>>>>>>>>>
                          avoided as in theory it can damage the whole
                          virtual group. You<br>
                          >>>>>>>>>> could<br>
                          >>>>>>>>>> use
                          clvm which avoids that by clustered locking,
                          and then every<br>
                          >>>>>>>>>>
                          participating machine can safely
                          create/modify/delete LV-s.<br>
                          >>>>>>>>>>
                          However,<br>
                          >>>>>>>>>> in a
                          nebula setup this is not necessary in every
                          case: you can<br>
                          >>>>>>>>>> make<br>
                          >>>>>>>>>> the
                          LVM metadata read only on your host servers,
                          and let only the<br>
                          >>>>>>>>>>
                          frontend modify it. Then it can use local
                          locking that does not<br>
                          >>>>>>>>>>
                          require clvm.<br>
                          >>>>>>>>>> -of
                          course the host servers can write the data
                          inside the<br>
                          >>>>>>>>>>
                          partitions<br>
                          >>>>>>>>>>
                          regardless that the metadata is read-only for
                          them. It should work<br>
                          >>>>>>>>>> just
                          fine as long as you don't start two VMs for
                          one partition.<br>
                          >>>>>>>>>><br>
                          >>>>>>>>>> We
                          are running this setup with a dual controller
                          Dell MD3600<br>
                          >>>>>>>>>>
                          storage<br>
                          >>>>>>>>>>
                          without issues so far. Before that, we used to
                          do the same with<br>
                          >>>>>>>>>> XEN<br>
                          >>>>>>>>>>
                          machines for years on an older EMC (that was
                          before nebula). Now<br>
                          >>>>>>>>>> with<br>
                          >>>>>>>>>>
                          nebula we have been using a home-grown module
                          for doing that,<br>
                          >>>>>>>>>> which<br>
                          >>>>>>>>>> I<br>
                          >>>>>>>>>> can
                          send you any time - we plan to submit that as
                          a feature<br>
                          >>>>>>>>>>
                          enhancement anyway. Also, there seems to be a
                          similar shared LVM<br>
                          >>>>>>>>>>
                          module in the nebula upstream which we could
                          not get to work yet,<br>
                          >>>>>>>>>> but<br>
                          >>>>>>>>>> did
                          not try much.<br>
                          >>>>>>>>>><br>
                          >>>>>>>>>> The
                          plus side of this setup is that you can make
                          live migration<br>
                          >>>>>>>>>> work<br>
                          >>>>>>>>>>
                          nicely. There are two points to consider
                          however: once you set the<br>
                          >>>>>>>>>> LVM<br>
                          >>>>>>>>>>
                          metadata read-only you wont be able to modify
                          the local LVMs in<br>
                          >>>>>>>>>> your<br>
                          >>>>>>>>>>
                          servers, if there are any. Also, in older
                          kernels, when you<br>
                          >>>>>>>>>>
                          modified<br>
                          >>>>>>>>>> the
                          LVM on one machine the others did not get
                          notified about the<br>
                          >>>>>>>>>>
                          changes, so you had to issue an lvs command.
                          However in new<br>
                          >>>>>>>>>>
                          kernels<br>
                          >>>>>>>>>> this
                          issue seems to be solved, the LVs get
                          instantly updated. I<br>
                          >>>>>>>>>> don't<br>
                          >>>>>>>>>> know
                          when and what exactly changed though.<br>
                          >>>>>>>>>><br>
                          >>>>>>>>>>
                          Cheers<br>
                          >>>>>>>>>>
                          Mihály Héder<br>
                          >>>>>>>>>> MTA
                          SZTAKI ITAK<br>
                          >>>>>>>>>><br>
                          >>>>>>>>>> On 18
                          January 2013 08:57, Miloš Kozák<<a
                            moz-do-not-send="true"
                            href="mailto:milos.kozak@lejmr.com"
                            target="_blank">milos.kozak@lejmr.com</a>><br>
                          >>>>>>>>>>
                          wrote:<br>
                          >>>>>>>>>>><br>
                          >>>>>>>>>>>
                          Hi, I am setting up a small installation of
                          opennebula with<br>
                          >>>>>>>>>>>
                          sharedstorage<br>
                          >>>>>>>>>>>
                          using iSCSI. THe storage is Equilogic EMC with
                          two controllers.<br>
                          >>>>>>>>>>>
                          Nowadays<br>
                          >>>>>>>>>>>
                          we<br>
                          >>>>>>>>>>>
                          have only two host servers so we use backed
                          direct connection<br>
                          >>>>>>>>>>>
                          between<br>
                          >>>>>>>>>>>
                          storage and each server, see attachment. For
                          this purpose we set<br>
                          >>>>>>>>>>>
                          up<br>
                          >>>>>>>>>>>
                          dm-multipath. Cause in the future we want to
                          add other servers<br>
                          >>>>>>>>>>>
                          and<br>
                          >>>>>>>>>>>
                          some<br>
                          >>>>>>>>>>>
                          other technology will be necessary in the
                          network segment.<br>
                          >>>>>>>>>>>
                          Thesedays<br>
                          >>>>>>>>>>>
                          we<br>
                          >>>>>>>>>>>
                          try<br>
                          >>>>>>>>>>>
                          to make it as same as possible with future
                          topology from<br>
                          >>>>>>>>>>>
                          protocols<br>
                          >>>>>>>>>>>
                          point<br>
                          >>>>>>>>>>>
                          of<br>
                          >>>>>>>>>>>
                          view.<br>
                          >>>>>>>>>>><br>
                          >>>>>>>>>>>
                          My question is related to the way how to
                          define datastore, which<br>
                          >>>>>>>>>>>
                          driver<br>
                          >>>>>>>>>>>
                          and<br>
                          >>>>>>>>>>>
                          TM is the best and which?<br>
                          >>>>>>>>>>><br>
                          >>>>>>>>>>>
                          My primal objective is to avoid GFS2 or any
                          other cluster<br>
                          >>>>>>>>>>>
                          filesystem<br>
                          >>>>>>>>>>> I<br>
                          >>>>>>>>>>>
                          would<br>
                          >>>>>>>>>>>
                          prefer to keep datastore as block devices.
                          Only option I see is<br>
                          >>>>>>>>>>>
                          to<br>
                          >>>>>>>>>>>
                          use<br>
                          >>>>>>>>>>>
                          LVM<br>
                          >>>>>>>>>>>
                          but I worry about concurent writes isn't it a
                          problem? I was<br>
                          >>>>>>>>>>>
                          googling<br>
                          >>>>>>>>>>> a<br>
                          >>>>>>>>>>>
                          bit<br>
                          >>>>>>>>>>>
                          and I found I would need to set up clvm - is
                          it really necessary?<br>
                          >>>>>>>>>>><br>
                          >>>>>>>>>>>
                          Or is better to use iSCSI driver, drop the
                          dm-multipath and hope?<br>
                          >>>>>>>>>>><br>
                          >>>>>>>>>>>
                          Thanks, Milos<br>
                          >>>>>>>>>>><br>
                          >>>>>>>>>>>
                          _______________________________________________<br>
                          >>>>>>>>>>>
                          Users mailing list<br>
                          >>>>>>>>>>> <a
                            moz-do-not-send="true"
                            href="mailto:Users@lists.opennebula.org"
                            target="_blank">Users@lists.opennebula.org</a><br>
                          >>>>>>>>>>> <a
                            moz-do-not-send="true"
                            href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org"
                            target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
                          >>>>>>>>>>><br>
                          >>>>>>>
                          _______________________________________________<br>
                          >>>>>>> Users mailing
                          list<br>
                          >>>>>>> <a
                            moz-do-not-send="true"
                            href="mailto:Users@lists.opennebula.org"
                            target="_blank">Users@lists.opennebula.org</a><br>
                          >>>>>>> <a
                            moz-do-not-send="true"
                            href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org"
                            target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
                          >>>>><br>
                          >>>>><br>
                          ><br>
_______________________________________________<br>
                          Users mailing list<br>
                          <a moz-do-not-send="true"
                            href="mailto:Users@lists.opennebula.org"
                            target="_blank">Users@lists.opennebula.org</a><br>
                          <a moz-do-not-send="true"
                            href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org"
                            target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
                        </div>
                      </div>
                    </blockquote>
                  </div>
                  <br>
                </blockquote>
                <br>
              </div>
            </div>
          </div>
          <br>
          _______________________________________________<br>
          Users mailing list<br>
          <a moz-do-not-send="true"
            href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
          <a moz-do-not-send="true"
            href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org"
            target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
          <br>
        </blockquote>
      </div>
      <br>
    </blockquote>
    <br>
  </body>
</html>