[one-users] infiniband

[Jaime Melis]

Hello Shankhadeep,

that sounds really nice. Would you be interested in contributing your code
to OpenNebula's ecosystem and/or publishing an entry in opennebula's blog?

Regards,
Jaime

On Sun, May 6, 2012 at 5:39 AM, Shankhadeep Shome <shank15217 at gmail.com>wrote:

>
>
> On Sat, May 5, 2012 at 11:38 PM, Shankhadeep Shome <shank15217 at gmail.com>wrote:
>
>> Hi Chris
>>
>> We have a solution we are using on Oracle Exalogic hardware (we are using
>> the bare metal boxes and gateway switches). I think I understand the
>> requirement, IB accessible storage from VMs is possible however its a bit
>> convoluted. Our solution was to create a one-to-one NAT from the VMs to the
>> IB IPoIB network. This allows the VMs to mount storage natively over the IB
>> network. The performance is pretty good, about 9Gbps per node with 64k MTU
>> sizes.  We created an open nebula driver for this and I'm happy to share it
>> with the community. The driver handles VM migrations by enabling/diabling
>> ip aliases on the host and can also be used to manipulate iptable rules on
>> source and destination when open nebula moves VMs around.
>>
>> Shankhadeep
>>
>>
>> On Sat, May 5, 2012 at 5:42 PM, Chris Barry <cbarry at rjmetrics.com> wrote:
>>
>>> Reading more, I see that the available methods for block store are
>>> iSCSI, and that the LUNS are attached to the host. From there, a symlink
>>> tree exposes the target to the guest in a predictable way on every host.
>>>
>>> So then to modify my question a bit, are all LUNs attached to all hosts
>>> simultaneously? Or does the attachment only happen when a migration is to
>>> occur? Also, is the LUN put into a read-only mode or something during
>>> migration on the original host to protect the data? Or, must a clustering
>>> filesystem be employed?
>>>
>>> Guess I have a lot to read :)
>>>
>>> Thanks
>>> -C
>>>
>>>
>>> On Sat, May 5, 2012 at 5:23 PM, Chris Barry <cbarry at rjmetrics.com>wrote:
>>>
>>>> Hi Gubda,
>>>>
>>>> Thank you for replying. My goal was to use a single shared iso image to
>>>> boot from, and use an in-memory minimal linux on each node that had no
>>>> 'disk' at all, then to mount logical data volume(s) from a centralized
>>>> storage system. Perhaps that is outside the scope of opennebula's design
>>>> goals, and may not be possible  - I'm just now investigating it.
>>>>
>>>> I see you are using NFS, but my desire is to use block storage instead,
>>>> ideally LVM, and not incur the performance penalties of IPoIB. It does
>>>> sound simple though, and that's always good. Do you have any performance
>>>> data on that setup in terms of IOPs and/or MB/s write speeds? It does sound
>>>> interesting.
>>>>
>>>> Thanks again,
>>>> -C
>>>>
>>>>
>>>> On Sat, May 5, 2012 at 4:01 PM, Guba Sándor <gubasanyi at gmail.com>wrote:
>>>>
>>>>>  Hi
>>>>>
>>>>> I'm using InfiniBand to make shared storage. My setup is simple: the
>>>>> opennebula installdir is shared on NFS with the worker nodes.
>>>>>
>>>>> - I don't understand what you mean on shared image. There will be a
>>>>> copy (or symlink if the image is persistent) on the NFS host and that is
>>>>> that the hypervisor will use over the network. Live migrate is available
>>>>> beacuse you don't move the image only another host will use it from the
>>>>> same spot. With my linux images I have about 30s delay when livemigrate.
>>>>> You can use qcow2 driver for shared image.
>>>>>
>>>>> - I don't understant exactly what you mean on "guest unaware". If you
>>>>> mean storage - host connection it has nothing to do with nebula. You can
>>>>> use any shared filesystem. NFS uses IPoIB connection.
>>>>>
>>>>> 2012-05-05 19:01 keltezéssel, Chris Barry írta:
>>>>>
>>>>> Greetings,
>>>>>
>>>>> I'm interested in hearing user accounts about using infiniband as the
>>>>> storage interconnect with OpenNebula if anyone has any thoughts to share.
>>>>> Specifically about:
>>>>> * using a shared image and live migrating it (e.g. no copying of
>>>>> images).
>>>>> * is the guest unaware of the infiniband or is it running IB drivers?
>>>>> * does the host expose the volumes to the guest, or does the guest
>>>>> connect directly?
>>>>> * I'd like to avoid iSCSI over IPoIB if possible.
>>>>> * clustering filesystem/LVM requirements.
>>>>> * file-based vdisk or logical volume usage?
>>>>>
>>>>> Anything, an experiences at all will be helpful.
>>>>>
>>>>> Thanks
>>>>> Christopher
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing listUsers at lists.opennebula.orghttp://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opennebula.org
>>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>>
>>>>>
>>>>
>>>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opennebula.org
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>>
>>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>


-- 
Jaime Melis
Project Engineer
OpenNebula - The Open Source Toolkit for Cloud Computing
www.OpenNebula.org | jmelis at opennebula.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120507/da8dd9d0/attachment-0003.htm>