[one-users] infiniband

[Shankhadeep Shome]

On Thu, May 10, 2012 at 1:01 AM, Shankhadeep Shome <shank15217 at gmail.com>wrote:

> What we did was expose the IPoIB network directly to the VM using 1 to 1
> NAT. The VMs themselves can now connect to a iscsi or NFS source and log in
> directly. I am not sure what would be faster, iSER to the host which is
> exposed to the VMs as a raw device or direct attached network storage. I
> think either way you lose some performance. If you want to take the iSER
> option then your best solution is to present an iSER volume to the hosts
> and use LVM to present storage to the VMs via virtio-blk mechanism. There
> is another currently proprietary way using Mellanox's SRIOV drivers to
> present an Infiniband virtual function directly to the VMs however these
> drivers have not been released to OFED yet and they have their own
> limitations.
>
> We are able to hit around 9Gbps over the IPoIB link to the VMs via NAT
> using 64K frames with connected mode IPoIB with netperf and max out our
> local NAS appliance throughput. Our cards are ConnectX-2 at 40Gbps. IPoIB
> will max out around 12-14 Gbps with multiple streams even on these cards.
> IPoIB performance is greatly dependent on the kernel and driver versions,
> if you are running an older kernel and drivers the performance is much
> lower keep that in mind. We are using the latest MLNXOFED drivers with
> Linux Kernel 3.0 to get this level of performance with large
> transmit/receive offload enabled.
>
>
> On Wed, May 9, 2012 at 3:58 PM, Christopher Barry <cbarry at rjmetrics.com>wrote:
>
>> On Wed, 2012-05-09 at 00:16 -0400, Shankhadeep Shome wrote:
>> > Hi Jamie
>> >
>> >
>> > Thanks for the info, I am creating a readme and cleaning up the driver
>> > scripts and will be uploading them shortly.
>> >
>> >
>> > Shank
>>
>> It's not clear to me if you are using iSER or iSCSI over IPoIB. Can you
>> clarify that? I'm envisioning iSER being used where the LVM volumes are
>> logged into from the assigned guest's host, and then exposed to the
>> guest as a local scsi block device and connected to via virtio.
>>
>> If this is how it works, does the host itself become a storage device
>> within 'one'? Or, is the infiniband enabled storage device seen as the
>> storage device in 'one'?
>>
>> Thanks,
>> -C
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120510/f8d6daa4/attachment-0003.htm>