[one-users] Infiniband, nodes and VMs

[Steven Timm]

We are running an Infiniband setup on FermiCloud using the SR-IOV feature
of our Mellanox clouds.  In the MPI jobs we have done we have used
IPoIB for communication.  For point to point communication between VM's
the bandwidth is similar to that between bare metal machines.
In a bigger cluster with small messages being passed there can
be some degradation but still it's decent performance.

As far as an image store accessible via infiniband we don't do
that currently, but it could be done for instance via an NFS data
store exported to your head node via ethernet and to your infiniband
nodes via IPOIB.

Steve


On Sun, 28 Sep 2014, Galimba wrote:

> Hello!I've been given the task to step up our game and connect a few nodes
> tru infiniband. This meaning: we've been working over gigabit ethernet this
> far, but now we want our ONE nodes hosting the VMs to be interconected tru
> IP over infiniband. We have the hardware to do so, but I must confess I lack
> the expertise to pull this off.
> 
> I've been googling around a bit and found a couple of links. Even tho I
> think the answers to both my questions are there (yes, it's possible, and no
> it's not that hard).
> 
> https://github.com/OpenNebula/addon-kvm-sr-iov
> http://blog.scottlowe.org/2009/12/02/what-is-sr-iov/
> http://wiki.opennebula.org/ecosystem:sr-iov
> http://pkg-ofed.alioth.debian.org/howto/infiniband-howto-5.html
> 
> I know little about infiniband, so here are a few questions I started with:
> 1) Is it hard to get the VMs to work with eachother over infiniband? I mean,
> if I have a client who wants his VMs to trade messages at really fast speed.
> I'd like them to be able to create VMs that work on a specific infiniband
> network at high speed.
> 
> 2) Is it easy to configure the hosts to deliver the images back and forth
> over infiniband? This would really speed up the deploy.
> 
> 3) Is someone using this setup? I mean, using a hardware interface shared
> between both the nodes and the vms.
> 
> I'd apreciate any input you guys can give me =)
> 
> best regards
> galimba
> 
> --
> 
> 
>

------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm at fnal.gov  http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing