[one-users] Transfer manager

[Borja Sotomayor]

Hi Alexandre,

>  In order to hide this lag, we have implemented in our system a cache that
> pre-copy disk images before submitting a VM, and start a new copy each time
> a VM is started for having a disk image ready for a the next submission. We
> can do that because we manage a template directory that the user can choose
> which template to submit. I will sent more details later if you are
> interested (I am writing a technical report describing what we are doing).

Regarding disk image transfers and image reuse/caching, you might be
interested in the following two papers:

Combining Batch Execution and Leasing Using Virtual Machines
http://haizea.cs.uchicago.edu/pubs/Haizea_HPDC2008.pdf

Overhead Matters: A Model for Virtual Resource Management
http://workspace.globus.org/papers/Overhead_Matters_VTDC06.pdf

They describe a series of strategies for disk image transfer
scheduling and reuse, which are implemented by the Haizea scheduler
(which can be used as a drop-in replacement for OpenNebula's
scheduler; more info on Haizea here: http://haizea.cs.uchicago.edu/),
and showing the impact of image transfers and image reuse on long
traces of requests (in the order of weeks). Once the TM is included in
the stable release of OpenNebula, we plan to further the integration
of OpenNebula and Haizea so that Haizea will also be able to direct
OpenNebula on when to perform transfers, and take them into account
when scheduling new VMs (as you point out, the lag in deploying large
images can be considerable and has to be taken into account by the
scheduler). I would certainly be interested in hearing more about your
image cache mechanism.

As for the TM being optional, I think that it's been designed in such
a way that it is loosely coupled to the rest of OpenNebula. In other
words, if you want to plug in another TM (or not use one at all), you
should be able to do so (although presumably the documentation for
this is still in development), but I'll let the OpenNebula folks have
the final word on this.

Cheers!
--Borja