[one-users] Combining multiple storage backends within a single VM

[Vivien Bernet-Rollande]

Hi.
I'm new to the OpenNebula Project. I'm currently testing if it can do what
we need. So far, it looks promising, but I have a problem.

Currently, Open Nebula assumes there is a single type of storage for each
hypervisor. It lets us write custom drivers, but only for one single storage
type at a time.

I have read
http://lists.opennebula.org/htdig.cgi/users-opennebula.org/2011-April/004678.html,
but this is not what I am trying to do.

I want to mix several types of storage on a single virtual machine. For
instance :
 - swap on local disks
 - data on NFS
or :
 - persistent data on shared block devices (iscsi, gluster, ceph, you name
it)
 - temporary OS on local storage.
 - CD images on a shared filesystem (NFS)
or even :
 - local SSD for a fast cache
 - local sata drive for the OS

I'm totally fine with the idea of writing (and sharing) a custom transfer
manager driver for this purpose. The question is : will it be enough, or
will I need do add some way to dispatch calls to various

I've come up with several ideas regarding how to deal with this from a
configuration standpoint.

The idea I like most is fairly straightforward : we specify some options for
the TM driver . Of course, the TM has to be custom written and do all the
work. For example :

DISK = [ IMAGE = "Debian base system",
         TM_OPTIONS = "storage=iscsi iscsi_type=solaris_zfs
iscsi_filer=1.2.3.4 iscsi_volume=vm_images" ]

With these options, the transfer manager could CLONE the image :
 - check the size of the source image
 - connect to the filer
 - create a new LUN in the volume with the right size
 - make sure the node sees the LUN
 - create a link in /srv/cloud/one/5/images/disk.0 pointing to that lun
(with some /dev/disk/by-path/ magic)
 - copy the data from wherever the image is stored onto the LUN.

With this in place, MKSWAP, MKIMAGE, LN and DELETE operations are
straightforward.

MV is a bit more complex if we want to be able to move data from one storage
backend to another, but can probably be figured out. Then I need to figure
where I want to store data, how, make sure everything interacts well. But
that's just a matter of scripting (and testing), and I can handle it.

The key operation is to have the TM_OPTIONS variable given to the transfer
manager. I've checked, it's possible to add variables in the VM template,
they are simply ignored.

The best way I can think of is to have the transfer manager take that field,
and add it an environment variable. This would be 100% compatible with
existing drivers, and allow me to do what I need.


The thing is, I'm not quite sure about how to implement this.

If I'm correct, I can get the value for TM_OPTIONS easily in
src/tm/TransferManager.cc with a simple disk->vector_value("TM_OPTIONS").
The problem is the commands are then simply written to a file, and it's kind
of hard to set an environment variable that would be forwarded to the
driver's script.

So I came up with another idea : instead of adding TM_OPTIONS as an
environment variable, append it to the line in the command files. For
instance, instead of :

LN opennebula:/srv/cloud/images/debian.img 1.1.1.1:
/var/lib/one/25/images/disk.0

We would have :

LN opennebula:/srv/cloud/images/debian.img
1.1.1.1:/var/lib/one/25/images/disk.0
storage=iscsi iscsi_type=nexenta iscsi_filer=1.2.3.4 iscsi_volume=vm_images

Then it would be the driver's work to parse the command line.

Since most drivers use $1 and $2 to handle their parameters, this would be
perfectly transparent to existing code. one_tm simply forwards '$*' to the
right driver. However, it would enable building much more powerful transfer
manager drivers.

Oh, and "CONTEXT" could take a similar TM_OPTIONS argument.

Now, a few more practical questions :
 - is there interest for this feature from the community ?
 - is the way I propose to add it to the VM template compliant to ONE's way
of doing things ? Is another name preferred (rather than TM_OPTIONS) ?
 - is appending those options at the end of the command the right way, or
will it break something further down the line ?
 - any chances to get this in 2.2 ? Or will I have to maintain my own
patched version ? (I'm using the .deb you guys provide by the way).


The other method I thought of is to have a StoragePool database object, with
a set of options. One would register such a StoragePool in a similar way to
Virtual Networks (ie, with a template and a command). The user could then
say "I want this disk to be in storage pool X". This moves the
responsability of managing storage from the TM driver to the core. It has
some nice implications (for instance we could attach a default StoragePool
to a cluster or a host, we could monitor available space in a StoragePool, a
pool could be marked as persistant or not, etc.), but is much more complex
to implement, and might break existing setups. Morever this would break
existing drivers, because we need a way to transfer images between storage
pools. But I like the idea of declaring "I have an NFS share here. Oh, and
some AoE there, and SCSI other here. And all those machines have two 1T
drives they can use", and having the VM select the type of storage they
need.

For me, the most important difference between the two solutions is that I
can have the first one up and running in a matter of days, while the other
might require several weeks. I will go and try the first one. I do share the
second one with you, however, because I think it would be an interesting,
even more extensible, long-term solution.


-- 
Vivien Bernet-Rollande
Systems & Networking Engineer
Alter Way Hosting
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20110415/c54ae72f/attachment-0001.htm>