[one-dev] OpenNebula storage cache?

[Ruben S. Montero]

Hi stuart

Sorry for the delay, the release process got us sidetracked. Anyway this is
a quite interesting feature.

There is a long-standing issue in our roadmap to implement caching for
images (basically at the TM level). This could be done for image based
Datastores/Transfer Managers, with the aid of a kind of caching daemon
taking care of concurrent copy/rm operations on the system ds.

Apart from the need to develop the cache keeper daemon, we have the problem
of block based datastores (like ceph, LVM). So, we were not able to devise
a solution general enough.

So, regarding your project:

* I think it is a great idea to have a generic interface to a cache system,
so we can adapt the implementation for block and file based datastores

 * In the repo, you have also code for a datastore driver. Sorry, I
couldn't have time to go through all the actions, but Why does the caching
system need to modify the DS driver? The cache should seat between the
image DS and the system DS, just where the TM works.

* I've moved the issue in OpenNebula to the backlog.

Thanks for the contribution, nice!

Ruben










On Thu, Apr 3, 2014 at 1:43 PM, Stuart Longland <stuartl at vrt.com.au> wrote:

> Hi all,
>
> I've been doing a bit of work in getting a combined Ceph+FlashCache
> driver going for OpenNebula.  At the moment I've got some initial
> proof-of-concept code which is based on the current stable OpenNebula
> release.  Things seem to be working but the testing has been light.
>
> Some rationale behind the idea is here:
>
> http://stuartl.longlandclan.yi.org/blog/2014/04/03/ceph-and-flashcache-in-opennebula/
>
> There you'll also find a link to the git repository.
>
> It got me thinking though.  Right now my cache and back-end drivers are
> very tightly coupled.  How viable would it be, to have a "cache" driver,
> much like we've got drivers for the actual datastores, and drivers for
> moving data between datastores?
>
> Right now my work has focussed on using Ceph RBDs and FlashCache
> together.  I'm curious to try bcache, but that will require re-writing
> chunks of code.  If someone wanted to use this driver with an iSCSI/CLVM
> back-end, they're in for similar rewriting tasks.
>
> A local cache is useful in any scenario where shared storage is accessed
> over a network.
>
> Are there any thoughts on how one might implement this?
>
> Regards,
> --
> Stuart Longland
> Systems Engineer
>      _ ___
> \  /|_) |                           T: +61 7 3535 9619
>  \/ | \ |     38b Douglas Street    F: +61 7 3535 9699
>    SYSTEMS    Milton QLD 4064       http://www.vrt.com.au
>
>
> _______________________________________________
> Dev mailing list
> Dev at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org
>
> --
> <http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org>
> --
> Ruben S. Montero, PhD
> Project co-Lead and Chief Architect<http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org>
> OpenNebula - Flexible Enterprise Cloud Made Simple
>  <http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org>
> www.OpenNebula.org | rsmontero at opennebula.org | @OpenNebula
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/dev-opennebula.org/attachments/20140429/5b1805b0/attachment-0001.htm>