[one-users] How to use Ceph/RBD for System Datastore

Wed Jul 10 16:14:52 PDT 2013

Hey Bill,

Thanks for this.  This works perfectly!

Thanks,
Jon A

On Wed, Jul 10, 2013 at 6:44 AM, Campbell, Bill <
bcampbell at axcess-financial.com> wrote:

> Not entirely.  You shouldn’t need to manually create/mount an RBD for the
> system datastore.  Since the system datastore holds the running VM
> deployment files (and not necessarily an RBD image, just a reference to it
> in the deployment file) then this directory does not necessarily need to be
> shared.****
>
> ** **
>
> Here’s what we do:****
>
> ** **
>
> **·         **OpenNebula system configured with no special exports/shares.
> ****
>
> **·         **The System datastore is modified to use the SSH transfer
> manager****
>
> **·         **We modify the SSH transfer manager pre/post migrate scripts
> (by default located in /var/lib/one/remotes/tm/ssh/) to copy files from the
> source host to the destination host prior to migration/delete files on
> source after successful migration.****
>
> ** **
>
> Don’t worry about mapping/unmapping RBD volumes.  When creating/importing
> images into the Ceph datastore the RBDs should be created at this point.
> So long as the Hypervisor nodes can see/interact with the Ceph cluster,
> when you deploy the VM it will use the RBD in the cluster for storage (no
> files copied/mapped locally, all handled by QEMU).****
>
> ** **
>
> ** **
>
> Here is the pre-migrate script we use (very simple):****
>
> ** **
>
> *#!/bin/bash*
>
> * *
>
> *SRC=$1*
>
> *DST=$2*
>
> *REMDIR=$3*
>
> *VMID=$4*
>
> *DSID=$5*
>
> *TEMPLATE=$6*
>
> * *
>
> *ssh $DST mkdir -p /var/lib/one/datastores/0/$VMID*
>
> * *
>
> *ssh $SRC scp /var/lib/one/datastores/0/$VMID/*
> $DST:/var/lib/one/datastores/0/$VMID/*
>
> * *
>
> *exit 0*
>
> ** **
>
> And the post-migrate script:****
>
> ** **
>
> *#!/bin/bash*
>
> ** **
>
> *SRC=$1*
>
> *DST=$2*
>
> *REMDIR=$3*
>
> *VMID=$4*
>
> *DSID=$5*
>
> *TEMPLATE=$6*
>
> * *
>
> *ssh $SRC rm -rf /var/lib/one/datastores/0/$VMID*
>
> * *
>
> *exit 0*
>
> ** **
>
> ** **
>
> Hope this helps!****
>
> ** **
>
> *From:* Jon [mailto:three18ti at gmail.com]
> *Sent:* Wednesday, July 10, 2013 12:01 AM
> *To:* Campbell, Bill
> *Cc:* Users OpenNebula
> *Subject:* Re: [one-users] How to use Ceph/RBD for System Datastore****
>
> ** **
>
> Hey Bill,****
>
> ** **
>
> Thanks for getting back to me.  ****
>
> ** **
>
> If I'm understanding you correctly, you're basically using the ssh
> transfer manager to perform live migrations?****
>
> Do you then create/mount one rbd per host?****
>
> ** **
>
> E.g.,  ****
>
> ** **
>
> host1:****
>
> mount /dev/rbd/rbd/host1-one-system /var/lib/one/datastores/0****
>
> ** **
>
> host2:****
>
> mount /dev/rbd/rbd/host2-one-system /var/lib/one/datastores/0****
>
> ** **
>
> then use the modified ssh drivers to perform the migrations?****
>
> ** **
>
> I would definitely be interested in learning how you accomplished that.***
> *
>
> ** **
>
> My other thought was to use CephFS for shared storage.  This would
> eliminate the need for a NFS/GlusterFS/CLVM, which is an extra layer of
> complexity I would like to avoid.  As I understand it though, CephFS isn't
> "ready for prime-time" which gives me pause...  ****
>
> ** **
>
> Thanks again,****
>
> Jon A****
>
> On Tue, Jul 9, 2013 at 7:55 PM, Campbell, Bill <
> bcampbell at axcess-financial.com> wrote:****
>
> Jon,****
>
> I think I understand what you are trying to do, but I think it doesn't
> quite work that way.  Let me try to explain (and please let me know if I
> don't explain it well enough ;-))****
>
> ** **
>
> I don't think that you can use Ceph directly as a system datastore.  The
> way the Ceph datastore driver works for migrations is leveraging whatever
> transfer method you have for the system datastore to perform the migration.
>  For example, if you use the 'shared' system datastore, then it will use
> that transfer manager's pre and post migration drivers.  For 'ssh', the ssh
> drivers, and so on.  The way the Ceph datastore is implemented is as Ceph
> Block Devices, so unfortunately there is not a way to use it as a simple
> shared volume.****
>
> ** **
>
> There are 2 potential solutions for getting live migrations working for
> your Ceph datastore VMs:****
>
>    - Create a shared NFS volume (or other 'sharable' filesystem, like
>    GFS2, OCFS2, etc., however these are much more complicated to configure and
>    usually not worth the hassle) and have the shared volume mounted to the
>    same location on each hypervisor node.  In a previous test deployment, we
>    just exported out the /var/lib/one/vms directory to the hypervisors.  At
>    this point, all of the hypervisors should be able to see the deployment
>    files in the same location and you should be able to perform a migration.
>    ****
>    - Use SSH as the transfer manager for your system datastore, and
>    modify the pre and post-migrate scripts to copy the deployment files from
>    the current VM host to the target VM host.  This is the method we use
>    currently in our deployment, as it is one less configuration step that we
>    have to worry about maintaining on each node, and makes expanding our
>    cluster much quicker and easier.  I can share with you the pre and
>    post-migrate scripts we use if you like.****
>
> Let me know if the above makes sense, and of course if you need any
> additional help please don't hesitate to bug me.  I'm very familiar with
> the Ceph drivers  ;-)****
>
> ** **
> ------------------------------
>
> *From: *"Jon" <three18ti at gmail.com>
> *To: *"Users OpenNebula" <users at lists.opennebula.org>
> *Sent: *Tuesday, July 9, 2013 8:05:51 PM
> *Subject: *[one-users]  How to use Ceph/RBD for System Datastore****
>
> ** **
>
> ** **
>
> Hello All,****
>
> ** **
>
> I am using Ceph as my storage back end and would like to know how to
> configure the system datastore, such that I can live migrate vms.****
>
> ** **
>
> Following the directions, I thought I could create a datastore, format it,
> and mount it at /var/lib/one/datastores/0 , however, I discovered, that
> isn't quite how things work.****
>
> ** **
>
> >>
> http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html**
> **
>
> You can read more about that at the above link, but long story short, to
> mount a shared filesystem it has to be a "clustered" filesystem (I think
> CephFS is the "clustered filesystem", in this case).****
>
> ** **
>
> I attempted to modify my system datastore config, however, I was unable to
> change the DS_MAD parameter, and vm creation errors out telling me there's
> no /var/lib/one/remotes/tm/ceph/mkswap driver (there isn't)****
>
> ** **
>
> >> oneadmin at red6:~$ onedatastore show 0 ****
>
> >> DATASTORE 0 INFORMATION
>         ****
>
> >> ID             : 0                   ****
>
> >> NAME           : system              ****
>
> >> USER           : oneadmin            ****
>
> >> GROUP          : oneadmin            ****
>
> >> CLUSTER        : -                   ****
>
> >> TYPE           : SYSTEM              ****
>
> >> DS_MAD         : -                   ****
>
> >> TM_MAD         : ceph                ****
>
> >> BASE PATH      : /var/lib/one/datastores/0****
>
> >> DISK_TYPE      : FILE                ****
>
> >> ****
>
> >> PERMISSIONS
>         ****
>
> >> OWNER          : um-                 ****
>
> >> GROUP          : u--                 ****
>
> >> OTHER          : ---                 ****
>
> >> ****
>
> >> DATASTORE TEMPLATE
>          ****
>
> >> DISK_TYPE="rbd"****
>
> >> DS_MAD="-"****
>
> >> TM_MAD="ceph"****
>
> >> TYPE="SYSTEM_DS"****
>
> >> ****
>
> >> IMAGES         ****
>
> ** **
>
> Maybe I'm just confused.  Can anyone provide some guidance on setting ceph
> up as the system datastore?****
>
> ** **
>
> Thanks,****
>
> Jon A****
>
> ** **
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org****
>
> ** **
>
> ** **
>
> *NOTICE: Protect the information in this message in accordance with the
> company's security policies. If you received this message in error,
> immediately notify the sender and destroy all copies.*****
>
> ** **
>
> ** **
>
> *NOTICE: Protect the information in this message in accordance with the
> company's security policies. If you received this message in error,
> immediately notify the sender and destroy all copies.*
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20130710/dde5bebf/attachment-0002.htm>