[one-users] Ceph datastore monitoring in OpenNebula

Alvaro Simon Garcia Alvaro.SimonGarcia at UGent.be
Thu Jul 17 05:11:48 PDT 2014


Hi Jaime

Sorry for the late reply, I didn't see your mail before because I was 
included in cc. With this patch you don't need to include a new user 
into your ceph keyring. If you want to use a ceph datastore you only 
need to include the user and pool into the datastore template that's 
all. In our case we use livbirt datastore and we have created two ceph 
datastore for testing porposes (each one use a different pool as well):


$ onedatastore show ceph
DATASTORE 103 INFORMATION
ID             : 103
NAME           : ceph
USER           : oneadmin
GROUP          : oneadmin
CLUSTER        : -
TYPE           : IMAGE
DS_MAD         : ceph
TM_MAD         : ceph
BASE PATH      : /var/lib/one//datastores/103
DISK_TYPE      : RBD

DATASTORE CAPACITY
TOTAL:         : 87.6T
FREE:          : 59.2T
USED:          : 28.4T
LIMIT:         : -

PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---

DATASTORE TEMPLATE
BASE_PATH="/var/lib/one//datastores/"
BRIDGE_LIST="hyp004.cubone.os"
CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
CEPH_USER="libvirt"
CLONE_TARGET="SELF"
DATASTORE_CAPACITY_CHECK="yes"
DISK_TYPE="RBD"
DS_MAD="ceph"
LN_TARGET="NONE"
POOL_NAME="one"
TM_MAD="ceph"

IMAGES
7
8
31
32
34
35


and the second one:

$ onedatastore show "ceph two"
DATASTORE 104 INFORMATION
ID             : 104
NAME           : ceph two
USER           : oneadmin
GROUP          : oneadmin
CLUSTER        : -
TYPE           : IMAGE
DS_MAD         : ceph
TM_MAD         : ceph
BASE PATH      : /var/lib/one//datastores/104
DISK_TYPE      : RBD

DATASTORE CAPACITY
TOTAL:         : 87.6T
FREE:          : 59.2T
USED:          : 28.4T
LIMIT:         : -

PERMISSIONS
OWNER          : um-
GROUP          : u--
OTHER          : ---

DATASTORE TEMPLATE
BASE_PATH="/var/lib/one//datastores/"
BRIDGE_LIST="hyp004.cubone.os"
CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
CEPH_USER="libvirt"
CLONE_TARGET="SELF"
DATASTORE_CAPACITY_CHECK="yes"
DISK_TYPE="RBD"
DS_MAD="ceph"
LN_TARGET="NONE"
POOL_NAME="two"
TM_MAD="ceph"

As you can see we are using different pools in each one (so we don't 
need to include the pool name into ceph.conf either, and we are able to 
use several ceph clusters) and this change simplifies the ONE conf and 
ceph cluster admin as well.

Cheers
Alvaro

On 03/07/14 17:08, Jaime Melis wrote:
> Hi Alvaro,
>
> thanks a lot for the contribution. I haven't tested it yet, but the 
> code seems to be perfect.
>
> My main issue here is that I'm not entirely sure if we need this. 
> Doesn't it make sense for the oneadmin user to be in the keyring and 
> be able to run rados?  You mentioned this, but I'm having some trouble 
> understanding what do you mean:
>
> > This feature could be useful as well if you want to monitor several 
> datastores that are using different ceph pools and users ids.
> > You only have to include the id and pool info into the ONE 
> datastore template and the monitoring script will use one or another 
> depending on the DS conf.
>
> With the current system you can monitor multiple ceph pools as long as 
> the oneadmin user has right, isn't that so?
>
> Joel, would you like to weigh in? would you find this useful?
>
> cheers,
> Jaime
>
>
> On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan <jfontan at opennebula.org 
> <mailto:jfontan at opennebula.org>> wrote:
>
>     Great, thanks! Our ceph dev is traveling right now, he will review the
>     patches when he arrives.
>
>     Cheers
>
>     On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
>     <Alvaro.SimonGarcia at ugent.be <mailto:Alvaro.SimonGarcia at ugent.be>>
>     wrote:
>     > Hi Javier
>     >
>     > We have modified the ceph monitor to take into account the
>     datastore ceph
>     > pool and also the ceph user. This is a generic solution that
>     could be useful
>     > for other datacenters as well, we have created a pull request in
>     github if
>     > you are agree about this change and you want to include it in
>     the next
>     > release.
>     >
>     > https://github.com/OpenNebula/one/pull/27
>     >
>     > We only have modified these lines in
>     > /var/lib/one/remotes/datastore/ceph/monitor:
>     >
>     >> --- monitor.orig.190614    2014-06-19 14:35:24.022755989 +0200
>     >> +++ monitor    2014-06-19 14:49:34.043187892 +0200
>     >> @@ -46,10 +46,12 @@
>     >>  while IFS= read -r -d '' element; do
>     >>      XPATH_ELEMENTS[i++]="$element"
>     >>  done < <($XPATH
>     /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST \
>     >> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>     >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>     >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>     >>
>     >>  BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>     >>  POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>     >> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>     >>
>     >>  HOST=`get_destination_host`
>     >>
>     >> @@ -61,7 +63,7 @@
>     >>  # ------------ Compute datastore usage -------------
>     >>
>     >>  MONITOR_SCRIPT=$(cat <<EOF
>     >> -$RADOS df | $AWK '{
>     >> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>     >>      if (\$1 == "total") {
>     >>
>     >>          space = int(\$3/1024)
>     >
>     >
>     > CEPH_USER and POOL_NAME should be mandatory to create the ceph
>     datastore.
>     >
>     > Cheers
>     > Alvaro
>     >
>     >
>     >
>     >> Hola Javi
>     >>
>     >> Thanks a lot for your feedback. Yes we will modify the current
>     >> monitoring scripts to take into account this. This feature could be
>     >> useful as well if you want to monitor several datastores that
>     are using
>     >> different ceph pools and users ids. You only have to include
>     the id and
>     >> pool info into the ONE datastore template and the monitoring
>     script will
>     >> use one or another depending on the DS conf.
>     >>
>     >>
>     >> Cheers and thanks!
>     >> Alvaro
>     >> On 2014-06-17 14:55, Javier Fontan wrote:
>     >>>
>     >>> CEPH_USER is used when generating the libvirt/kvm deployment
>     file but
>     >>> not for DS monitoring:
>     >>>
>     >>> * Deployment file generation:
>     >>>
>     >>>
>     https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>     >>> * Monitoring:
>     >>>
>     https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>     >>>
>     >>> Ceph is not may area of expertise but you may need to add those
>     >>> parameters to the monitor script a maybe to other scripts that
>     use the
>     >>> "rados" command. It may also be possible to modify the RADOS
>     command
>     >>> to have those parameters instead of modifying all the scripts:
>     >>>
>     >>>
>     >>>
>     https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>     >>>
>     >>> As I said I don't know much about Ceph and it may be those
>     credentials
>     >>> could be set in a config file or so.
>     >>>
>     >>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>     >>> <Alvaro.SimonGarcia at ugent.be
>     <mailto:Alvaro.SimonGarcia at ugent.be>> wrote:
>     >>>>
>     >>>> Hi
>     >>>>
>     >>>> We have included the admin keyring instead of libvirt user and it
>     >>>> works...
>     >>>> that means that we can run rbd or qemu-img wihtout the
>     libvirt id, but
>     >>>> is
>     >>>> not the best solution. We have included the user into
>     datastore conf:
>     >>>>
>     >>>> CEPH_USER="libvirt"
>     >>>>
>     >>>> but it seems that is not used by opennebula at the end
>     >>>>
>     >>>> Cheers
>     >>>> Alvaro
>     >>>>
>     >>>>
>     >>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>     >>>>>
>     >>>>> Hi all
>     >>>>>
>     >>>>>
>     >>>>> We have included our ONE nodes into Ceph cluster, cephx auth
>     is working
>     >>>>> but OpenNebula is not able to detect the free space:
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>>> $ onedatastore show 103
>     >>>>>> DATASTORE 103 INFORMATION
>     >>>>>> ID             : 103
>     >>>>>> NAME           : ceph
>     >>>>>> USER           : oneadmin
>     >>>>>> GROUP          : oneadmin
>     >>>>>> CLUSTER        : -
>     >>>>>> TYPE           : IMAGE
>     >>>>>> DS_MAD         : ceph
>     >>>>>> TM_MAD         : ceph
>     >>>>>> BASE PATH      : /var/lib/one//datastores/103
>     >>>>>> DISK_TYPE      : RBD
>     >>>>>>
>     >>>>>> DATASTORE CAPACITY
>     >>>>>> TOTAL:         : 0M
>     >>>>>> FREE:          : 0M
>     >>>>>> USED:          : 0M
>     >>>>>> LIMIT:         : -
>     >>>>>>
>     >>>>>> PERMISSIONS
>     >>>>>> OWNER          : um-
>     >>>>>> GROUP          : u--
>     >>>>>> OTHER          : ---
>     >>>>>
>     >>>>>
>     >>>>>> $ onedatastore list
>     >>>>>>      ID NAME  SIZE AVAIL CLUSTER      IMAGES TYPE DS
>     >>>>>> TM
>     >>>>>>       0 system  114.8G 85%   -                 0 sys -
>     >>>>>> shared
>     >>>>>>       1 default 114.9G 84%   -                 2 img fs
>     >>>>>> ssh
>     >>>>>>       2 files 114.9G 84%   -                 0 fil fs
>     >>>>>> ssh
>     >>>>>>     103 ceph  0M -     -                 0 img ceph
>     >>>>>> ceph
>     >>>>>
>     >>>>>
>     >>>>> but if we run rados as oneadmin user:
>     >>>>>
>     >>>>>> $  rados df -p one --id libvirt
>     >>>>>> pool name       category           KB      objects clones
>     >>>>>> degraded      unfound rd        rd KB           wr        wr
>     >>>>>> KB
>     >>>>>> one             -            0 0            0
>     >>>>>> 0           0            0 0          0            0
>     >>>>>>      total used         1581852           37
>     >>>>>>      total avail   140846865180
>     >>>>>>      total space   140848447032
>     >>>>>
>     >>>>>
>     >>>>> It's working correctly (we are using one pool and libvirt
>     ceph id)
>     >>>>>
>     >>>>> the oned.log only shows this info:
>     >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore
>     default (1)
>     >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore
>     files (2)
>     >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore ceph
>     (103)
>     >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default (1)
>     successfully
>     >>>>> monitored.
>     >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2)
>     successfully
>     >>>>> monitored.
>     >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103)
>     successfully
>     >>>>> monitored.
>     >>>>>
>     >>>>> Any clue about how to debug this issue?
>     >>>>>
>     >>>>> Thanks in advance!
>     >>>>> Alvaro
>     >>>>>
>     >>>>>
>     >>>>>
>     >>>>> _______________________________________________
>     >>>>> Users mailing list
>     >>>>> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
>     >>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>     >>>>
>     >>>> _______________________________________________
>     >>>> Users mailing list
>     >>>> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
>     >>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>     >>>
>     >>>
>     >> _______________________________________________
>     >> Users mailing list
>     >> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
>     >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>     >
>     >
>
>
>
>     --
>     Javier Fontán Muiños
>     Developer
>     OpenNebula - Flexible Enterprise Cloud Made Simple
>     www.OpenNebula.org <http://www.OpenNebula.org> | @OpenNebula |
>     github.com/jfontan <http://github.com/jfontan>
>     _______________________________________________
>     Users mailing list
>     Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
>     http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
>
>
> -- 
> Jaime Melis
> Project Engineer
> OpenNebula - Flexible Enterprise Cloud Made Simple
> www.OpenNebula.org <http://www.OpenNebula.org> | jmelis at opennebula.org 
> <mailto:jmelis at opennebula.org>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140717/0c585a24/attachment-0001.htm>


More information about the Users mailing list