[one-users] Ceph datastore monitoring in OpenNebula

Jaime Melis jmelis at opennebula.org
Mon Aug 11 01:25:36 PDT 2014


Hi Alvaro,

Could you also provide a patch for the documentation?


On Mon, Aug 11, 2014 at 10:22 AM, Alvaro Simon Garcia <
Alvaro.SimonGarcia at ugent.be> wrote:

>  Hi Jaime, Javier
>
> Do you think that it would be possible to include these ceph patches into
> next 4.8 release? At this moment we are applying theses patches by hand in
> our configuration to use different Ceph pools.
>
> https://github.com/OpenNebula/one/pull/27
> https://github.com/OpenNebula/one/pull/28
> https://github.com/OpenNebula/one/pull/29
> https://github.com/OpenNebula/one/pull/30
> https://github.com/OpenNebula/one/pull/31
>
> Cheers
> Alvaro
>
>
> On 17/07/14 14:11, Alvaro Simon Garcia wrote:
>
> Hi Jaime
>
> Sorry for the late reply, I didn't see your mail before because I was
> included in cc. With this patch you don't need to include a new user into
> your ceph keyring. If you want to use a ceph datastore you only need to
> include the user and pool into the datastore template that's all. In our
> case we use livbirt datastore and we have created two ceph datastore for
> testing porposes (each one use a different pool as well):
>
>
> $ onedatastore show ceph
> DATASTORE 103
> INFORMATION
> ID             : 103
> NAME           : ceph
> USER           : oneadmin
> GROUP          : oneadmin
> CLUSTER        : -
> TYPE           : IMAGE
> DS_MAD         : ceph
> TM_MAD         : ceph
> BASE PATH      : /var/lib/one//datastores/103
> DISK_TYPE      : RBD
>
> DATASTORE
> CAPACITY
> TOTAL:         : 87.6T
> FREE:          : 59.2T
> USED:          : 28.4T
> LIMIT:         : -
>
> PERMISSIONS
>
> OWNER          : um-
> GROUP          : u--
> OTHER          : ---
>
> DATASTORE
> TEMPLATE
> BASE_PATH="/var/lib/one//datastores/"
> BRIDGE_LIST="hyp004.cubone.os"
> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> CEPH_USER="libvirt"
> CLONE_TARGET="SELF"
> DATASTORE_CAPACITY_CHECK="yes"
> DISK_TYPE="RBD"
> DS_MAD="ceph"
> LN_TARGET="NONE"
> POOL_NAME="one"
> TM_MAD="ceph"
>
> IMAGES
> 7
> 8
> 31
> 32
> 34
> 35
>
>
> and the second one:
>
> $ onedatastore show "ceph two"
> DATASTORE 104
> INFORMATION
> ID             : 104
> NAME           : ceph two
> USER           : oneadmin
> GROUP          : oneadmin
> CLUSTER        : -
> TYPE           : IMAGE
> DS_MAD         : ceph
> TM_MAD         : ceph
> BASE PATH      : /var/lib/one//datastores/104
> DISK_TYPE      : RBD
>
> DATASTORE
> CAPACITY
> TOTAL:         : 87.6T
> FREE:          : 59.2T
> USED:          : 28.4T
> LIMIT:         : -
>
> PERMISSIONS
>
> OWNER          : um-
> GROUP          : u--
> OTHER          : ---
>
> DATASTORE
> TEMPLATE
> BASE_PATH="/var/lib/one//datastores/"
> BRIDGE_LIST="hyp004.cubone.os"
> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> CEPH_USER="libvirt"
> CLONE_TARGET="SELF"
> DATASTORE_CAPACITY_CHECK="yes"
> DISK_TYPE="RBD"
> DS_MAD="ceph"
> LN_TARGET="NONE"
> POOL_NAME="two"
> TM_MAD="ceph"
>
> As you can see we are using different pools in each one (so we don't need
> to include the pool name into ceph.conf either, and we are able to use
> several ceph clusters) and this change simplifies the ONE conf and ceph
> cluster admin as well.
>
> Cheers
> Alvaro
>
> On 03/07/14 17:08, Jaime Melis wrote:
>
> Hi Alvaro,
>
>  thanks a lot for the contribution. I haven't tested it yet, but the code
> seems to be perfect.
>
>  My main issue here is that I'm not entirely sure if we need this.
> Doesn't it make sense for the oneadmin user to be in the keyring and be
> able to run rados?  You mentioned this, but I'm having some trouble
> understanding what do you mean:
>
>  > This feature could be useful as well if you want to monitor several
> datastores that are using different ceph pools and users ids.
> > You only have to include the id and pool info into the ONE datastore template
> and the monitoring script will use one or another depending on the DS
> conf.
>
>  With the current system you can monitor multiple ceph pools as long as
> the oneadmin user has right, isn't that so?
>
>  Joel, would you like to weigh in? would you find this useful?
>
>  cheers,
> Jaime
>
>
> On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan <jfontan at opennebula.org>
> wrote:
>
>> Great, thanks! Our ceph dev is traveling right now, he will review the
>> patches when he arrives.
>>
>> Cheers
>>
>> On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
>>  <Alvaro.SimonGarcia at ugent.be> wrote:
>> > Hi Javier
>> >
>> > We have modified the ceph monitor to take into account the datastore
>> ceph
>> > pool and also the ceph user. This is a generic solution that could be
>> useful
>> > for other datacenters as well, we have created a pull request in github
>> if
>> > you are agree about this change and you want to include it in the next
>> > release.
>> >
>> > https://github.com/OpenNebula/one/pull/27
>> >
>> > We only have modified these lines in
>> > /var/lib/one/remotes/datastore/ceph/monitor:
>> >
>> >> --- monitor.orig.190614    2014-06-19 14:35:24.022755989 +0200
>> >> +++ monitor    2014-06-19 14:49:34.043187892 +0200
>> >> @@ -46,10 +46,12 @@
>> >>  while IFS= read -r -d '' element; do
>> >>      XPATH_ELEMENTS[i++]="$element"
>> >>  done < <($XPATH /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST
>> \
>> >> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>> >>
>> >>  BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>> >>  POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>> >> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>> >>
>> >>  HOST=`get_destination_host`
>> >>
>> >> @@ -61,7 +63,7 @@
>> >>  # ------------ Compute datastore usage -------------
>> >>
>> >>  MONITOR_SCRIPT=$(cat <<EOF
>> >> -$RADOS df | $AWK '{
>> >> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>> >>      if (\$1 == "total") {
>> >>
>> >>          space = int(\$3/1024)
>> >
>> >
>> > CEPH_USER and POOL_NAME should be mandatory to create the ceph
>> datastore.
>> >
>> > Cheers
>> > Alvaro
>> >
>> >
>> >
>> >> Hola Javi
>> >>
>> >> Thanks a lot for your feedback. Yes we will modify the current
>> >> monitoring scripts to take into account this. This feature could be
>> >> useful as well if you want to monitor several datastores that are using
>> >> different ceph pools and users ids. You only have to include the id and
>> >> pool info into the ONE datastore template and the monitoring script
>> will
>> >> use one or another depending on the DS conf.
>> >>
>> >>
>> >> Cheers and thanks!
>> >> Alvaro
>> >> On 2014-06-17 14:55, Javier Fontan wrote:
>> >>>
>> >>> CEPH_USER is used when generating the libvirt/kvm deployment file but
>> >>> not for DS monitoring:
>> >>>
>> >>> * Deployment file generation:
>> >>>
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>> >>> * Monitoring:
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>> >>>
>> >>> Ceph is not may area of expertise but you may need to add those
>> >>> parameters to the monitor script a maybe to other scripts that use the
>> >>> "rados" command. It may also be possible to modify the RADOS command
>> >>> to have those parameters instead of modifying all the scripts:
>> >>>
>> >>>
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>> >>>
>> >>> As I said I don't know much about Ceph and it may be those credentials
>> >>> could be set in a config file or so.
>> >>>
>> >>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>> >>> <Alvaro.SimonGarcia at ugent.be> wrote:
>> >>>>
>> >>>> Hi
>> >>>>
>> >>>> We have included the admin keyring instead of libvirt user and it
>> >>>> works...
>> >>>> that means that we can run rbd or qemu-img wihtout the libvirt id,
>> but
>> >>>> is
>> >>>> not the best solution. We have included the user into datastore conf:
>> >>>>
>> >>>> CEPH_USER="libvirt"
>> >>>>
>> >>>> but it seems that is not used by opennebula at the end
>> >>>>
>> >>>> Cheers
>> >>>> Alvaro
>> >>>>
>> >>>>
>> >>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>> >>>>>
>> >>>>> Hi all
>> >>>>>
>> >>>>>
>> >>>>> We have included our ONE nodes into Ceph cluster, cephx auth is
>> working
>> >>>>> but OpenNebula is not able to detect the free space:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> $ onedatastore show 103
>> >>>>>> DATASTORE 103 INFORMATION
>> >>>>>> ID             : 103
>> >>>>>> NAME           : ceph
>> >>>>>> USER           : oneadmin
>> >>>>>> GROUP          : oneadmin
>> >>>>>> CLUSTER        : -
>> >>>>>> TYPE           : IMAGE
>> >>>>>> DS_MAD         : ceph
>> >>>>>> TM_MAD         : ceph
>> >>>>>> BASE PATH      : /var/lib/one//datastores/103
>> >>>>>> DISK_TYPE      : RBD
>> >>>>>>
>> >>>>>> DATASTORE CAPACITY
>> >>>>>> TOTAL:         : 0M
>> >>>>>> FREE:          : 0M
>> >>>>>> USED:          : 0M
>> >>>>>> LIMIT:         : -
>> >>>>>>
>> >>>>>> PERMISSIONS
>> >>>>>> OWNER          : um-
>> >>>>>> GROUP          : u--
>> >>>>>> OTHER          : ---
>> >>>>>
>> >>>>>
>> >>>>>> $ onedatastore list
>> >>>>>>      ID NAME                SIZE AVAIL CLUSTER      IMAGES TYPE DS
>> >>>>>> TM
>> >>>>>>       0 system            114.8G 85%   -                 0 sys -
>> >>>>>> shared
>> >>>>>>       1 default           114.9G 84%   -                 2 img fs
>> >>>>>> ssh
>> >>>>>>       2 files             114.9G 84%   -                 0 fil fs
>> >>>>>> ssh
>> >>>>>>     103 ceph                  0M -     -                 0 img ceph
>> >>>>>> ceph
>> >>>>>
>> >>>>>
>> >>>>> but if we run rados as oneadmin user:
>> >>>>>
>> >>>>>> $  rados df -p one --id libvirt
>> >>>>>> pool name       category                 KB      objects clones
>> >>>>>> degraded      unfound           rd        rd KB           wr
>>  wr
>> >>>>>> KB
>> >>>>>> one             -                          0 0            0
>> >>>>>> 0           0            0 0            0            0
>> >>>>>>      total used         1581852           37
>> >>>>>>      total avail   140846865180
>> >>>>>>      total space   140848447032
>> >>>>>
>> >>>>>
>> >>>>> It's working correctly (we are using one pool and libvirt ceph id)
>> >>>>>
>> >>>>> the oned.log only shows this info:
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore default (1)
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore files (2)
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore ceph (103)
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default (1)
>> successfully
>> >>>>> monitored.
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2) successfully
>> >>>>> monitored.
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103) successfully
>> >>>>> monitored.
>> >>>>>
>> >>>>> Any clue about how to debug this issue?
>> >>>>>
>> >>>>> Thanks in advance!
>> >>>>> Alvaro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Users mailing list
>> >>>>> Users at lists.opennebula.org
>> >>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >>>>
>> >>>> _______________________________________________
>> >>>> Users mailing list
>> >>>> Users at lists.opennebula.org
>> >>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >>>
>> >>>
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users at lists.opennebula.org
>> >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >
>> >
>>
>>
>>
>>  --
>> Javier Fontán Muiños
>> Developer
>> OpenNebula - Flexible Enterprise Cloud Made Simple
>> www.OpenNebula.org | @OpenNebula | github.com/jfontan
>>  _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>
>
>
>  --
>  Jaime Melis
> Project Engineer
> OpenNebula - Flexible Enterprise Cloud Made Simple
> www.OpenNebula.org | jmelis at opennebula.org
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>


-- 
Jaime Melis
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org | jmelis at opennebula.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140811/520af1c3/attachment-0001.htm>


More information about the Users mailing list