[one-users] Ceph datastore monitoring in OpenNebula
Alvaro Simon Garcia
Alvaro.SimonGarcia at UGent.be
Fri Oct 10 03:05:23 PDT 2014
On 10/10/14 10:45, Ruben S. Montero wrote:
> FYI
>
> This is now in master and ready for OpenNebula 4.10.Thanks Alvaro for
> the contribution :)
Thanks a lot Ruben!
Cheers
A
>
> On Thu, Jul 17, 2014 at 2:11 PM, Alvaro Simon Garcia
> <Alvaro.SimonGarcia at ugent.be <mailto:Alvaro.SimonGarcia at ugent.be>> wrote:
>
> Hi Jaime
>
> Sorry for the late reply, I didn't see your mail before because I
> was included in cc. With this patch you don't need to include a
> new user into your ceph keyring. If you want to use a ceph
> datastore you only need to include the user and pool into the
> datastore template that's all. In our case we use livbirt
> datastore and we have created two ceph datastore for testing
> porposes (each one use a different pool as well):
>
>
> $ onedatastore show ceph
> DATASTORE 103 INFORMATION
> ID : 103
> NAME : ceph
> USER : oneadmin
> GROUP : oneadmin
> CLUSTER : -
> TYPE : IMAGE
> DS_MAD : ceph
> TM_MAD : ceph
> BASE PATH : /var/lib/one//datastores/103
> DISK_TYPE : RBD
>
> DATASTORE CAPACITY
> TOTAL: : 87.6T
> FREE: : 59.2T
> USED: : 28.4T
> LIMIT: : -
>
> PERMISSIONS
> OWNER : um-
> GROUP : u--
> OTHER : ---
>
> DATASTORE TEMPLATE
> BASE_PATH="/var/lib/one//datastores/"
> BRIDGE_LIST="hyp004.cubone.os"
> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> CEPH_USER="libvirt"
> CLONE_TARGET="SELF"
> DATASTORE_CAPACITY_CHECK="yes"
> DISK_TYPE="RBD"
> DS_MAD="ceph"
> LN_TARGET="NONE"
> POOL_NAME="one"
> TM_MAD="ceph"
>
> IMAGES
> 7
> 8
> 31
> 32
> 34
> 35
>
>
> and the second one:
>
> $ onedatastore show "ceph two"
> DATASTORE 104 INFORMATION
> ID : 104
> NAME : ceph two
> USER : oneadmin
> GROUP : oneadmin
> CLUSTER : -
> TYPE : IMAGE
> DS_MAD : ceph
> TM_MAD : ceph
> BASE PATH : /var/lib/one//datastores/104
> DISK_TYPE : RBD
>
> DATASTORE CAPACITY
> TOTAL: : 87.6T
> FREE: : 59.2T
> USED: : 28.4T
> LIMIT: : -
>
> PERMISSIONS
> OWNER : um-
> GROUP : u--
> OTHER : ---
>
> DATASTORE TEMPLATE
> BASE_PATH="/var/lib/one//datastores/"
> BRIDGE_LIST="hyp004.cubone.os"
> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> CEPH_USER="libvirt"
> CLONE_TARGET="SELF"
> DATASTORE_CAPACITY_CHECK="yes"
> DISK_TYPE="RBD"
> DS_MAD="ceph"
> LN_TARGET="NONE"
> POOL_NAME="two"
> TM_MAD="ceph"
>
> As you can see we are using different pools in each one (so we
> don't need to include the pool name into ceph.conf either, and we
> are able to use several ceph clusters) and this change simplifies
> the ONE conf and ceph cluster admin as well.
>
> Cheers
> Alvaro
>
> On 03/07/14 17:08, Jaime Melis wrote:
>> Hi Alvaro,
>>
>> thanks a lot for the contribution. I haven't tested it yet, but
>> the code seems to be perfect.
>>
>> My main issue here is that I'm not entirely sure if we need this.
>> Doesn't it make sense for the oneadmin user to be in the keyring
>> and be able to run rados? You mentioned this, but I'm having
>> some trouble understanding what do you mean:
>>
>> > This feature could be useful as well if you want to monitor
>> several datastores that are using different ceph pools and users ids.
>> > You only have to include the id and pool info into the ONE
>> datastore template and the monitoring script will use one or
>> another depending on the DS conf.
>>
>> With the current system you can monitor multiple ceph pools as
>> long as the oneadmin user has right, isn't that so?
>>
>> Joel, would you like to weigh in? would you find this useful?
>>
>> cheers,
>> Jaime
>>
>>
>> On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan
>> <jfontan at opennebula.org <mailto:jfontan at opennebula.org>> wrote:
>>
>> Great, thanks! Our ceph dev is traveling right now, he will
>> review the
>> patches when he arrives.
>>
>> Cheers
>>
>> On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
>> <Alvaro.SimonGarcia at ugent.be
>> <mailto:Alvaro.SimonGarcia at ugent.be>> wrote:
>> > Hi Javier
>> >
>> > We have modified the ceph monitor to take into account the
>> datastore ceph
>> > pool and also the ceph user. This is a generic solution
>> that could be useful
>> > for other datacenters as well, we have created a pull
>> request in github if
>> > you are agree about this change and you want to include it
>> in the next
>> > release.
>> >
>> > https://github.com/OpenNebula/one/pull/27
>> >
>> > We only have modified these lines in
>> > /var/lib/one/remotes/datastore/ceph/monitor:
>> >
>> >> --- monitor.orig.190614 2014-06-19 14:35:24.022755989 +0200
>> >> +++ monitor 2014-06-19 14:49:34.043187892 +0200
>> >> @@ -46,10 +46,12 @@
>> >> while IFS= read -r -d '' element; do
>> >> XPATH_ELEMENTS[i++]="$element"
>> >> done < <($XPATH
>> /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST \
>> >> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>> >>
>> >> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>> >> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>> >> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>> >>
>> >> HOST=`get_destination_host`
>> >>
>> >> @@ -61,7 +63,7 @@
>> >> # ------------ Compute datastore usage -------------
>> >>
>> >> MONITOR_SCRIPT=$(cat <<EOF
>> >> -$RADOS df | $AWK '{
>> >> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>> >> if (\$1 == "total") {
>> >>
>> >> space = int(\$3/1024)
>> >
>> >
>> > CEPH_USER and POOL_NAME should be mandatory to create the
>> ceph datastore.
>> >
>> > Cheers
>> > Alvaro
>> >
>> >
>> >
>> >> Hola Javi
>> >>
>> >> Thanks a lot for your feedback. Yes we will modify the current
>> >> monitoring scripts to take into account this. This feature
>> could be
>> >> useful as well if you want to monitor several datastores
>> that are using
>> >> different ceph pools and users ids. You only have to
>> include the id and
>> >> pool info into the ONE datastore template and the
>> monitoring script will
>> >> use one or another depending on the DS conf.
>> >>
>> >>
>> >> Cheers and thanks!
>> >> Alvaro
>> >> On 2014-06-17 14:55, Javier Fontan wrote:
>> >>>
>> >>> CEPH_USER is used when generating the libvirt/kvm
>> deployment file but
>> >>> not for DS monitoring:
>> >>>
>> >>> * Deployment file generation:
>> >>>
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>> >>> * Monitoring:
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>> >>>
>> >>> Ceph is not may area of expertise but you may need to add
>> those
>> >>> parameters to the monitor script a maybe to other scripts
>> that use the
>> >>> "rados" command. It may also be possible to modify the
>> RADOS command
>> >>> to have those parameters instead of modifying all the
>> scripts:
>> >>>
>> >>>
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>> >>>
>> >>> As I said I don't know much about Ceph and it may be
>> those credentials
>> >>> could be set in a config file or so.
>> >>>
>> >>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>> >>> <Alvaro.SimonGarcia at ugent.be
>> <mailto:Alvaro.SimonGarcia at ugent.be>> wrote:
>> >>>>
>> >>>> Hi
>> >>>>
>> >>>> We have included the admin keyring instead of libvirt
>> user and it
>> >>>> works...
>> >>>> that means that we can run rbd or qemu-img wihtout the
>> libvirt id, but
>> >>>> is
>> >>>> not the best solution. We have included the user into
>> datastore conf:
>> >>>>
>> >>>> CEPH_USER="libvirt"
>> >>>>
>> >>>> but it seems that is not used by opennebula at the end
>> >>>>
>> >>>> Cheers
>> >>>> Alvaro
>> >>>>
>> >>>>
>> >>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>> >>>>>
>> >>>>> Hi all
>> >>>>>
>> >>>>>
>> >>>>> We have included our ONE nodes into Ceph cluster, cephx
>> auth is working
>> >>>>> but OpenNebula is not able to detect the free space:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> $ onedatastore show 103
>> >>>>>> DATASTORE 103 INFORMATION
>> >>>>>> ID : 103
>> >>>>>> NAME : ceph
>> >>>>>> USER : oneadmin
>> >>>>>> GROUP : oneadmin
>> >>>>>> CLUSTER : -
>> >>>>>> TYPE : IMAGE
>> >>>>>> DS_MAD : ceph
>> >>>>>> TM_MAD : ceph
>> >>>>>> BASE PATH : /var/lib/one//datastores/103
>> >>>>>> DISK_TYPE : RBD
>> >>>>>>
>> >>>>>> DATASTORE CAPACITY
>> >>>>>> TOTAL: : 0M
>> >>>>>> FREE: : 0M
>> >>>>>> USED: : 0M
>> >>>>>> LIMIT: : -
>> >>>>>>
>> >>>>>> PERMISSIONS
>> >>>>>> OWNER : um-
>> >>>>>> GROUP : u--
>> >>>>>> OTHER : ---
>> >>>>>
>> >>>>>
>> >>>>>> $ onedatastore list
>> >>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS
>> >>>>>> TM
>> >>>>>> 0 system 114.8G 85% - 0 sys -
>> >>>>>> shared
>> >>>>>> 1 default 114.9G 84% - 2
>> img fs
>> >>>>>> ssh
>> >>>>>> 2 files 114.9G 84% - 0 fil fs
>> >>>>>> ssh
>> >>>>>> 103 ceph 0M - - 0 img
>> ceph
>> >>>>>> ceph
>> >>>>>
>> >>>>>
>> >>>>> but if we run rados as oneadmin user:
>> >>>>>
>> >>>>>> $ rados df -p one --id libvirt
>> >>>>>> pool name category KB objects clones
>> >>>>>> degraded unfound rd rd KB
>> wr wr
>> >>>>>> KB
>> >>>>>> one - 0 0 0
>> >>>>>> 0 0 0 0 0 0
>> >>>>>> total used 1581852 37
>> >>>>>> total avail 140846865180
>> >>>>>> total space 140848447032
>> >>>>>
>> >>>>>
>> >>>>> It's working correctly (we are using one pool and
>> libvirt ceph id)
>> >>>>>
>> >>>>> the oned.log only shows this info:
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore
>> default (1)
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore
>> files (2)
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore
>> ceph (103)
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default
>> (1) successfully
>> >>>>> monitored.
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2)
>> successfully
>> >>>>> monitored.
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103)
>> successfully
>> >>>>> monitored.
>> >>>>>
>> >>>>> Any clue about how to debug this issue?
>> >>>>>
>> >>>>> Thanks in advance!
>> >>>>> Alvaro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Users mailing list
>> >>>>> Users at lists.opennebula.org
>> <mailto:Users at lists.opennebula.org>
>> >>>>>
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >>>>
>> >>>> _______________________________________________
>> >>>> Users mailing list
>> >>>> Users at lists.opennebula.org
>> <mailto:Users at lists.opennebula.org>
>> >>>>
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >>>
>> >>>
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
>> >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >
>> >
>>
>>
>>
>> --
>> Javier Fontán Muiños
>> Developer
>> OpenNebula - Flexible Enterprise Cloud Made Simple
>> www.OpenNebula.org <http://www.OpenNebula.org> | @OpenNebula
>> | github.com/jfontan <http://github.com/jfontan>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>>
>>
>>
>> --
>> Jaime Melis
>> Project Engineer
>> OpenNebula - Flexible Enterprise Cloud Made Simple
>> www.OpenNebula.org <http://www.OpenNebula.org> |
>> jmelis at opennebula.org <mailto:jmelis at opennebula.org>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
>
>
> --
> --
> Ruben S. Montero, PhD
> Project co-Lead and Chief Architect
> OpenNebula - Flexible Enterprise Cloud Made Simple
> www.OpenNebula.org <http://www.OpenNebula.org> |
> rsmontero at opennebula.org <mailto:rsmontero at opennebula.org> | @OpenNebula
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20141010/30a3c44f/attachment-0001.htm>
More information about the Users
mailing list