[one-users] Ceph datastore monitoring in OpenNebula
Jaime Melis
jmelis at opennebula.org
Mon Aug 11 01:25:36 PDT 2014
Hi Alvaro,
Could you also provide a patch for the documentation?
On Mon, Aug 11, 2014 at 10:22 AM, Alvaro Simon Garcia <
Alvaro.SimonGarcia at ugent.be> wrote:
> Hi Jaime, Javier
>
> Do you think that it would be possible to include these ceph patches into
> next 4.8 release? At this moment we are applying theses patches by hand in
> our configuration to use different Ceph pools.
>
> https://github.com/OpenNebula/one/pull/27
> https://github.com/OpenNebula/one/pull/28
> https://github.com/OpenNebula/one/pull/29
> https://github.com/OpenNebula/one/pull/30
> https://github.com/OpenNebula/one/pull/31
>
> Cheers
> Alvaro
>
>
> On 17/07/14 14:11, Alvaro Simon Garcia wrote:
>
> Hi Jaime
>
> Sorry for the late reply, I didn't see your mail before because I was
> included in cc. With this patch you don't need to include a new user into
> your ceph keyring. If you want to use a ceph datastore you only need to
> include the user and pool into the datastore template that's all. In our
> case we use livbirt datastore and we have created two ceph datastore for
> testing porposes (each one use a different pool as well):
>
>
> $ onedatastore show ceph
> DATASTORE 103
> INFORMATION
> ID : 103
> NAME : ceph
> USER : oneadmin
> GROUP : oneadmin
> CLUSTER : -
> TYPE : IMAGE
> DS_MAD : ceph
> TM_MAD : ceph
> BASE PATH : /var/lib/one//datastores/103
> DISK_TYPE : RBD
>
> DATASTORE
> CAPACITY
> TOTAL: : 87.6T
> FREE: : 59.2T
> USED: : 28.4T
> LIMIT: : -
>
> PERMISSIONS
>
> OWNER : um-
> GROUP : u--
> OTHER : ---
>
> DATASTORE
> TEMPLATE
> BASE_PATH="/var/lib/one//datastores/"
> BRIDGE_LIST="hyp004.cubone.os"
> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> CEPH_USER="libvirt"
> CLONE_TARGET="SELF"
> DATASTORE_CAPACITY_CHECK="yes"
> DISK_TYPE="RBD"
> DS_MAD="ceph"
> LN_TARGET="NONE"
> POOL_NAME="one"
> TM_MAD="ceph"
>
> IMAGES
> 7
> 8
> 31
> 32
> 34
> 35
>
>
> and the second one:
>
> $ onedatastore show "ceph two"
> DATASTORE 104
> INFORMATION
> ID : 104
> NAME : ceph two
> USER : oneadmin
> GROUP : oneadmin
> CLUSTER : -
> TYPE : IMAGE
> DS_MAD : ceph
> TM_MAD : ceph
> BASE PATH : /var/lib/one//datastores/104
> DISK_TYPE : RBD
>
> DATASTORE
> CAPACITY
> TOTAL: : 87.6T
> FREE: : 59.2T
> USED: : 28.4T
> LIMIT: : -
>
> PERMISSIONS
>
> OWNER : um-
> GROUP : u--
> OTHER : ---
>
> DATASTORE
> TEMPLATE
> BASE_PATH="/var/lib/one//datastores/"
> BRIDGE_LIST="hyp004.cubone.os"
> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
> CEPH_USER="libvirt"
> CLONE_TARGET="SELF"
> DATASTORE_CAPACITY_CHECK="yes"
> DISK_TYPE="RBD"
> DS_MAD="ceph"
> LN_TARGET="NONE"
> POOL_NAME="two"
> TM_MAD="ceph"
>
> As you can see we are using different pools in each one (so we don't need
> to include the pool name into ceph.conf either, and we are able to use
> several ceph clusters) and this change simplifies the ONE conf and ceph
> cluster admin as well.
>
> Cheers
> Alvaro
>
> On 03/07/14 17:08, Jaime Melis wrote:
>
> Hi Alvaro,
>
> thanks a lot for the contribution. I haven't tested it yet, but the code
> seems to be perfect.
>
> My main issue here is that I'm not entirely sure if we need this.
> Doesn't it make sense for the oneadmin user to be in the keyring and be
> able to run rados? You mentioned this, but I'm having some trouble
> understanding what do you mean:
>
> > This feature could be useful as well if you want to monitor several
> datastores that are using different ceph pools and users ids.
> > You only have to include the id and pool info into the ONE datastore template
> and the monitoring script will use one or another depending on the DS
> conf.
>
> With the current system you can monitor multiple ceph pools as long as
> the oneadmin user has right, isn't that so?
>
> Joel, would you like to weigh in? would you find this useful?
>
> cheers,
> Jaime
>
>
> On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan <jfontan at opennebula.org>
> wrote:
>
>> Great, thanks! Our ceph dev is traveling right now, he will review the
>> patches when he arrives.
>>
>> Cheers
>>
>> On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
>> <Alvaro.SimonGarcia at ugent.be> wrote:
>> > Hi Javier
>> >
>> > We have modified the ceph monitor to take into account the datastore
>> ceph
>> > pool and also the ceph user. This is a generic solution that could be
>> useful
>> > for other datacenters as well, we have created a pull request in github
>> if
>> > you are agree about this change and you want to include it in the next
>> > release.
>> >
>> > https://github.com/OpenNebula/one/pull/27
>> >
>> > We only have modified these lines in
>> > /var/lib/one/remotes/datastore/ceph/monitor:
>> >
>> >> --- monitor.orig.190614 2014-06-19 14:35:24.022755989 +0200
>> >> +++ monitor 2014-06-19 14:49:34.043187892 +0200
>> >> @@ -46,10 +46,12 @@
>> >> while IFS= read -r -d '' element; do
>> >> XPATH_ELEMENTS[i++]="$element"
>> >> done < <($XPATH /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST
>> \
>> >> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>> >>
>> >> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>> >> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>> >> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>> >>
>> >> HOST=`get_destination_host`
>> >>
>> >> @@ -61,7 +63,7 @@
>> >> # ------------ Compute datastore usage -------------
>> >>
>> >> MONITOR_SCRIPT=$(cat <<EOF
>> >> -$RADOS df | $AWK '{
>> >> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>> >> if (\$1 == "total") {
>> >>
>> >> space = int(\$3/1024)
>> >
>> >
>> > CEPH_USER and POOL_NAME should be mandatory to create the ceph
>> datastore.
>> >
>> > Cheers
>> > Alvaro
>> >
>> >
>> >
>> >> Hola Javi
>> >>
>> >> Thanks a lot for your feedback. Yes we will modify the current
>> >> monitoring scripts to take into account this. This feature could be
>> >> useful as well if you want to monitor several datastores that are using
>> >> different ceph pools and users ids. You only have to include the id and
>> >> pool info into the ONE datastore template and the monitoring script
>> will
>> >> use one or another depending on the DS conf.
>> >>
>> >>
>> >> Cheers and thanks!
>> >> Alvaro
>> >> On 2014-06-17 14:55, Javier Fontan wrote:
>> >>>
>> >>> CEPH_USER is used when generating the libvirt/kvm deployment file but
>> >>> not for DS monitoring:
>> >>>
>> >>> * Deployment file generation:
>> >>>
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>> >>> * Monitoring:
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>> >>>
>> >>> Ceph is not may area of expertise but you may need to add those
>> >>> parameters to the monitor script a maybe to other scripts that use the
>> >>> "rados" command. It may also be possible to modify the RADOS command
>> >>> to have those parameters instead of modifying all the scripts:
>> >>>
>> >>>
>> >>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>> >>>
>> >>> As I said I don't know much about Ceph and it may be those credentials
>> >>> could be set in a config file or so.
>> >>>
>> >>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>> >>> <Alvaro.SimonGarcia at ugent.be> wrote:
>> >>>>
>> >>>> Hi
>> >>>>
>> >>>> We have included the admin keyring instead of libvirt user and it
>> >>>> works...
>> >>>> that means that we can run rbd or qemu-img wihtout the libvirt id,
>> but
>> >>>> is
>> >>>> not the best solution. We have included the user into datastore conf:
>> >>>>
>> >>>> CEPH_USER="libvirt"
>> >>>>
>> >>>> but it seems that is not used by opennebula at the end
>> >>>>
>> >>>> Cheers
>> >>>> Alvaro
>> >>>>
>> >>>>
>> >>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>> >>>>>
>> >>>>> Hi all
>> >>>>>
>> >>>>>
>> >>>>> We have included our ONE nodes into Ceph cluster, cephx auth is
>> working
>> >>>>> but OpenNebula is not able to detect the free space:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>> $ onedatastore show 103
>> >>>>>> DATASTORE 103 INFORMATION
>> >>>>>> ID : 103
>> >>>>>> NAME : ceph
>> >>>>>> USER : oneadmin
>> >>>>>> GROUP : oneadmin
>> >>>>>> CLUSTER : -
>> >>>>>> TYPE : IMAGE
>> >>>>>> DS_MAD : ceph
>> >>>>>> TM_MAD : ceph
>> >>>>>> BASE PATH : /var/lib/one//datastores/103
>> >>>>>> DISK_TYPE : RBD
>> >>>>>>
>> >>>>>> DATASTORE CAPACITY
>> >>>>>> TOTAL: : 0M
>> >>>>>> FREE: : 0M
>> >>>>>> USED: : 0M
>> >>>>>> LIMIT: : -
>> >>>>>>
>> >>>>>> PERMISSIONS
>> >>>>>> OWNER : um-
>> >>>>>> GROUP : u--
>> >>>>>> OTHER : ---
>> >>>>>
>> >>>>>
>> >>>>>> $ onedatastore list
>> >>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS
>> >>>>>> TM
>> >>>>>> 0 system 114.8G 85% - 0 sys -
>> >>>>>> shared
>> >>>>>> 1 default 114.9G 84% - 2 img fs
>> >>>>>> ssh
>> >>>>>> 2 files 114.9G 84% - 0 fil fs
>> >>>>>> ssh
>> >>>>>> 103 ceph 0M - - 0 img ceph
>> >>>>>> ceph
>> >>>>>
>> >>>>>
>> >>>>> but if we run rados as oneadmin user:
>> >>>>>
>> >>>>>> $ rados df -p one --id libvirt
>> >>>>>> pool name category KB objects clones
>> >>>>>> degraded unfound rd rd KB wr
>> wr
>> >>>>>> KB
>> >>>>>> one - 0 0 0
>> >>>>>> 0 0 0 0 0 0
>> >>>>>> total used 1581852 37
>> >>>>>> total avail 140846865180
>> >>>>>> total space 140848447032
>> >>>>>
>> >>>>>
>> >>>>> It's working correctly (we are using one pool and libvirt ceph id)
>> >>>>>
>> >>>>> the oned.log only shows this info:
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore default (1)
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore files (2)
>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore ceph (103)
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default (1)
>> successfully
>> >>>>> monitored.
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2) successfully
>> >>>>> monitored.
>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103) successfully
>> >>>>> monitored.
>> >>>>>
>> >>>>> Any clue about how to debug this issue?
>> >>>>>
>> >>>>> Thanks in advance!
>> >>>>> Alvaro
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> _______________________________________________
>> >>>>> Users mailing list
>> >>>>> Users at lists.opennebula.org
>> >>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >>>>
>> >>>> _______________________________________________
>> >>>> Users mailing list
>> >>>> Users at lists.opennebula.org
>> >>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >>>
>> >>>
>> >> _______________________________________________
>> >> Users mailing list
>> >> Users at lists.opennebula.org
>> >> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> >
>> >
>>
>>
>>
>> --
>> Javier Fontán Muiños
>> Developer
>> OpenNebula - Flexible Enterprise Cloud Made Simple
>> www.OpenNebula.org | @OpenNebula | github.com/jfontan
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
>
>
>
> --
> Jaime Melis
> Project Engineer
> OpenNebula - Flexible Enterprise Cloud Made Simple
> www.OpenNebula.org | jmelis at opennebula.org
>
>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
--
Jaime Melis
Project Engineer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org | jmelis at opennebula.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140811/520af1c3/attachment-0001.htm>
More information about the Users
mailing list