[one-users] Ceph datastore monitoring in OpenNebula
Alvaro Simon Garcia
Alvaro.SimonGarcia at UGent.be
Mon Aug 11 01:43:16 PDT 2014
Hi Jaime
On 11/08/14 10:28, Jaime Melis wrote:
> By the way, unfortunately this won't make it 4.8 as it's already
> feature freeze (and about to be released). I know you reported this a
> while back and in the end we didn't include it because we weren't
> sure. Sorry for not revisiting this issue earlier :S
>
> Before providing the patch for the docs, let me discuss this with the
> team, see if we can include it for 4.8.1
Ok don't worry, we will include everything into a single new pull
request to simplify the patch.
Cheers
Alvaro
> ...
>
> cheers,
> Jaime
>
> On Mon, Aug 11, 2014 at 10:25 AM, Jaime Melis <jmelis at opennebula.org
> <mailto:jmelis at opennebula.org>> wrote:
>
> Hi Alvaro,
>
> Could you also provide a patch for the documentation?
>
>
> On Mon, Aug 11, 2014 at 10:22 AM, Alvaro Simon Garcia
> <Alvaro.SimonGarcia at ugent.be <mailto:Alvaro.SimonGarcia at ugent.be>>
> wrote:
>
> Hi Jaime, Javier
>
> Do you think that it would be possible to include these ceph
> patches into next 4.8 release? At this moment we are applying
> theses patches by hand in our configuration to use different
> Ceph pools.
>
> https://github.com/OpenNebula/one/pull/27
> https://github.com/OpenNebula/one/pull/28
> https://github.com/OpenNebula/one/pull/29
> https://github.com/OpenNebula/one/pull/30
> https://github.com/OpenNebula/one/pull/31
>
> Cheers
> Alvaro
>
>
> On 17/07/14 14:11, Alvaro Simon Garcia wrote:
>> Hi Jaime
>>
>> Sorry for the late reply, I didn't see your mail before
>> because I was included in cc. With this patch you don't need
>> to include a new user into your ceph keyring. If you want to
>> use a ceph datastore you only need to include the user and
>> pool into the datastore template that's all. In our case we
>> use livbirt datastore and we have created two ceph datastore
>> for testing porposes (each one use a different pool as well):
>>
>>
>> $ onedatastore show ceph
>> DATASTORE 103 INFORMATION
>> ID : 103
>> NAME : ceph
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : -
>> TYPE : IMAGE
>> DS_MAD : ceph
>> TM_MAD : ceph
>> BASE PATH : /var/lib/one//datastores/103
>> DISK_TYPE : RBD
>>
>> DATASTORE CAPACITY
>> TOTAL: : 87.6T
>> FREE: : 59.2T
>> USED: : 28.4T
>> LIMIT: : -
>>
>> PERMISSIONS
>> OWNER : um-
>> GROUP : u--
>> OTHER : ---
>>
>> DATASTORE TEMPLATE
>> BASE_PATH="/var/lib/one//datastores/"
>> BRIDGE_LIST="hyp004.cubone.os"
>> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
>> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> CEPH_USER="libvirt"
>> CLONE_TARGET="SELF"
>> DATASTORE_CAPACITY_CHECK="yes"
>> DISK_TYPE="RBD"
>> DS_MAD="ceph"
>> LN_TARGET="NONE"
>> POOL_NAME="one"
>> TM_MAD="ceph"
>>
>> IMAGES
>> 7
>> 8
>> 31
>> 32
>> 34
>> 35
>>
>>
>> and the second one:
>>
>> $ onedatastore show "ceph two"
>> DATASTORE 104 INFORMATION
>> ID : 104
>> NAME : ceph two
>> USER : oneadmin
>> GROUP : oneadmin
>> CLUSTER : -
>> TYPE : IMAGE
>> DS_MAD : ceph
>> TM_MAD : ceph
>> BASE PATH : /var/lib/one//datastores/104
>> DISK_TYPE : RBD
>>
>> DATASTORE CAPACITY
>> TOTAL: : 87.6T
>> FREE: : 59.2T
>> USED: : 28.4T
>> LIMIT: : -
>>
>> PERMISSIONS
>> OWNER : um-
>> GROUP : u--
>> OTHER : ---
>>
>> DATASTORE TEMPLATE
>> BASE_PATH="/var/lib/one//datastores/"
>> BRIDGE_LIST="hyp004.cubone.os"
>> CEPH_HOST="ceph001.cubone.os ceph002.cubone.os ceph003.cubone.os"
>> CEPH_SECRET="xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
>> CEPH_USER="libvirt"
>> CLONE_TARGET="SELF"
>> DATASTORE_CAPACITY_CHECK="yes"
>> DISK_TYPE="RBD"
>> DS_MAD="ceph"
>> LN_TARGET="NONE"
>> POOL_NAME="two"
>> TM_MAD="ceph"
>>
>> As you can see we are using different pools in each one (so
>> we don't need to include the pool name into ceph.conf either,
>> and we are able to use several ceph clusters) and this change
>> simplifies the ONE conf and ceph cluster admin as well.
>>
>> Cheers
>> Alvaro
>>
>> On 03/07/14 17:08, Jaime Melis wrote:
>>> Hi Alvaro,
>>>
>>> thanks a lot for the contribution. I haven't tested it yet,
>>> but the code seems to be perfect.
>>>
>>> My main issue here is that I'm not entirely sure if we need
>>> this. Doesn't it make sense for the oneadmin user to be in
>>> the keyring and be able to run rados? You mentioned this,
>>> but I'm having some trouble understanding what do you mean:
>>>
>>> > This feature could be useful as well if you want to monitor
>>> several datastores that are using different ceph pools and
>>> users ids.
>>> > You only have to include the id and pool info into the ONE
>>> datastore template and the monitoring script will use one or
>>> another depending on the DS conf.
>>>
>>> With the current system you can monitor multiple ceph pools
>>> as long as the oneadmin user has right, isn't that so?
>>>
>>> Joel, would you like to weigh in? would you find this useful?
>>>
>>> cheers,
>>> Jaime
>>>
>>>
>>> On Fri, Jun 20, 2014 at 3:23 PM, Javier Fontan
>>> <jfontan at opennebula.org <mailto:jfontan at opennebula.org>> wrote:
>>>
>>> Great, thanks! Our ceph dev is traveling right now, he
>>> will review the
>>> patches when he arrives.
>>>
>>> Cheers
>>>
>>> On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
>>> <Alvaro.SimonGarcia at ugent.be
>>> <mailto:Alvaro.SimonGarcia at ugent.be>> wrote:
>>> > Hi Javier
>>> >
>>> > We have modified the ceph monitor to take into account
>>> the datastore ceph
>>> > pool and also the ceph user. This is a generic
>>> solution that could be useful
>>> > for other datacenters as well, we have created a pull
>>> request in github if
>>> > you are agree about this change and you want to
>>> include it in the next
>>> > release.
>>> >
>>> > https://github.com/OpenNebula/one/pull/27
>>> >
>>> > We only have modified these lines in
>>> > /var/lib/one/remotes/datastore/ceph/monitor:
>>> >
>>> >> --- monitor.orig.190614 2014-06-19
>>> 14:35:24.022755989 +0200
>>> >> +++ monitor 2014-06-19 14:49:34.043187892 +0200
>>> >> @@ -46,10 +46,12 @@
>>> >> while IFS= read -r -d '' element; do
>>> >> XPATH_ELEMENTS[i++]="$element"
>>> >> done < <($XPATH
>>> /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST \
>>> >> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>>> >> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>>> >>
>>> >> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>>> >> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>>> >> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>>> >>
>>> >> HOST=`get_destination_host`
>>> >>
>>> >> @@ -61,7 +63,7 @@
>>> >> # ------------ Compute datastore usage -------------
>>> >>
>>> >> MONITOR_SCRIPT=$(cat <<EOF
>>> >> -$RADOS df | $AWK '{
>>> >> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>>> >> if (\$1 == "total") {
>>> >>
>>> >> space = int(\$3/1024)
>>> >
>>> >
>>> > CEPH_USER and POOL_NAME should be mandatory to create
>>> the ceph datastore.
>>> >
>>> > Cheers
>>> > Alvaro
>>> >
>>> >
>>> >
>>> >> Hola Javi
>>> >>
>>> >> Thanks a lot for your feedback. Yes we will modify
>>> the current
>>> >> monitoring scripts to take into account this. This
>>> feature could be
>>> >> useful as well if you want to monitor several
>>> datastores that are using
>>> >> different ceph pools and users ids. You only have to
>>> include the id and
>>> >> pool info into the ONE datastore template and the
>>> monitoring script will
>>> >> use one or another depending on the DS conf.
>>> >>
>>> >>
>>> >> Cheers and thanks!
>>> >> Alvaro
>>> >> On 2014-06-17 14:55, Javier Fontan wrote:
>>> >>>
>>> >>> CEPH_USER is used when generating the libvirt/kvm
>>> deployment file but
>>> >>> not for DS monitoring:
>>> >>>
>>> >>> * Deployment file generation:
>>> >>>
>>> >>>
>>> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>>> >>> * Monitoring:
>>> >>>
>>> https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>>> >>>
>>> >>> Ceph is not may area of expertise but you may need
>>> to add those
>>> >>> parameters to the monitor script a maybe to other
>>> scripts that use the
>>> >>> "rados" command. It may also be possible to modify
>>> the RADOS command
>>> >>> to have those parameters instead of modifying all
>>> the scripts:
>>> >>>
>>> >>>
>>> >>>
>>> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>>> >>>
>>> >>> As I said I don't know much about Ceph and it may be
>>> those credentials
>>> >>> could be set in a config file or so.
>>> >>>
>>> >>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>>> >>> <Alvaro.SimonGarcia at ugent.be
>>> <mailto:Alvaro.SimonGarcia at ugent.be>> wrote:
>>> >>>>
>>> >>>> Hi
>>> >>>>
>>> >>>> We have included the admin keyring instead of
>>> libvirt user and it
>>> >>>> works...
>>> >>>> that means that we can run rbd or qemu-img wihtout
>>> the libvirt id, but
>>> >>>> is
>>> >>>> not the best solution. We have included the user
>>> into datastore conf:
>>> >>>>
>>> >>>> CEPH_USER="libvirt"
>>> >>>>
>>> >>>> but it seems that is not used by opennebula at the end
>>> >>>>
>>> >>>> Cheers
>>> >>>> Alvaro
>>> >>>>
>>> >>>>
>>> >>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>>> >>>>>
>>> >>>>> Hi all
>>> >>>>>
>>> >>>>>
>>> >>>>> We have included our ONE nodes into Ceph cluster,
>>> cephx auth is working
>>> >>>>> but OpenNebula is not able to detect the free space:
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>>> $ onedatastore show 103
>>> >>>>>> DATASTORE 103 INFORMATION
>>> >>>>>> ID : 103
>>> >>>>>> NAME : ceph
>>> >>>>>> USER : oneadmin
>>> >>>>>> GROUP : oneadmin
>>> >>>>>> CLUSTER : -
>>> >>>>>> TYPE : IMAGE
>>> >>>>>> DS_MAD : ceph
>>> >>>>>> TM_MAD : ceph
>>> >>>>>> BASE PATH : /var/lib/one//datastores/103
>>> >>>>>> DISK_TYPE : RBD
>>> >>>>>>
>>> >>>>>> DATASTORE CAPACITY
>>> >>>>>> TOTAL: : 0M
>>> >>>>>> FREE: : 0M
>>> >>>>>> USED: : 0M
>>> >>>>>> LIMIT: : -
>>> >>>>>>
>>> >>>>>> PERMISSIONS
>>> >>>>>> OWNER : um-
>>> >>>>>> GROUP : u--
>>> >>>>>> OTHER : ---
>>> >>>>>
>>> >>>>>
>>> >>>>>> $ onedatastore list
>>> >>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS
>>> >>>>>> TM
>>> >>>>>> 0 system 114.8G 85% - 0 sys -
>>> >>>>>> shared
>>> >>>>>> 1 default 114.9G 84% - 2 img fs
>>> >>>>>> ssh
>>> >>>>>> 2 files 114.9G 84% - 0 fil fs
>>> >>>>>> ssh
>>> >>>>>> 103 ceph 0M - - 0 img ceph
>>> >>>>>> ceph
>>> >>>>>
>>> >>>>>
>>> >>>>> but if we run rados as oneadmin user:
>>> >>>>>
>>> >>>>>> $ rados df -p one --id libvirt
>>> >>>>>> pool name category KB
>>> objects clones
>>> >>>>>> degraded unfound rd rd KB
>>> wr wr
>>> >>>>>> KB
>>> >>>>>> one - 0 0 0
>>> >>>>>> 0 0 0 0 0
>>> 0
>>> >>>>>> total used 1581852 37
>>> >>>>>> total avail 140846865180
>>> >>>>>> total space 140848447032
>>> >>>>>
>>> >>>>>
>>> >>>>> It's working correctly (we are using one pool and
>>> libvirt ceph id)
>>> >>>>>
>>> >>>>> the oned.log only shows this info:
>>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring
>>> datastore default (1)
>>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring
>>> datastore files (2)
>>> >>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring
>>> datastore ceph (103)
>>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore
>>> default (1) successfully
>>> >>>>> monitored.
>>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files
>>> (2) successfully
>>> >>>>> monitored.
>>> >>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph
>>> (103) successfully
>>> >>>>> monitored.
>>> >>>>>
>>> >>>>> Any clue about how to debug this issue?
>>> >>>>>
>>> >>>>> Thanks in advance!
>>> >>>>> Alvaro
>>> >>>>>
>>> >>>>>
>>> >>>>>
>>> >>>>> _______________________________________________
>>> >>>>> Users mailing list
>>> >>>>> Users at lists.opennebula.org
>>> <mailto:Users at lists.opennebula.org>
>>> >>>>>
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>> >>>>
>>> >>>> _______________________________________________
>>> >>>> Users mailing list
>>> >>>> Users at lists.opennebula.org
>>> <mailto:Users at lists.opennebula.org>
>>> >>>>
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>> >>>
>>> >>>
>>> >> _______________________________________________
>>> >> Users mailing list
>>> >> Users at lists.opennebula.org
>>> <mailto:Users at lists.opennebula.org>
>>> >>
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>> >
>>> >
>>>
>>>
>>>
>>> --
>>> Javier Fontán Muiños
>>> Developer
>>> OpenNebula - Flexible Enterprise Cloud Made Simple
>>> www.OpenNebula.org <http://www.OpenNebula.org> |
>>> @OpenNebula | github.com/jfontan <http://github.com/jfontan>
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opennebula.org
>>> <mailto:Users at lists.opennebula.org>
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>>
>>>
>>>
>>> --
>>> Jaime Melis
>>> Project Engineer
>>> OpenNebula - Flexible Enterprise Cloud Made Simple
>>> www.OpenNebula.org <http://www.OpenNebula.org> |
>>> jmelis at opennebula.org <mailto:jmelis at opennebula.org>
>>
>
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org <mailto:Users at lists.opennebula.org>
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
>
>
> --
> Jaime Melis
> Project Engineer
> OpenNebula - Flexible Enterprise Cloud Made Simple
> www.OpenNebula.org <http://www.OpenNebula.org> |
> jmelis at opennebula.org <mailto:jmelis at opennebula.org>
>
>
>
>
> --
> Jaime Melis
> Project Engineer
> OpenNebula - Flexible Enterprise Cloud Made Simple
> www.OpenNebula.org <http://www.OpenNebula.org> | jmelis at opennebula.org
> <mailto:jmelis at opennebula.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140811/e6a5824e/attachment-0001.htm>
More information about the Users
mailing list