[one-users] Ceph datastore monitoring in OpenNebula
Alvaro Simon Garcia
Alvaro.SimonGarcia at UGent.be
Thu Jun 19 06:21:21 PDT 2014
Hi Javier
We have modified the ceph monitor to take into account the datastore
ceph pool and also the ceph user. This is a generic solution that could
be useful for other datacenters as well, we have created a pull request
in github if you are agree about this change and you want to include it
in the next release.
https://github.com/OpenNebula/one/pull/27
We only have modified these lines in
/var/lib/one/remotes/datastore/ceph/monitor:
> --- monitor.orig.190614 2014-06-19 14:35:24.022755989 +0200
> +++ monitor 2014-06-19 14:49:34.043187892 +0200
> @@ -46,10 +46,12 @@
> while IFS= read -r -d '' element; do
> XPATH_ELEMENTS[i++]="$element"
> done < <($XPATH /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST \
> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>
> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>
> HOST=`get_destination_host`
>
> @@ -61,7 +63,7 @@
> # ------------ Compute datastore usage -------------
>
> MONITOR_SCRIPT=$(cat <<EOF
> -$RADOS df | $AWK '{
> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
> if (\$1 == "total") {
>
> space = int(\$3/1024)
CEPH_USER and POOL_NAME should be mandatory to create the ceph datastore.
Cheers
Alvaro
> Hola Javi
>
> Thanks a lot for your feedback. Yes we will modify the current
> monitoring scripts to take into account this. This feature could be
> useful as well if you want to monitor several datastores that are using
> different ceph pools and users ids. You only have to include the id and
> pool info into the ONE datastore template and the monitoring script will
> use one or another depending on the DS conf.
>
>
> Cheers and thanks!
> Alvaro
> On 2014-06-17 14:55, Javier Fontan wrote:
>> CEPH_USER is used when generating the libvirt/kvm deployment file but
>> not for DS monitoring:
>>
>> * Deployment file generation:
>> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>> * Monitoring: https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>>
>> Ceph is not may area of expertise but you may need to add those
>> parameters to the monitor script a maybe to other scripts that use the
>> "rados" command. It may also be possible to modify the RADOS command
>> to have those parameters instead of modifying all the scripts:
>>
>> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>>
>> As I said I don't know much about Ceph and it may be those credentials
>> could be set in a config file or so.
>>
>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>> <Alvaro.SimonGarcia at ugent.be> wrote:
>>> Hi
>>>
>>> We have included the admin keyring instead of libvirt user and it works...
>>> that means that we can run rbd or qemu-img wihtout the libvirt id, but is
>>> not the best solution. We have included the user into datastore conf:
>>>
>>> CEPH_USER="libvirt"
>>>
>>> but it seems that is not used by opennebula at the end
>>>
>>> Cheers
>>> Alvaro
>>>
>>>
>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>>>> Hi all
>>>>
>>>>
>>>> We have included our ONE nodes into Ceph cluster, cephx auth is working
>>>> but OpenNebula is not able to detect the free space:
>>>>
>>>>
>>>>
>>>>> $ onedatastore show 103
>>>>> DATASTORE 103 INFORMATION
>>>>> ID : 103
>>>>> NAME : ceph
>>>>> USER : oneadmin
>>>>> GROUP : oneadmin
>>>>> CLUSTER : -
>>>>> TYPE : IMAGE
>>>>> DS_MAD : ceph
>>>>> TM_MAD : ceph
>>>>> BASE PATH : /var/lib/one//datastores/103
>>>>> DISK_TYPE : RBD
>>>>>
>>>>> DATASTORE CAPACITY
>>>>> TOTAL: : 0M
>>>>> FREE: : 0M
>>>>> USED: : 0M
>>>>> LIMIT: : -
>>>>>
>>>>> PERMISSIONS
>>>>> OWNER : um-
>>>>> GROUP : u--
>>>>> OTHER : ---
>>>>
>>>>> $ onedatastore list
>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS TM
>>>>> 0 system 114.8G 85% - 0 sys -
>>>>> shared
>>>>> 1 default 114.9G 84% - 2 img fs ssh
>>>>> 2 files 114.9G 84% - 0 fil fs ssh
>>>>> 103 ceph 0M - - 0 img ceph
>>>>> ceph
>>>>
>>>> but if we run rados as oneadmin user:
>>>>
>>>>> $ rados df -p one --id libvirt
>>>>> pool name category KB objects clones
>>>>> degraded unfound rd rd KB wr wr KB
>>>>> one - 0 0 0
>>>>> 0 0 0 0 0 0
>>>>> total used 1581852 37
>>>>> total avail 140846865180
>>>>> total space 140848447032
>>>>
>>>> It's working correctly (we are using one pool and libvirt ceph id)
>>>>
>>>> the oned.log only shows this info:
>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore default (1)
>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore files (2)
>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore ceph (103)
>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default (1) successfully
>>>> monitored.
>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2) successfully
>>>> monitored.
>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103) successfully
>>>> monitored.
>>>>
>>>> Any clue about how to debug this issue?
>>>>
>>>> Thanks in advance!
>>>> Alvaro
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opennebula.org
>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>> _______________________________________________
>>> Users mailing list
>>> Users at lists.opennebula.org
>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
More information about the Users
mailing list