[one-users] Ceph datastore monitoring in OpenNebula
Javier Fontan
jfontan at opennebula.org
Fri Jun 20 06:23:14 PDT 2014
Great, thanks! Our ceph dev is traveling right now, he will review the
patches when he arrives.
Cheers
On Thu, Jun 19, 2014 at 3:21 PM, Alvaro Simon Garcia
<Alvaro.SimonGarcia at ugent.be> wrote:
> Hi Javier
>
> We have modified the ceph monitor to take into account the datastore ceph
> pool and also the ceph user. This is a generic solution that could be useful
> for other datacenters as well, we have created a pull request in github if
> you are agree about this change and you want to include it in the next
> release.
>
> https://github.com/OpenNebula/one/pull/27
>
> We only have modified these lines in
> /var/lib/one/remotes/datastore/ceph/monitor:
>
>> --- monitor.orig.190614 2014-06-19 14:35:24.022755989 +0200
>> +++ monitor 2014-06-19 14:49:34.043187892 +0200
>> @@ -46,10 +46,12 @@
>> while IFS= read -r -d '' element; do
>> XPATH_ELEMENTS[i++]="$element"
>> done < <($XPATH /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/BRIDGE_LIST \
>> - /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME)
>> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/POOL_NAME \
>> + /DS_DRIVER_ACTION_DATA/DATASTORE/TEMPLATE/CEPH_USER)
>>
>> BRIDGE_LIST="${XPATH_ELEMENTS[j++]}"
>> POOL_NAME="${XPATH_ELEMENTS[j++]:-$POOL_NAME}"
>> +CEPH_USER="${XPATH_ELEMENTS[j++]}"
>>
>> HOST=`get_destination_host`
>>
>> @@ -61,7 +63,7 @@
>> # ------------ Compute datastore usage -------------
>>
>> MONITOR_SCRIPT=$(cat <<EOF
>> -$RADOS df | $AWK '{
>> +$RADOS df -p ${POOL_NAME} --id ${CEPH_USER}| $AWK '{
>> if (\$1 == "total") {
>>
>> space = int(\$3/1024)
>
>
> CEPH_USER and POOL_NAME should be mandatory to create the ceph datastore.
>
> Cheers
> Alvaro
>
>
>
>> Hola Javi
>>
>> Thanks a lot for your feedback. Yes we will modify the current
>> monitoring scripts to take into account this. This feature could be
>> useful as well if you want to monitor several datastores that are using
>> different ceph pools and users ids. You only have to include the id and
>> pool info into the ONE datastore template and the monitoring script will
>> use one or another depending on the DS conf.
>>
>>
>> Cheers and thanks!
>> Alvaro
>> On 2014-06-17 14:55, Javier Fontan wrote:
>>>
>>> CEPH_USER is used when generating the libvirt/kvm deployment file but
>>> not for DS monitoring:
>>>
>>> * Deployment file generation:
>>>
>>> https://github.com/OpenNebula/one/blob/one-4.6/src/vmm/LibVirtDriverKVM.cc#L461
>>> * Monitoring:
>>> https://github.com/OpenNebula/one/blob/one-4.6/src/datastore_mad/remotes/ceph/monitor#L64
>>>
>>> Ceph is not may area of expertise but you may need to add those
>>> parameters to the monitor script a maybe to other scripts that use the
>>> "rados" command. It may also be possible to modify the RADOS command
>>> to have those parameters instead of modifying all the scripts:
>>>
>>>
>>> https://github.com/OpenNebula/one/blob/one-4.6/src/mad/sh/scripts_common.sh#L40
>>>
>>> As I said I don't know much about Ceph and it may be those credentials
>>> could be set in a config file or so.
>>>
>>> On Tue, Jun 17, 2014 at 11:19 AM, Alvaro Simon Garcia
>>> <Alvaro.SimonGarcia at ugent.be> wrote:
>>>>
>>>> Hi
>>>>
>>>> We have included the admin keyring instead of libvirt user and it
>>>> works...
>>>> that means that we can run rbd or qemu-img wihtout the libvirt id, but
>>>> is
>>>> not the best solution. We have included the user into datastore conf:
>>>>
>>>> CEPH_USER="libvirt"
>>>>
>>>> but it seems that is not used by opennebula at the end
>>>>
>>>> Cheers
>>>> Alvaro
>>>>
>>>>
>>>> On 2014-06-17 10:09, Alvaro Simon Garcia wrote:
>>>>>
>>>>> Hi all
>>>>>
>>>>>
>>>>> We have included our ONE nodes into Ceph cluster, cephx auth is working
>>>>> but OpenNebula is not able to detect the free space:
>>>>>
>>>>>
>>>>>
>>>>>> $ onedatastore show 103
>>>>>> DATASTORE 103 INFORMATION
>>>>>> ID : 103
>>>>>> NAME : ceph
>>>>>> USER : oneadmin
>>>>>> GROUP : oneadmin
>>>>>> CLUSTER : -
>>>>>> TYPE : IMAGE
>>>>>> DS_MAD : ceph
>>>>>> TM_MAD : ceph
>>>>>> BASE PATH : /var/lib/one//datastores/103
>>>>>> DISK_TYPE : RBD
>>>>>>
>>>>>> DATASTORE CAPACITY
>>>>>> TOTAL: : 0M
>>>>>> FREE: : 0M
>>>>>> USED: : 0M
>>>>>> LIMIT: : -
>>>>>>
>>>>>> PERMISSIONS
>>>>>> OWNER : um-
>>>>>> GROUP : u--
>>>>>> OTHER : ---
>>>>>
>>>>>
>>>>>> $ onedatastore list
>>>>>> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS
>>>>>> TM
>>>>>> 0 system 114.8G 85% - 0 sys -
>>>>>> shared
>>>>>> 1 default 114.9G 84% - 2 img fs
>>>>>> ssh
>>>>>> 2 files 114.9G 84% - 0 fil fs
>>>>>> ssh
>>>>>> 103 ceph 0M - - 0 img ceph
>>>>>> ceph
>>>>>
>>>>>
>>>>> but if we run rados as oneadmin user:
>>>>>
>>>>>> $ rados df -p one --id libvirt
>>>>>> pool name category KB objects clones
>>>>>> degraded unfound rd rd KB wr wr
>>>>>> KB
>>>>>> one - 0 0 0
>>>>>> 0 0 0 0 0 0
>>>>>> total used 1581852 37
>>>>>> total avail 140846865180
>>>>>> total space 140848447032
>>>>>
>>>>>
>>>>> It's working correctly (we are using one pool and libvirt ceph id)
>>>>>
>>>>> the oned.log only shows this info:
>>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore default (1)
>>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore files (2)
>>>>> Tue Jun 17 10:06:37 2014 [InM][D]: Monitoring datastore ceph (103)
>>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore default (1) successfully
>>>>> monitored.
>>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore files (2) successfully
>>>>> monitored.
>>>>> Tue Jun 17 10:06:37 2014 [ImM][D]: Datastore ceph (103) successfully
>>>>> monitored.
>>>>>
>>>>> Any clue about how to debug this issue?
>>>>>
>>>>> Thanks in advance!
>>>>> Alvaro
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list
>>>>> Users at lists.opennebula.org
>>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opennebula.org
>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
--
Javier Fontán Muiños
Developer
OpenNebula - Flexible Enterprise Cloud Made Simple
www.OpenNebula.org | @OpenNebula | github.com/jfontan
More information about the Users
mailing list