[one-users] Configuring Ceph datastore 4.6

Wed Sep 10 20:42:25 PDT 2014

RBD support is disabled in the Centos 7.0 qemu release packages. There's a
block in the .spec file that reads:

%if %{rhev}
        --enable-live-block-ops \
        --enable-ceph-support \
%else
        --disable-live-block-ops \
        --disable-ceph-support \
%endif

rhev is defined 0 at the top of the file, setting this to 1 and rebuilding
after sorting the build dependencies yields some new packages with RBD
support and a -rhev suffix that install and work on Centos 7.0 just fine.

2014-09-11 9:42 GMT+12:00 Steven Timm <timm at fnal.gov>:

> The first and most obvious problem below was that we were running an
> old version of qemu-img and qemu-kvm that ships with RHEL6/Centos6/SL6
> that doesn't support the "rbd" format.  We were able to find
> a modified version that the Ceph people had back-ported and now
> we can import an image into the datastore and have
> gotten as far to getting a deployment.0 written on the hypervisor.
> It can't contact Ceph as yet, is getting connection refused but
> we think that is an authentication issue.
>
> http://ceph.com/packages/qemu-kvm/redhat/x86_64/
> is where these packages for 6.2 live.
>
> Hopefully this all gets easier pretty soon now that RedHat has bought Ceph
> and all the right packages will be in RHEL7.  Or will they be only
> proprietarily available in Redhat Enterprise Virtualization?  Has anyone
> tried yet?
>
> Steve
>
>
>
>
> On Wed, 10 Sep 2014, Steven Timm wrote:
>
>
>>
>> I have configured a Ceph datastore on one 4.6 and have gotten as
>> far as to get opennebula to accept the datastore.  But when we
>> try to do the first oneimage create into the datastore we get the
>> following error in oned.log :
>>
>> Wed Sep 10 13:10:44 2014 [ImM][I]: Command execution fail:
>> /var/lib/one/remotes/datastore/ceph/cp PERTX0RSSVZFUl9BQ1RJT05fREFUQT
>> 48SU1BR0U+PElEPjE4PC9JRD48VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+
>> b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+
>> Y2VwaHRlc3Q8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPj
>> E8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC
>> 9PV05FUl9BPjxHUk9VUF9VPjA8L0dST1VQX1U+PEdST1VQX00+
>> MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09U
>> SEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC
>> 9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjM8L0
>> RJU0tfVFlQRT48UEVSU0lTVEVOVD4wPC9QRVJTSVNURU5UPjxSRUdUSU1FPj
>> E0MTAzNzI1NTU8L1JFR1RJTUU+PFNPVVJDRT48L1NPVVJDRT48UEFUSD
>> 4vY2xvdWQvaW1hZ2VzL3RpbW0vNDBnYi5xY293MjwvUEFUSD48RlNUWVBFPj
>> wvRlNUWVBFPjxTSVpFPjQwOTYwPC9TSVpFPjxTVEFURT40PC9TVEFURT48Ul
>> VOTklOR19WTVM+MDwvUlVOTklOR19WTVM+PENMT05JTkdfT1BTPjA8L0NMT05JTk
>> dfT1BTPjxDTE9OSU5HX0lEPi0xPC9DTE9OSU5HX0lEPjxEQVRBU1RPUkVfSU
>> Q+MTAzPC9EQVRBU1RPUkVfSUQ+PERBVEFTVE9SRT5jZXBoX2RhdGFzdG
>> 9yZTwvREFUQVNUT1JFPjxWTVM+PC9WTVM+PENMT05FUz48L0NMT05FUz48VEVNUE
>> xBVEU+PERFU0NSSVBU
>>
> S
>
>> U9OPjwhW0NEQVRBW3Rlc3QgY2VwaCBnb2xkZW4gaW1nXV0+
>> PC9ERVNDUklQVElPTj48REVWX1BSRUZJWD48IVtDREFUQVtoZF1dPjwvREVW
>> X1BSRUZJWD48RFJJVkVSPjwhW0NEQVRBW3Fjb3cyXV0+PC9EUklWRVI+
>> PC9URU1QTEFURT48L0lNQUdFPjxEQVRBU1RPUkU+PElEPjEwMzwvSUQ+PFVJRD4wPC9VSUQ+
>> PEdJRD4wPC9HSUQ+PFVOQU1FPm9uZWFkbWluPC9VTkFNRT48R05BTUU+
>> b25lYWRtaW48L0dOQU1FPjxOQU1FPmNlcGhfZGF0YXN0b3JlPC9OQU1FPjxQ
>> RVJNSVNTSU9OUz48T1dORVJfVT4xPC9PV05FUl9VPjxPV05FUl9NPjE8L09X
>> TkVSX00+PE9XTkVSX0E+MDwvT1dORVJfQT48R1JPVVBfVT4xPC
>> 9HUk9VUF9VPjxHUk9VUF9NPjA8L0dST1VQX00+PEdST1VQX0E+
>> MDwvR1JPVVBfQT48T1RIRVJfVT4wPC9PVEhFUl9VPjxPVEhFUl9NPjA8L09U
>> SEVSX00+PE9USEVSX0E+MDwvT1RIRVJfQT48L1BFUk1JU1NJT05TPjxEU19NQUQ+
>> Y2VwaDwvRFNfTUFEPjxUTV9NQUQ+Y2VwaDwvVE1fTUFEPjxCQVNFX1BBVEg+
>> L3Zhci9saWIvb25lLy9kYXRhc3RvcmVzLzEwMzwvQkFTRV9QQVRIPjxUWVBFPjA8L1RZUEU+
>> PERJU0tfVFlQRT4zPC9ESVNLX1RZUEU+PENMVVNURVJfSUQ+LTE8L0NMVVNURVJfSUQ+
>> PENMVVNURVI+PC9DTFVTVEVSPjxUT1RBTF9NQj42MTAyNzMyODwvVE9UQUxfTUI+
>> PEZSRUVfTUI+NjA4NTE1NTI8L0ZSRUVfTUI+PFVTRURfTUI+
>> MTc1Nzc2PC9VU0VEX01CPjxJTUFHRVM+PC9JTUFHR
>>
> V
>
>> M+PFRFTVBMQVRFPjxCQVNFX1BBVEg+PCFbQ0RBVEFbL3Zhci9saWIvb25lLy
>> 9kYXRhc3RvcmVzL11dPjwvQkFTRV9QQVRIPjxCUklER0VfTElTVD48IVtDRE
>> FUQVtvbmU0ZGV2XV0+PC9CUklER0VfTElTVD48Q0VQSF9IT1
>> NUPjwhW0NEQVRBW3N0a2VuZGNhMDFhIHN0a2VuZGNhMDRhIHN0a2VuZGNhMD
>> JhXV0+PC9DRVBIX0hPU1Q+PENFUEhfU0VDUkVUPjwhW0NEQVRBWy
>> 9ldGMvY2VwaC9jZXBoLmNsaWVudC5hZG1pbi5rZXlyaW5nXV0+
>> PC9DRVBIX1NFQ1JFVD48Q0xPTkVfVEFSR0VUPjwhW0NEQVRBW1NFTEZdXT48
>> L0NMT05FX1RBUkdFVD48REFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjwhW0NE
>> QVRBW3llc11dPjwvREFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjxESVNLX1RZ
>> UEU+PCFbQ0RBVEFbUkJEXV0+PC9ESVNLX1RZUEU+PERTX01BRD48IVtDREFUQVtjZXBoXV
>> 0+PC9EU19NQUQ+PExOX1RBUkdFVD48IVtDREFUQVtOT05FXV0+PC9MTl9UQVJHRVQ+
>> PFBPT0xfTkFNRT48IVtDREFUQVtvbmVdXT48L1BPT0xfTkFNRT48U1RBR0lOR19ESVI+
>> PCFbQ0RBVEFbL3Zhci9saWIvb25lL2NlcGgtdG1wXV0+
>> PC9TVEFHSU5HX0RJUj48VE1fTUFEPjwhW0NEQVRBW2NlcGhdXT48L1RNX01B
>> RD48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg== 18
>> Wed Sep 10 13:10:44 2014 [ImM][I]: cp: Copying local image
>> /cloud/images/timm/40gb.qcow2 to the image repository
>> Wed Sep 10 13:10:44 2014 [ImM][E]: cp: Command "    set -e
>> Wed Sep 10 13:10:44 2014 [ImM][I]:
>> Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "" = "2" ]; then
>> Wed Sep 10 13:10:44 2014 [ImM][I]: FORMAT=$(qemu-img info
>> /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c | grep "^file
>> format:" | awk '{print }')
>> Wed Sep 10 13:10:44 2014 [ImM][I]:
>> Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "$FORMAT" != "raw" ]; then
>> Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert -O raw
>> /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
>> /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
>> Wed Sep 10 13:10:44 2014 [ImM][I]: mv /var/lib/one/ceph-tmp/
>> 68bec7e25cb73c98a31a48117022d72c.raw /var/lib/one/ceph-tmp/
>> 68bec7e25cb73c98a31a48117022d72c
>> Wed Sep 10 13:10:44 2014 [ImM][I]: fi
>> Wed Sep 10 13:10:44 2014 [ImM][I]:
>> Wed Sep 10 13:10:44 2014 [ImM][I]: rbd import --format 2
>> /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c one/one-18
>> Wed Sep 10 13:10:44 2014 [ImM][I]: else
>> Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert /var/lib/one/ceph-tmp/
>> 68bec7e25cb73c98a31a48117022d72c rbd:one/one-18
>> Wed Sep 10 13:10:44 2014 [ImM][I]: fi
>> Wed Sep 10 13:10:44 2014 [ImM][I]:
>> Wed Sep 10 13:10:44 2014 [ImM][I]: # remove original
>> Wed Sep 10 13:10:44 2014 [ImM][I]: rm -f /var/lib/one/ceph-tmp/
>> 68bec7e25cb73c98a31a48117022d72c" failed: Unknown protocol
>> 'rbd:one/one-18'
>> Wed Sep 10 13:10:44 2014 [ImM][E]: Error registering one/one-18 in one4dev
>> Wed Sep 10 13:10:44 2014 [ImM][I]: ExitCode: 1
>> Wed Sep 10 13:10:44 2014 [ImM][E]: Error copying image in the datastore:
>> Error registering one/one-18 in one4dev
>>
>>
>>
>>
>> ---
>>
>> Clear we are afailed to register the rbd, but not clear why.. any
>> or clues on why we failed are helpful.
>> Several places in the docs refer to libvirt 1.x.  Has anyone
>> made this work on RHEL6/Centos 6 (we are running the newer kernel so
>> we do have the rbd.o kernel module available, and rbd import/export
>> works from the command line.)
>>
>> Steve Timm
>>
>>
>> ------------------------------------------------------------------
>> Steven C. Timm, Ph.D  (630) 840-8525
>> timm at fnal.gov  http://home.fnal.gov/~timm/
>> Fermilab Scientific Computing Division, Scientific Computing Services
>> Quad.
>> Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
>>
>>
> ------------------------------------------------------------------
> Steven C. Timm, Ph.D  (630) 840-8525
> timm at fnal.gov  http://home.fnal.gov/~timm/
> Fermilab Scientific Computing Division, Scientific Computing Services Quad.
> Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20140911/b0e2804c/attachment.htm>