[one-users] Configuring Ceph datastore 4.6

Wed Sep 17 11:53:54 PDT 2014

This is the kvm command our libvirt is trying to do, from
/var/log/libvirt/qemu/one-58.log

Clearly it is an authentication error, the only question is
what and why.. we have made the client.libvirt secret
as directed at

http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html

The example there suggests to set CEPH_USER="libvirt"
but I don't have a "libvirt" user defined on my system (most RHEL6/SL6 
systems don't.).  Should we use a different user?

2014-09-17 18:43:51.579+0000: starting up
LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none 
/usr/libexec/qemu-kvm -name one-58 -S -M rhel6.3.0 -enable-kvm -m 4096 
-smp 2,sockets=2,cores=1,threads=1 -uuid 
01ef1f2d-58e2-e089-0012-537426cd1bad -nodefconfig -nodefaults -chardev 
socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-58.monitor,server,nowait 
-mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc 
-no-shutdown -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
-drive 
file=/var/lib/one//datastores/102/58/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none 
-device 
virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 
-drive 
file=/var/lib/one//datastores/102/58/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw 
-device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 
-netdev tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device 
virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3 
-chardev pty,id=charserial0 -device 
isa-serial,chardev=charserial0,id=serial0 -vnc 127.0.0.1:58 -k en-us -vga 
cirrus -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
char device redirected to /dev/pts/3
qemu-kvm: -drive 
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: 
error reading header from one-19-58-0

On Wed, 17 Sep 2014, Steven Timm wrote:

>
> Does anyone have a working Ceph datastore that would
> be willing to send me a copy of what the deployment.0 of
> your successfully-launched Ceph VM looks like?  Likewise
> a ps -ef of the corresponding running KVM process?
>
> We are getting close to being able to launch a virtual machine from
> Ceph but it is still not quite clear what libvirt is doing
> under the covers to talk to rbd, and if it can be tested manually.
>
> Steve Timm
>
>
> On Wed, 10 Sep 2014, Steven Timm wrote:
>
>>  The first and most obvious problem below was that we were running an
>>  old version of qemu-img and qemu-kvm that ships with RHEL6/Centos6/SL6
>>  that doesn't support the "rbd" format.  We were able to find
>>  a modified version that the Ceph people had back-ported and now
>>  we can import an image into the datastore and have
>>  gotten as far to getting a deployment.0 written on the hypervisor.
>>  It can't contact Ceph as yet, is getting connection refused but
>>  we think that is an authentication issue.
>>
>>  http://ceph.com/packages/qemu-kvm/redhat/x86_64/
>>  is where these packages for 6.2 live.
>>
>>  Hopefully this all gets easier pretty soon now that RedHat has bought Ceph
>>  and all the right packages will be in RHEL7.  Or will they be only
>>  proprietarily available in Redhat Enterprise Virtualization?  Has anyone
>>  tried yet?
>>
>>  Steve
>> 
>> 
>>
>>  On Wed, 10 Sep 2014, Steven Timm wrote:
>> 
>> > 
>> > 
>> >   I have configured a Ceph datastore on one 4.6 and have gotten as
>> >   far as to get opennebula to accept the datastore.  But when we
>> >   try to do the first oneimage create into the datastore we get the
>> >   following error in oned.log :
>> > 
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: Command execution fail:
>> >   /var/lib/one/remotes/datastore/ceph/cp
>> >   PERTX0RSSVZFUl9BQ1RJT05fREFUQT48SU1BR0U+PElEPjE4PC9JRD48VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+Y2VwaHRlc3Q8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjxHUk9VUF9VPjA8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjM8L0RJU0tfVFlQRT48UEVSU0lTVEVOVD4wPC9QRVJTSVNURU5UPjxSRUdUSU1FPjE0MTAzNzI1NTU8L1JFR1RJTUU+PFNPVVJDRT48L1NPVVJDRT48UEFUSD4vY2xvdWQvaW1hZ2VzL3RpbW0vNDBnYi5xY293MjwvUEFUSD48RlNUWVBFPjwvRlNUWVBFPjxTSVpFPjQwOTYwPC9TSVpFPjxTVEFURT40PC9TVEFURT48UlVOTklOR19WTVM+MDwvUlVOTklOR19WTVM+PENMT05JTkdfT1BTPjA8L0NMT05JTkdfT1BTPjxDTE9OSU5HX0lEPi0xPC9DTE9OSU5HX0lEPjxEQVRBU1RPUkVfSUQ+MTAzPC9EQVRBU1RPUkVfSUQ+PERBVEFTVE9SRT5jZXBoX2RhdGFzdG9yZTwvREFUQVNUT1JFPjxWTVM+PC9WTVM+PENMT05FUz48L0NMT05FUz48VEVNUExBVEU+PERFU0N 
>> >   SSV
>> >   BU
>>  S
>> >   U9OPjwhW0NEQVRBW3Rlc3QgY2VwaCBnb2xkZW4gaW1nXV0+PC9ERVNDUklQVElPTj48REVWX1BSRUZJWD48IVtDREFUQVtoZF1dPjwvREVWX1BSRUZJWD48RFJJVkVSPjwhW0NEQVRBW3Fjb3cyXV0+PC9EUklWRVI+PC9URU1QTEFURT48L0lNQUdFPjxEQVRBU1RPUkU+PElEPjEwMzwvSUQ+PFVJRD4wPC9VSUQ+PEdJRD4wPC9HSUQ+PFVOQU1FPm9uZWFkbWluPC9VTkFNRT48R05BTUU+b25lYWRtaW48L0dOQU1FPjxOQU1FPmNlcGhfZGF0YXN0b3JlPC9OQU1FPjxQRVJNSVNTSU9OUz48T1dORVJfVT4xPC9PV05FUl9VPjxPV05FUl9NPjE8L09XTkVSX00+PE9XTkVSX0E+MDwvT1dORVJfQT48R1JPVVBfVT4xPC9HUk9VUF9VPjxHUk9VUF9NPjA8L0dST1VQX00+PEdST1VQX0E+MDwvR1JPVVBfQT48T1RIRVJfVT4wPC9PVEhFUl9VPjxPVEhFUl9NPjA8L09USEVSX00+PE9USEVSX0E+MDwvT1RIRVJfQT48L1BFUk1JU1NJT05TPjxEU19NQUQ+Y2VwaDwvRFNfTUFEPjxUTV9NQUQ+Y2VwaDwvVE1fTUFEPjxCQVNFX1BBVEg+L3Zhci9saWIvb25lLy9kYXRhc3RvcmVzLzEwMzwvQkFTRV9QQVRIPjxUWVBFPjA8L1RZUEU+PERJU0tfVFlQRT4zPC9ESVNLX1RZUEU+PENMVVNURVJfSUQ+LTE8L0NMVVNURVJfSUQ+PENMVVNURVI+PC9DTFVTVEVSPjxUT1RBTF9NQj42MTAyNzMyODwvVE9UQUxfTUI+PEZSRUVfTUI+NjA4NTE1NTI8L0ZSRUVfTUI+PFVTRURfTUI+MTc1Nzc2PC9VU0VEX01CPjxJTUFHRVM+PC9J 
>> >   TUF
>> >   HR
>>  V
>> >   M+PFRFTVBMQVRFPjxCQVNFX1BBVEg+PCFbQ0RBVEFbL3Zhci9saWIvb25lLy9kYXRhc3RvcmVzL11dPjwvQkFTRV9QQVRIPjxCUklER0VfTElTVD48IVtDREFUQVtvbmU0ZGV2XV0+PC9CUklER0VfTElTVD48Q0VQSF9IT1NUPjwhW0NEQVRBW3N0a2VuZGNhMDFhIHN0a2VuZGNhMDRhIHN0a2VuZGNhMDJhXV0+PC9DRVBIX0hPU1Q+PENFUEhfU0VDUkVUPjwhW0NEQVRBWy9ldGMvY2VwaC9jZXBoLmNsaWVudC5hZG1pbi5rZXlyaW5nXV0+PC9DRVBIX1NFQ1JFVD48Q0xPTkVfVEFSR0VUPjwhW0NEQVRBW1NFTEZdXT48L0NMT05FX1RBUkdFVD48REFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjwhW0NEQVRBW3llc11dPjwvREFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjxESVNLX1RZUEU+PCFbQ0RBVEFbUkJEXV0+PC9ESVNLX1RZUEU+PERTX01BRD48IVtDREFUQVtjZXBoXV0+PC9EU19NQUQ+PExOX1RBUkdFVD48IVtDREFUQVtOT05FXV0+PC9MTl9UQVJHRVQ+PFBPT0xfTkFNRT48IVtDREFUQVtvbmVdXT48L1BPT0xfTkFNRT48U1RBR0lOR19ESVI+PCFbQ0RBVEFbL3Zhci9saWIvb25lL2NlcGgtdG1wXV0+PC9TVEFHSU5HX0RJUj48VE1fTUFEPjwhW0NEQVRBW2NlcGhdXT48L1RNX01BRD48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg==
>> >   18
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: cp: Copying local image
>> >   /cloud/images/timm/40gb.qcow2 to the image repository
>> >   Wed Sep 10 13:10:44 2014 [ImM][E]: cp: Command "    set -e
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]:
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "" = "2" ]; then
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: FORMAT=$(qemu-img info
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c | grep "^file
>> >   format:" 
>> > |   awk '{print }')
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]:
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "$FORMAT" != "raw" ]; then
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert -O raw
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: mv
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: fi
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]:
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: rbd import --format 2
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c one/one-18
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: else
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c rbd:one/one-18
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: fi
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]:
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: # remove original
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: rm -f
>> >   /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c" failed: Unknown
>> >   protocol 'rbd:one/one-18'
>> >   Wed Sep 10 13:10:44 2014 [ImM][E]: Error registering one/one-18 in 
>> >   one4dev
>> >   Wed Sep 10 13:10:44 2014 [ImM][I]: ExitCode: 1
>> >   Wed Sep 10 13:10:44 2014 [ImM][E]: Error copying image in the 
>> >   datastore:
>> >   Error registering one/one-18 in one4dev
>> > 
>> > 
>> > 
>> > 
>> >   ---
>> > 
>> >   Clear we are afailed to register the rbd, but not clear why.. any
>> >   or clues on why we failed are helpful.
>> >   Several places in the docs refer to libvirt 1.x.  Has anyone
>> >   made this work on RHEL6/Centos 6 (we are running the newer kernel so
>> >   we do have the rbd.o kernel module available, and rbd import/export
>> >   works from the command line.)
>> > 
>> >   Steve Timm
>> > 
>> > 
>> >   ------------------------------------------------------------------
>> >   Steven C. Timm, Ph.D  (630) 840-8525
>> >   timm at fnal.gov  http://home.fnal.gov/~timm/
>> >   Fermilab Scientific Computing Division, Scientific Computing Services
>> >   Quad.
>> >   Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
>> > 
>>
>>  ------------------------------------------------------------------
>>  Steven C. Timm, Ph.D  (630) 840-8525
>>  timm at fnal.gov  http://home.fnal.gov/~timm/
>>  Fermilab Scientific Computing Division, Scientific Computing Services
>>  Quad.
>>  Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
>>  _______________________________________________
>>  Users mailing list
>>  Users at lists.opennebula.org
>>  http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> 
>
> ------------------------------------------------------------------
> Steven C. Timm, Ph.D  (630) 840-8525
> timm at fnal.gov  http://home.fnal.gov/~timm/
> Fermilab Scientific Computing Division, Scientific Computing Services Quad.
> Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>

------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm at fnal.gov  http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing