[one-users] Configuring Ceph datastore 4.6

Steven Timm timm at fnal.gov
Wed Sep 17 12:38:34 PDT 2014 

Note that if I replace the client.libvirt ceph key with the
client.admin key I get a different error:

qemu-kvm: -drive file=rbd:one/one-19-59-0:id=admin:key=AQAGNMlTED9CHhAAxWyfQvJkM
R1PyTEZ+1lFUQ==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca0
4a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=non
e: could not open disk image 
rbd:one/one-19-59-0:id=admin:key=AQAGNMlTED9CHhAAxW
yfQvJkMR1PyTEZ+1lFUQ==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;st
kendca04a\:6789\;stkendca02a\:6789: Invalid argument


as opposed to:

qemu-kvm: -drive 
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: 
error reading header from one-19-58-0
qemu-kvm: -drive 
file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: 
could not open disk image 
rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789: 
Operation not permitted

Anybody who has knowledge of the internals of qemu-kvm and the arguments 
it wants, please let me know.. in particular a working deployment.0 and/or 
ps output from the qemu-kvm process would be very valuable indeed.

Steve Timm



On Wed, 17 Sep 2014, Steven Timm wrote:

>
> This is the kvm command our libvirt is trying to do, from
> /var/log/libvirt/qemu/one-58.log
>
> Clearly it is an authentication error, the only question is
> what and why.. we have made the client.libvirt secret
> as directed at
>
> http://docs.opennebula.org/4.8/administration/storage/ceph_ds.html
>
> The example there suggests to set CEPH_USER="libvirt"
> but I don't have a "libvirt" user defined on my system (most RHEL6/SL6 
> systems don't.).  Should we use a different user?
>
>
> 2014-09-17 18:43:51.579+0000: starting up
> LC_ALL=C PATH=/sbin:/usr/sbin:/bin:/usr/bin QEMU_AUDIO_DRV=none 
> /usr/libexec/qemu-kvm -name one-58 -S -M rhel6.3.0 -enable-kvm -m 4096 -smp 
> 2,sockets=2,cores=1,threads=1 -uuid 01ef1f2d-58e2-e089-0012-537426cd1bad 
> -nodefconfig -nodefaults -chardev 
> socket,id=charmonitor,path=/var/lib/libvirt/qemu/one-58.monitor,server,nowait 
> -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown 
> -device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive 
> file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none 
> -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 
> -drive 
> file=/var/lib/one//datastores/102/58/disk.1,if=none,id=drive-virtio-disk1,format=raw,cache=none 
> -device 
> virtio-blk-pci,scsi=off,bus=pci.0,addr=0x5,drive=drive-virtio-disk1,id=virtio-disk1 
> -drive 
> file=/var/lib/one//datastores/102/58/disk.2,if=none,media=cdrom,id=drive-ide0-1-0,readonly=on,format=raw 
> -device ide-drive,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -netdev 
> tap,fd=22,id=hostnet0,vhost=on,vhostfd=23 -device 
> virtio-net-pci,netdev=hostnet0,id=net0,mac=54:52:00:02:0b:04,bus=pci.0,addr=0x3 
> -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 
> -vnc 127.0.0.1:58 -k en-us -vga cirrus -device 
> virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x6
> char device redirected to /dev/pts/3
> qemu-kvm: -drive 
> file=rbd:one/one-19-58-0:id=libvirt:key=AQCv0RlUEOPfIhAAZLbGzwrfjyC39iapXVOFEg==:auth_supported=cephx\;none:mon_host=stkendca01a\:6789\;stkendca04a\:6789\;stkendca02a\:6789,if=none,id=drive-virtio-disk0,format=qcow2,cache=none: 
> error reading header from one-19-58-0
>
>
>
>
>
>
> On Wed, 17 Sep 2014, Steven Timm wrote:
>
>>
>>  Does anyone have a working Ceph datastore that would
>>  be willing to send me a copy of what the deployment.0 of
>>  your successfully-launched Ceph VM looks like?  Likewise
>>  a ps -ef of the corresponding running KVM process?
>>
>>  We are getting close to being able to launch a virtual machine from
>>  Ceph but it is still not quite clear what libvirt is doing
>>  under the covers to talk to rbd, and if it can be tested manually.
>>
>>  Steve Timm
>> 
>>
>>  On Wed, 10 Sep 2014, Steven Timm wrote:
>> 
>> >   The first and most obvious problem below was that we were running an
>> >   old version of qemu-img and qemu-kvm that ships with RHEL6/Centos6/SL6
>> >   that doesn't support the "rbd" format.  We were able to find
>> >   a modified version that the Ceph people had back-ported and now
>> >   we can import an image into the datastore and have
>> >   gotten as far to getting a deployment.0 written on the hypervisor.
>> >   It can't contact Ceph as yet, is getting connection refused but
>> >   we think that is an authentication issue.
>> > 
>> >   http://ceph.com/packages/qemu-kvm/redhat/x86_64/
>> >   is where these packages for 6.2 live.
>> > 
>> >   Hopefully this all gets easier pretty soon now that RedHat has bought 
>> >   Ceph
>> >   and all the right packages will be in RHEL7.  Or will they be only
>> >   proprietarily available in Redhat Enterprise Virtualization?  Has 
>> >   anyone
>> >   tried yet?
>> > 
>> >   Steve
>> > 
>> > 
>> > 
>> >   On Wed, 10 Sep 2014, Steven Timm wrote:
>> > 
>> > > 
>> > > 
>> > >    I have configured a Ceph datastore on one 4.6 and have gotten as
>> > >    far as to get opennebula to accept the datastore.  But when we
>> > >    try to do the first oneimage create into the datastore we get the
>> > >    following error in oned.log :
>> > > 
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: Command execution fail:
>> > >    /var/lib/one/remotes/datastore/ceph/cp
>> > >    PERTX0RSSVZFUl9BQ1RJT05fREFUQT48SU1BR0U+PElEPjE4PC9JRD48VUlEPjA8L1VJRD48R0lEPjA8L0dJRD48VU5BTUU+b25lYWRtaW48L1VOQU1FPjxHTkFNRT5vbmVhZG1pbjwvR05BTUU+PE5BTUU+Y2VwaHRlc3Q8L05BTUU+PFBFUk1JU1NJT05TPjxPV05FUl9VPjE8L09XTkVSX1U+PE9XTkVSX00+MTwvT1dORVJfTT48T1dORVJfQT4wPC9PV05FUl9BPjxHUk9VUF9VPjA8L0dST1VQX1U+PEdST1VQX00+MDwvR1JPVVBfTT48R1JPVVBfQT4wPC9HUk9VUF9BPjxPVEhFUl9VPjA8L09USEVSX1U+PE9USEVSX00+MDwvT1RIRVJfTT48T1RIRVJfQT4wPC9PVEhFUl9BPjwvUEVSTUlTU0lPTlM+PFRZUEU+MDwvVFlQRT48RElTS19UWVBFPjM8L0RJU0tfVFlQRT48UEVSU0lTVEVOVD4wPC9QRVJTSVNURU5UPjxSRUdUSU1FPjE0MTAzNzI1NTU8L1JFR1RJTUU+PFNPVVJDRT48L1NPVVJDRT48UEFUSD4vY2xvdWQvaW1hZ2VzL3RpbW0vNDBnYi5xY293MjwvUEFUSD48RlNUWVBFPjwvRlNUWVBFPjxTSVpFPjQwOTYwPC9TSVpFPjxTVEFURT40PC9TVEFURT48UlVOTklOR19WTVM+MDwvUlVOTklOR19WTVM+PENMT05JTkdfT1BTPjA8L0NMT05JTkdfT1BTPjxDTE9OSU5HX0lEPi0xPC9DTE9OSU5HX0lEPjxEQVRBU1RPUkVfSUQ+MTAzPC9EQVRBU1RPUkVfSUQ+PERBVEFTVE9SRT5jZXBoX2RhdGFzdG9yZTwvREFUQVNUT1JFPjxWTVM+PC9WTVM+PENMT05FUz48L0NMT05FUz48VEVNUExBVEU+PERF 
>> > >    U0N
> 
>> > >    SSV
>> > >    BU
>> >   S
>> > >    U9OPjwhW0NEQVRBW3Rlc3QgY2VwaCBnb2xkZW4gaW1nXV0+PC9ERVNDUklQVElPTj48REVWX1BSRUZJWD48IVtDREFUQVtoZF1dPjwvREVWX1BSRUZJWD48RFJJVkVSPjwhW0NEQVRBW3Fjb3cyXV0+PC9EUklWRVI+PC9URU1QTEFURT48L0lNQUdFPjxEQVRBU1RPUkU+PElEPjEwMzwvSUQ+PFVJRD4wPC9VSUQ+PEdJRD4wPC9HSUQ+PFVOQU1FPm9uZWFkbWluPC9VTkFNRT48R05BTUU+b25lYWRtaW48L0dOQU1FPjxOQU1FPmNlcGhfZGF0YXN0b3JlPC9OQU1FPjxQRVJNSVNTSU9OUz48T1dORVJfVT4xPC9PV05FUl9VPjxPV05FUl9NPjE8L09XTkVSX00+PE9XTkVSX0E+MDwvT1dORVJfQT48R1JPVVBfVT4xPC9HUk9VUF9VPjxHUk9VUF9NPjA8L0dST1VQX00+PEdST1VQX0E+MDwvR1JPVVBfQT48T1RIRVJfVT4wPC9PVEhFUl9VPjxPVEhFUl9NPjA8L09USEVSX00+PE9USEVSX0E+MDwvT1RIRVJfQT48L1BFUk1JU1NJT05TPjxEU19NQUQ+Y2VwaDwvRFNfTUFEPjxUTV9NQUQ+Y2VwaDwvVE1fTUFEPjxCQVNFX1BBVEg+L3Zhci9saWIvb25lLy9kYXRhc3RvcmVzLzEwMzwvQkFTRV9QQVRIPjxUWVBFPjA8L1RZUEU+PERJU0tfVFlQRT4zPC9ESVNLX1RZUEU+PENMVVNURVJfSUQ+LTE8L0NMVVNURVJfSUQ+PENMVVNURVI+PC9DTFVTVEVSPjxUT1RBTF9NQj42MTAyNzMyODwvVE9UQUxfTUI+PEZSRUVfTUI+NjA4NTE1NTI8L0ZSRUVfTUI+PFVTRURfTUI+MTc1Nzc2PC9VU0VEX01CPjxJTUFHRVM+P 
>> > >    C9J
> 
>> > >    TUF
>> > >    HR
>> >   V
>> > >    M+PFRFTVBMQVRFPjxCQVNFX1BBVEg+PCFbQ0RBVEFbL3Zhci9saWIvb25lLy9kYXRhc3RvcmVzL11dPjwvQkFTRV9QQVRIPjxCUklER0VfTElTVD48IVtDREFUQVtvbmU0ZGV2XV0+PC9CUklER0VfTElTVD48Q0VQSF9IT1NUPjwhW0NEQVRBW3N0a2VuZGNhMDFhIHN0a2VuZGNhMDRhIHN0a2VuZGNhMDJhXV0+PC9DRVBIX0hPU1Q+PENFUEhfU0VDUkVUPjwhW0NEQVRBWy9ldGMvY2VwaC9jZXBoLmNsaWVudC5hZG1pbi5rZXlyaW5nXV0+PC9DRVBIX1NFQ1JFVD48Q0xPTkVfVEFSR0VUPjwhW0NEQVRBW1NFTEZdXT48L0NMT05FX1RBUkdFVD48REFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjwhW0NEQVRBW3llc11dPjwvREFUQVNUT1JFX0NBUEFDSVRZX0NIRUNLPjxESVNLX1RZUEU+PCFbQ0RBVEFbUkJEXV0+PC9ESVNLX1RZUEU+PERTX01BRD48IVtDREFUQVtjZXBoXV0+PC9EU19NQUQ+PExOX1RBUkdFVD48IVtDREFUQVtOT05FXV0+PC9MTl9UQVJHRVQ+PFBPT0xfTkFNRT48IVtDREFUQVtvbmVdXT48L1BPT0xfTkFNRT48U1RBR0lOR19ESVI+PCFbQ0RBVEFbL3Zhci9saWIvb25lL2NlcGgtdG1wXV0+PC9TVEFHSU5HX0RJUj48VE1fTUFEPjwhW0NEQVRBW2NlcGhdXT48L1RNX01BRD48L1RFTVBMQVRFPjwvREFUQVNUT1JFPjwvRFNfRFJJVkVSX0FDVElPTl9EQVRBPg==
>> > >    18
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: cp: Copying local image
>> > >    /cloud/images/timm/40gb.qcow2 to the image repository
>> > >    Wed Sep 10 13:10:44 2014 [ImM][E]: cp: Command "    set -e
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]:
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "" = "2" ]; then
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: FORMAT=$(qemu-img info
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c | grep "^file
>> > >    format:" 
>> > > |    awk '{print }')
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]:
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: if [ "$FORMAT" != "raw" ]; then
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert -O raw
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: mv
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c.raw
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: fi
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]:
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: rbd import --format 2
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c one/one-18
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: else
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: qemu-img convert
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c 
>> > >    rbd:one/one-18
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: fi
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]:
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: # remove original
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: rm -f
>> > >    /var/lib/one/ceph-tmp/68bec7e25cb73c98a31a48117022d72c" failed: 
>> > >    Unknown
>> > >    protocol 'rbd:one/one-18'
>> > >    Wed Sep 10 13:10:44 2014 [ImM][E]: Error registering one/one-18 in 
>> > >    one4dev
>> > >    Wed Sep 10 13:10:44 2014 [ImM][I]: ExitCode: 1
>> > >    Wed Sep 10 13:10:44 2014 [ImM][E]: Error copying image in the 
>> > >    datastore:
>> > >    Error registering one/one-18 in one4dev
>> > > 
>> > > 
>> > > 
>> > > 
>> > >    ---
>> > > 
>> > >    Clear we are afailed to register the rbd, but not clear why.. any
>> > >    or clues on why we failed are helpful.
>> > >    Several places in the docs refer to libvirt 1.x.  Has anyone
>> > >    made this work on RHEL6/Centos 6 (we are running the newer kernel so
>> > >    we do have the rbd.o kernel module available, and rbd import/export
>> > >    works from the command line.)
>> > > 
>> > >    Steve Timm
>> > > 
>> > > 
>> > >    ------------------------------------------------------------------
>> > >    Steven C. Timm, Ph.D  (630) 840-8525
>> > >    timm at fnal.gov  http://home.fnal.gov/~timm/
>> > >    Fermilab Scientific Computing Division, Scientific Computing 
>> > >    Services
>> > >    Quad.
>> > >    Grid and Cloud Services Dept., Associate Dept. Head for Cloud 
>> > >    Computing
>> > > 
>> > 
>> >   ------------------------------------------------------------------
>> >   Steven C. Timm, Ph.D  (630) 840-8525
>> >   timm at fnal.gov  http://home.fnal.gov/~timm/
>> >   Fermilab Scientific Computing Division, Scientific Computing Services
>> >   Quad.
>> >   Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
>> >   _______________________________________________
>> >   Users mailing list
>> >   Users at lists.opennebula.org
>> >   http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> > 
>>
>>  ------------------------------------------------------------------
>>  Steven C. Timm, Ph.D  (630) 840-8525
>>  timm at fnal.gov  http://home.fnal.gov/~timm/
>>  Fermilab Scientific Computing Division, Scientific Computing Services
>>  Quad.
>>  Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
>>  _______________________________________________
>>  Users mailing list
>>  Users at lists.opennebula.org
>>  http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>> 
>
> ------------------------------------------------------------------
> Steven C. Timm, Ph.D  (630) 840-8525
> timm at fnal.gov  http://home.fnal.gov/~timm/
> Fermilab Scientific Computing Division, Scientific Computing Services Quad.
> Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>

------------------------------------------------------------------
Steven C. Timm, Ph.D  (630) 840-8525
timm at fnal.gov  http://home.fnal.gov/~timm/
Fermilab Scientific Computing Division, Scientific Computing Services Quad.
Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing

More information about the Users mailing list