Hello,<div><br></div><div>Ok, after upgrading to 4.2 I am still having trouble with qemu-img convert segfaulting. I attempted to upgrade the qemu-utils package to upgrade qemu-img, however, I could only find qemu-utils 1.5.0 from debian. However, `qemu-img convert -O rbd ...` reports "file type rbd not recognized".</div>
<div><br></div><div>I made the following change which allows the creation of images.</div><div><br></div><div><div>/var/lib/one/remotes/datastore/ceph/cp</div><div>118c118</div><div>< $QEMU_IMG convert -O raw $TMP_DST rbd:$RBD_SOURCE</div>
<div>---</div><div>> $QEMU_IMG convert -O rbd $TMP_DST rbd:$RBD_SOURCE</div></div><div><br></div><div><div>/var/lib/one/remotes/datastore/ceph/mkfs</div><div>96c96</div><div>< $QEMU_IMG convert -O raw $TMP_DST rbd:$RBD_SOURCE</div>
<div>---</div><div>> $QEMU_IMG convert -O rbd $TMP_DST rbd:$RBD_SOURCE</div><div><br></div><div>Now the problem is when I go to deploy a VM, I get the following error:</div><div><br></div><div><div>>> root@red6:/var/lib/one/remotes/datastore/ceph# cat /var/log/one/2.log</div>
<div>>> Sun Jul 28 18:52:29 2013 [DiM][I]: New VM state is ACTIVE.</div><div>>> Sun Jul 28 18:52:29 2013 [LCM][I]: New VM state is PROLOG.</div><div>>> Sun Jul 28 18:52:29 2013 [TM][I]: Command execution fail: /var/lib/one/remotes/tm/ceph/clone red6:one/one-6 red6:/var/lib/one//datastores/0/2/disk.0 2 102</div>
<div>>> Sun Jul 28 18:52:29 2013 [TM][E]: clone: Command "rbd copy one/one-6 one/one-6-2-0" failed: You must specify a subsystem to invoke.</div><div>>> Sun Jul 28 18:52:29 2013 [TM][I]: usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]</div>
<div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [-D [bind_address:]port] [-e escape_char] [-F configfile]</div><div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [-I pkcs11] [-i identity_file]</div><div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [-L [bind_address:]port:host:hostport]</div>
<div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]</div><div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [-R [bind_address:]port:host:hostport] [-S ctl_path]</div>
<div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [-W host:port] [-w local_tun[:remote_tun]]</div><div>>> Sun Jul 28 18:52:29 2013 [TM][I]: [user@]hostname [command]</div><div>>> Sun Jul 28 18:52:29 2013 [TM][E]: Error cloning one/one-6 to one/one-6-2-0 in</div>
<div>>> Sun Jul 28 18:52:29 2013 [TM][I]: ExitCode: 255</div><div>>> Sun Jul 28 18:52:29 2013 [TM][E]: Error executing image transfer script: Error cloning one/one-6 to one/one-6-2-0 in</div><div>>> Sun Jul 28 18:52:29 2013 [DiM][I]: New VM state is FAILED</div>
</div><div><br></div><div>Running the rbd command completes successfully:</div><div><br></div><div>>> root@red6:/var/lib/one/remotes/datastore/ceph# rbd copy one/one-6 one/one-6-2-0</div><div>>> Image copy: 100% complete...done.</div>
<div><br></div><div>But running the OpenNebula commad results in ssh dumping its syntax:</div><div><br></div><div><div>oneadmin@red6:~$ /var/lib/one/remotes/tm/ceph/clone red6:one/one-6 red6:/var/lib/one//datastores/0/2/disk.0 2 102 2>&1 | perl -pe 's//>> /'</div>
<div>>> ERROR: clone: Command "rbd copy one/one-6 one/one-6-2-0" failed: You must specify a subsystem to invoke.</div><div>>> usage: ssh [-1246AaCfgKkMNnqsTtVvXxYy] [-b bind_address] [-c cipher_spec]</div>
<div>>> [-D [bind_address:]port] [-e escape_char] [-F configfile]</div><div>>> [-I pkcs11] [-i identity_file]</div><div>>> [-L [bind_address:]port:host:hostport]</div><div>
>> [-l login_name] [-m mac_spec] [-O ctl_cmd] [-o option] [-p port]</div><div>>> [-R [bind_address:]port:host:hostport] [-S ctl_path]</div><div>>> [-W host:port] [-w local_tun[:remote_tun]]</div>
<div>>> [user@]hostname [command]</div><div>>> ERROR MESSAGE --8<------</div><div>>> Error cloning one/one-6 to one/one-6-2-0 in </div><div>>> ERROR MESSAGE ------>8--</div></div><div>
<br></div><div>I tried to rewrite the ssh_exec_and_log command so I could see the full command that was being executed, but all I get is the original error message:</div><div><br></div><div><div>function ssh_exec_and_log</div>
<div>{</div><div># SSH_EXEC_ERR=`$SSH $1 sh -s 2>&1 1>/dev/null <<EOF</div><div> SSH_CMD="$SSH $1 sh -s $2"</div><div> SSH_EXEC_ERR=`$SSH_CMD`</div><div> SSH_EXEC_RC=$?</div><div><br>
</div><div> echo $SSH_CMD</div><div> if [ $SSH_EXEC_RC -ne 0 ]; then</div><div> log_error "Command \"$2\" failed: $SSH_EXEC_ERR \n ssh_command: $SSH_CMD"</div><div><br></div><div> if [ -n "$3" ]; then</div>
<div> error_message "$3 $SSH_CMD"</div><div> else</div><div> error_message "Error executing $2: $SSH_EXEC_ERR $SSH_CMD"</div><div> fi</div><div><br></div><div> exit $SSH_EXEC_RC</div>
<div> fi</div><div>}</div></div><div><br></div><br><div class="gmail_quote">On Mon, Jul 15, 2013 at 8:43 AM, Michael Curran <span dir="ltr"><<a href="mailto:michael.curran@connectsolutions.com" target="_blank">michael.curran@connectsolutions.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks Ruben!!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I am considering upgrading to 4.2 (on 4.0.1 right now) to see if it resolves some of my VM creation issues with ESXi as well (like not creating the pcibridge
for the network card to be discovered)<u></u><u></u></span></p><div class="im">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Michael Curran | connectsolutions | Lead Network Architect<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Phone <a href="tel:614.568.2285" value="+16145682285" target="_blank">614.568.2285</a> | Mobile <a href="tel:614.403.6320" value="+16144036320" target="_blank">614.403.6320</a> |
<a href="http://www.connectsolutions.com" target="_blank"><span style="color:#0563c1">www.connectsolutions.com</span></a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
</div><p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Ruben S. Montero [mailto:<a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a>]
<br>
<b>Sent:</b> Monday, July 15, 2013 10:26 AM<br>
<b>To:</b> Michael Curran<br>
<b>Cc:</b> Campbell, Bill; Jon; Users OpenNebula</span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [one-users] How to use Ceph/RBD for System Datastore<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Michael<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">This could be a bug in the VMware drivers, (not removing images) hopefully solved in 4.2. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Please note that OpenNebula should not allow you to delete a datastore with images in it (as show in onedatastore show under the IMAGES section)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">ruben<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Jul 15, 2013 at 3:38 PM, Michael Curran <<a href="mailto:michael.curran@connectsolutions.com" target="_blank">michael.curran@connectsolutions.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">For me, Im using 12.04 with Libvirt 1.06 , ESXi instead of Ceph/RBD and none of my images or vm’s
delete from the datastore when I delete them.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Michael Curran | connectsolutions | Lead Network Architect</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Phone <a href="tel:614.568.2285" value="+16145682285" target="_blank">614.568.2285</a> | Mobile <a href="tel:614.403.6320" value="+16144036320" target="_blank">614.403.6320</a> |
<a href="http://www.connectsolutions.com" target="_blank"><span style="color:#0563c1">www.connectsolutions.com</span></a></span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">
<a href="mailto:users-bounces@lists.opennebula.org" target="_blank">users-bounces@lists.opennebula.org</a> [mailto:<a href="mailto:users-bounces@lists.opennebula.org" target="_blank">users-bounces@lists.opennebula.org</a>]
<b>On Behalf Of </b>Campbell, Bill<br>
<b>Sent:</b> Monday, July 15, 2013 9:04 AM<br>
<b>To:</b> Jon</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
<b>Cc:</b> Users OpenNebula<br>
<b>Subject:</b> Re: [one-users] How to use Ceph/RBD for System Datastore<u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">That's a strange one (the delete issue). I've not run across that, I'm hoping it's not related to 13.04. I've
done all my testing/deployment on the latest LTS release, Ubuntu 12.04, and everything seems to be working okay.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">I did submit a feature request to the development team for further optimization of the Ceph driver, as newer versions
of libvirt handle cephx authentication differently (which a method of utilizing appropriately does not exist in OpenNebula at the moment). I'm wondering if this may be part of the issue.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">As an aside, we are still on 3.8.3 using our internally developed Ceph driver (of which we've submitted the current
version of to OpenNebula team for consideration of inclusion) that takes advantage of Ceph's RBD format 2 images (COW clones, etc.) which no longer uses qemu-img for creation of images but rather 'rbd import' as it can specify the format of the image. The
Development request with my submitted/updated driver is below, however be advised that this would probably be unsupported by the OpenNebula team so if you ran into any issues then you know the drill. ;-) It *should* work on ONE 4.0, however not 13.04 (due
to the libvirt issues mentioned).</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">I still think if possible for you, attempt to use Ubuntu 12.04 for your virtualization systems until the updated
Ceph driver is available (there's lots of additional information that must be created/included in the deployment file as referenced in the dev request). </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Below is the dev request I mentioned, but be advised if you use anything attached to this post then it would be
unsupported.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""><a href="http://dev.opennebula.org/issues/1796#change-4486" target="_blank">http://dev.opennebula.org/issues/1796#change-4486</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<hr size="2" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif"">From:
</span></b><span style="font-family:"Helvetica","sans-serif"">"Jon" <<a href="mailto:three18ti@gmail.com" target="_blank">three18ti@gmail.com</a>><br>
<b>Cc: </b>"Users OpenNebula" <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>><br>
<b>Sent: </b>Sunday, July 14, 2013 5:28:55 PM<br>
<b>Subject: </b>Re: [one-users] How to use Ceph/RBD for System Datastore</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Hi Bill,</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">I'm using Ubuntu 13.04 and libvirtd (libvirt) 1.0.2.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">As Jens recommended I attempted to run the qemu-img command with "raw" instead of "rbd"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">When I manually run the command:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> qemu-img convert -O raw /var/tmp/506f2a15417925478414f1c36f8228f7 rbd:one/one-26</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">I'm able to mark the image as active, but when I try to instantiate an image I get the error:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> missing DISK mandatory attributes (SOURCE, TM_MAD, CLONE, DATASTORE_ID) for VM 32, DISK 0</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Also, I've noticed, when I remove the image from OpenNebula, it does not remove the image from the ceph datastore.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Any ideas?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Thanks,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Jon A</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Helvetica","sans-serif"">On Thu, Jul 11, 2013 at 5:49 AM, Campbell, Bill <<a href="mailto:bcampbell@axcess-financial.com" target="_blank">bcampbell@axcess-financial.com</a>>
wrote:</span><u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Which distribution are you using for the storage node that OpenNebula accesses? (this is the node where qemu-img
would be run I do believe). This looks like a potential problem with that qemu-img binary.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">
<hr size="2" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif"">From:
</span></b><span style="font-family:"Helvetica","sans-serif"">"Jon" <<a href="mailto:three18ti@gmail.com" target="_blank">three18ti@gmail.com</a>><br>
<b>Cc: </b>"Users OpenNebula" <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>><br>
<b>Sent: </b>Wednesday, July 10, 2013 8:03:40 PM</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""><br>
<b>Subject: </b>Re: [one-users] How to use Ceph/RBD for System Datastore</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Hey Bill,</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">So let me ask you this. How do you create rbd images? I'm attempting to create a blank datastore, but qemu-img convert keeps
segfaulting.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Any suggestions are appreciated.</span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">root@red6:~# qemu-img convert -O rbd /var/tmp/61e14679af7dd1e0e1e09e230c89f82a rbd:one/one-17</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Segmentation fault (core dumped)</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">The oned log indicates the same:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:30 2013 [ImM][I]: Creating disk at of 5120Mb (type: ext4)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: Command execution fail: /var/lib/one/remotes/datastore/ceph/mkfs </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][E]: mkfs: Command " set -e</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: # create and format</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: dd if=/dev/zero of=/var/tmp/61e14679af7dd1e0e1e09e230c89f82a bs=1 count=1 seek=5120M</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: mkfs -t ext4 -F /var/tmp/61e14679af7dd1e0e1e09e230c89f82a</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: # create rbd</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: qemu-img convert -O rbd /var/tmp/61e14679af7dd1e0e1e09e230c89f82a rbd:one/one-17</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: # remove original</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: rm -f /var/tmp/61e14679af7dd1e0e1e09e230c89f82a" failed: 1+0 records in</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: 1+0 records out</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: 1 byte (1 B) copied, 0.000232576 s, 4.3 kB/s</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: mke2fs 1.42.5 (29-Jul-2012)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: Segmentation fault (core dumped)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][E]: Error registering one/one-17 in localhost</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][I]: ExitCode: 139</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Wed Jul 10 17:52:31 2013 [ImM][E]: Error creating datablock: Error registering one/one-17 in localhost</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">My image definition looks like this:</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">IMAGE 17 INFORMATION </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">ID : 17 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">NAME : ubuntu-server-13.04-x86_64</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">USER : oneadmin </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">GROUP : oneadmin </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">DATASTORE : rbd1 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">TYPE : DATABLOCK </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">REGISTER TIME : 07/10 17:52:30 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">PERSISTENT : Yes </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">SOURCE : </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">FSTYPE : ext4 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">SIZE : 5G </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">STATE : err </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">RUNNING_VMS : 0 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">PERMISSIONS </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">OWNER : um- </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">GROUP : --- </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">OTHER : --- </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">IMAGE TEMPLATE </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">DESCRIPTION="ubuntu-server-13.04-x86_64"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">DEV_PREFIX="hd"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">ERROR="Wed Jul 10 17:52:31 2013 : Error creating datablock: Error registering one/one-17 in localhost"</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Thanks,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Jon A</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Helvetica","sans-serif"">On Wed, Jul 10, 2013 at 5:14 PM, Jon <<a href="mailto:three18ti@gmail.com" target="_blank">three18ti@gmail.com</a>> wrote:</span><u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Hey Bill,</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Thanks for this. This works perfectly!</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Thanks,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Jon A</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><span style="font-family:"Helvetica","sans-serif"">On Wed, Jul 10, 2013 at 6:44 AM, Campbell, Bill <<a href="mailto:bcampbell@axcess-financial.com" target="_blank">bcampbell@axcess-financial.com</a>>
wrote:</span><u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Not entirely. You shouldn’t need to manually create/mount an RBD for the system datastore. Since
the system datastore holds the running VM deployment files (and not necessarily an RBD image, just a reference to it in the deployment file) then this directory does not necessarily need to be shared.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Here’s what we do:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">OpenNebula system configured with no special exports/shares.</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The System datastore is modified to use the SSH transfer manager</span><u></u><u></u></p>
<p><span style="font-size:11.0pt;font-family:Symbol;color:#1f497d">·</span><span style="font-size:7.0pt;color:#1f497d">
</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We modify the SSH transfer manager pre/post migrate scripts (by default located in /var/lib/one/remotes/tm/ssh/) to copy files from the source host to the destination host
prior to migration/delete files on source after successful migration.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Don’t worry about mapping/unmapping RBD volumes. When creating/importing images into the Ceph datastore
the RBDs should be created at this point. So long as the Hypervisor nodes can see/interact with the Ceph cluster, when you deploy the VM it will use the RBD in the cluster for storage (no files copied/mapped locally, all handled by QEMU).</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Here is the pre-migrate script we use (very simple):</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">#!/bin/bash</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">SRC=$1</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">DST=$2</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">REMDIR=$3</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">VMID=$4</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">DSID=$5</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">TEMPLATE=$6</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ssh $DST mkdir -p /var/lib/one/datastores/0/$VMID</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ssh $SRC scp /var/lib/one/datastores/0/$VMID/* $DST:/var/lib/one/datastores/0/$VMID/</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">exit 0</span></i><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">And the post-migrate script:</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">#!/bin/bash</span></i><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">SRC=$1</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">DST=$2</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">REMDIR=$3</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">VMID=$4</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">DSID=$5</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">TEMPLATE=$6</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">ssh $SRC rm -rf /var/lib/one/datastores/0/$VMID</span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></i><u></u><u></u></p>
<p class="MsoNormal"><i><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">exit 0</span></i><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hope this helps!</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Jon [mailto:<a href="mailto:three18ti@gmail.com" target="_blank">three18ti@gmail.com</a>]
<br>
<b>Sent:</b> Wednesday, July 10, 2013 12:01 AM<br>
<b>To:</b> Campbell, Bill<br>
<b>Cc:</b> Users OpenNebula<br>
<b>Subject:</b> Re: [one-users] How to use Ceph/RBD for System Datastore</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Hey Bill,<u></u><u></u></p>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for getting back to me. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">If I'm understanding you correctly, you're basically using the ssh transfer manager to perform live migrations?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Do you then create/mount one rbd per host?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">E.g., <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">host1:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">mount /dev/rbd/rbd/host1-one-system /var/lib/one/datastores/0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">host2:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">mount /dev/rbd/rbd/host2-one-system /var/lib/one/datastores/0<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">then use the modified ssh drivers to perform the migrations?<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">I would definitely be interested in learning how you accomplished that.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">My other thought was to use CephFS for shared storage. This would eliminate the need for a NFS/GlusterFS/CLVM, which is an extra layer of complexity I would like to avoid. As
I understand it though, CephFS isn't "ready for prime-time" which gives me pause... <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks again,<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">Jon A<u></u><u></u></p>
<div>
<p class="MsoNormal">On Tue, Jul 9, 2013 at 7:55 PM, Campbell, Bill <<a href="mailto:bcampbell@axcess-financial.com" target="_blank">bcampbell@axcess-financial.com</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt">
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Jon,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">I think I understand what you are trying to do, but I think it doesn't quite work that way. Let me try to explain
(and please let me know if I don't explain it well enough ;-))</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">I don't think that you can use Ceph directly as a system datastore. The way the Ceph datastore driver works for
migrations is leveraging whatever transfer method you have for the system datastore to perform the migration. For example, if you use the 'shared' system datastore, then it will use that transfer manager's pre and post migration drivers. For 'ssh', the ssh
drivers, and so on. The way the Ceph datastore is implemented is as Ceph Block Devices, so unfortunately there is not a way to use it as a simple shared volume.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">There are 2 potential solutions for getting live migrations working for your Ceph datastore VMs:</span><u></u><u></u></p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal">
<span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Create a shared NFS volume (or other 'sharable' filesystem, like GFS2, OCFS2, etc., however these are much more complicated to configure and usually not worth the hassle) and have the shared volume
mounted to the same location on each hypervisor node. In a previous test deployment, we just exported out the /var/lib/one/vms directory to the hypervisors. At this point, all of the hypervisors should be able to see the deployment files in the same location
and you should be able to perform a migration.</span><u></u><u></u></li><li class="MsoNormal">
<span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Use SSH as the transfer manager for your system datastore, and modify the pre and post-migrate scripts to copy the deployment files from the current VM host to the target VM host. This is the
method we use currently in our deployment, as it is one less configuration step that we have to worry about maintaining on each node, and makes expanding our cluster much quicker and easier. I can share with you the pre and post-migrate scripts we use if
you like.</span><u></u><u></u></li></ul>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Let me know if the above makes sense, and of course if you need any additional help please don't hesitate to bug
me. I'm very familiar with the Ceph drivers ;-)</span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
<div class="MsoNormal" align="center" style="text-align:center"><span style="font-family:"Helvetica","sans-serif"">
<hr size="2" width="100%" align="center">
</span></div>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif"">From:
</span></b><span style="font-family:"Helvetica","sans-serif"">"Jon" <<a href="mailto:three18ti@gmail.com" target="_blank">three18ti@gmail.com</a>><br>
<b>To: </b>"Users OpenNebula" <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>><br>
<b>Sent: </b>Tuesday, July 9, 2013 8:05:51 PM<br>
<b>Subject: </b>[one-users] How to use Ceph/RBD for System Datastore</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Hello All,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">I am using Ceph as my storage back end and would like to know how to configure the system datastore, such that I can live migrate
vms.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Following the directions, I thought I could create a datastore, format it, and mount it at /var/lib/one/datastores/0 , however,
I discovered, that isn't quite how things work.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> <a href="http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html" target="_blank">http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html</a></span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">You can read more about that at the above link, but long story short, to mount a shared filesystem it has to be a "clustered"
filesystem (I think CephFS is the "clustered filesystem", in this case).</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">I attempted to modify my system datastore config, however, I was unable to change the DS_MAD parameter, and vm creation errors
out telling me there's no /var/lib/one/remotes/tm/ceph/mkswap driver (there isn't)</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> oneadmin@red6:~$ onedatastore show 0 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> DATASTORE 0 INFORMATION </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> ID : 0 </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> NAME : system </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> USER : oneadmin </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> GROUP : oneadmin </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> CLUSTER : - </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> TYPE : SYSTEM </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> DS_MAD : - </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> TM_MAD : ceph </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> BASE PATH : /var/lib/one/datastores/0</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> DISK_TYPE : FILE </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> PERMISSIONS </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> OWNER : um- </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> GROUP : u-- </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> OTHER : --- </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> DATASTORE TEMPLATE </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> DISK_TYPE="rbd"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> DS_MAD="-"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> TM_MAD="ceph"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> TYPE="SYSTEM_DS"</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">>> IMAGES </span><u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Maybe I'm just confused. Can anyone provide some guidance on setting ceph up as the system datastore?</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Thanks,</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">Jon A</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif"">_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<div>
<p class="MsoNormal"><b>NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message in error, immediately notify the sender and destroy
all copies.</b><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif"">NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message
in error, immediately notify the sender and destroy all copies.</span></b><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a></span><u></u><u></u></p>
</div>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
<div>
<p class="MsoNormal"><b><span style="font-family:"Helvetica","sans-serif"">NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message
in error, immediately notify the sender and destroy all copies.</span></b><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""> </span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span style="font-family:"Helvetica","sans-serif""><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a></span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> </span><u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message in error, immediately notify the sender and destroy
all copies.</b><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><u></u><u></u></p>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Join us at OpenNebulaConf2013 in Berlin, 24-26 September, 2013<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">Ruben S. Montero, PhD<br>
Project co-Lead and Chief Architect<br>
OpenNebula - The Open Source Solution for Data Center Virtualization<br>
<a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">
rsmontero@opennebula.org</a> | @OpenNebula<u></u><u></u></p>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br></div>