<p dir="ltr">Note: you will have to mount a different rbd for each /var/lib/oneadmin/datastores/0, you can't mount the same rbd on multiple hosts (rbds are not "cluster aware" per se. That is to say, unless you include a clustered filesystem on top of your rbds--google "ceph iscsi"--but at that point, why not use cephfs other than it not being "redy for primetime" and adding additional infrastructure complexity (which using iscsi on top of ceph does anyway and provides a single point of failure, defeating the purpose of ceph)).</p>
<p dir="ltr">I rewrote my ssh transfer drivers to rsync the files before live migrating the vm. Though, there are obvious pitfalls with doing that as there is no grantee of consistency. </p>
<p dir="ltr">Ultimately, my solution was to use a separate rbd for swap, or not configure any at all (libvirt does support memory ballooning, though I'm not sure that O:N provides any controls), that way only non-vm files are ever stored in datastores/0 (I.e. only OpenNebula files are stored there).</p>
<div class="gmail_quote">On Dec 4, 2013 3:13 AM, "Kenneth" <<a href="mailto:kenneth@apolloglobal.net">kenneth@apolloglobal.net</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<u></u>
<div style="font-family:Verdana,Geneva,sans-serif">
<p>Going directly to RBD layer will be used once you use ceph as IMAGE datastore. Nebula will be interfacing directly with ceph and your images will be in RBD format.This is already the "direct way". This is the same things as using KVM with ceph RBD.</p>
<p>The only thing you may want to use is the cephfs (which is not RBD) is when you use it for the SYSTEM datastore and use shared for TM. This is what I use. Besides, system datastore does't contain a lot of files so if this method is inefficient, I won't notice it all.</p>
<p>But if you also want to use RBD for the system datastore, you can still use it. You just mount an RBD image in the /var/lib/datastore/0/ of you nodes.</p>
<div>---<br>
<pre><span style="font-size:small;font-family:verdana,geneva"><span style="font-size:x-small;font-family:helvetica">Thanks,
Kenneth</span><br><span style="font-size:x-small;font-family:helvetica">Apollo Global Corp.</span><br></span></pre>
</div>
<p>On 12/04/2013 04:08 PM, Mario Giammarco wrote:</p>
<blockquote type="cite" style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div dir="ltr">
<div>
<div>
<div>
<div>
<div>I have read all posts of this interesting thread.</div>
You suggest to use ceph as a shared filesystem and it is a good idea I agree.<br><br></div>
But I supposed that, because kvm supports ceph rbd and because opennebula supports ceph, there is a "direct way" to use it. </div>
I mean not using ceph DFS layer and going directly to rbd layer (also for system datastore)</div>
I do not understand what advantages has opennebula actual ceph support, can you explain to me?<br><br>Thanks,</div>
Mario</div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">2013/12/3 Jaime Melis <span><<a href="mailto:jmelis@c12g.com" target="_blank">jmelis@c12g.com</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Mario,
<div> </div>
<div>Cephfs CAN be used a shared filesystem datastore. I don't completely agree with Kenneth's recommendation of using 'ssh' as the TM for the system datastore. I think you can go for 'shared' as long as you have the /var/lib/one/datastores/... shared via Cephfs. OpenNebula doesn't care about what DFS solution you're using, it will simply assume files are already there.</div>
<div> </div>
<div>Another thing worth mentioning, from 4.4 onwards the HOST attribute of the datastore should be renamed as BRIDGE_LIST.</div>
<div> </div>
<div>cheers,<br>Jaime</div>
</div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">
<div>
<div>On Tue, Dec 3, 2013 at 11:28 AM, Kenneth <span><<a href="mailto:kenneth@apolloglobal.net" target="_blank">kenneth@apolloglobal.net</a>></span> wrote:</div>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div><span style="text-decoration:underline"></span>
<div style="font-family:Verdana,Geneva,sans-serif">
<p>Actually, I'm using ceph as the system datastore. I used the cephfs (CEPH FUSE) and mounted it on all nodes on /var/lib/one/datastores/0/</p>
<p>Regarding ssh for trasfer driver, I haven't really used it since I'm all on ceph, both system and image datastore. I may be wrong but that is how I understand it from the docs.</p>
<div>
<div>---<br>
<pre><span style="font-size:small;font-family:verdana,geneva"><span style="font-size:x-small;font-family:helvetica">Thanks,
Kenneth</span><br><span style="font-size:x-small;font-family:helvetica">Apollo Global Corp.</span><br></span></pre>
</div>
</div>
<div>
<div>
<p>On 12/03/2013 06:11 PM, Mario Giammarco wrote:</p>
<blockquote style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div dir="ltr">
<div>
<div>
<div>
<div>My problem was that because ceph is a distributed filesystem (and so it can be used as an alternative to nfs) I supposed I can use as a shared system datastore.</div>
Reading your reply I can see it is not true. Probably official documentation should clarify this.<br><br></div>
Infact I hoped to use ceph as system datastore because ceph is fault tolerant and nfs is not.<br><br></div>
Thanks for help,</div>
Mario</div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">2013/12/3 Kenneth <span><<a href="mailto:kenneth@apolloglobal.net" target="_blank">kenneth@apolloglobal.net</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span style="text-decoration:underline"></span>
<div style="font-family:Verdana,Geneva,sans-serif">
<p>Ceph won't be the default image datastore, but you can always choose it whenever you create an image.</p>
<p>You said you don't have an NFS disk and your just use plain disk on your system datastore so you <strong>should</strong> use ssh in order to have live migrations.</p>
<p>Mine uses shared as datastore since I mounted a shared folder on each nebula node.</p>
<div>
<div>---<br>
<pre><span style="font-size:small;font-family:verdana,geneva"><span style="font-size:x-small;font-family:helvetica">Thanks,
Kenneth</span><br><span style="font-size:x-small;font-family:helvetica">Apollo Global Corp.</span><br></span></pre>
</div>
</div>
<div>
<div>
<p>On 12/03/2013 03:01 PM, Mario Giammarco wrote:</p>
<blockquote style="padding-left:5px;border-left:#1010ff 2px solid;margin-left:5px;width:100%">
<div dir="ltr">First, thanks you for your very detailed reply!<br>
<div>
<div class="gmail_extra"><br><br>
<div class="gmail_quote">2013/12/3 Kenneth <span><<a href="mailto:kenneth@apolloglobal.net" target="_blank">kenneth@apolloglobal.net</a>></span><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span style="text-decoration:underline"></span>
<div style="font-family:Verdana,Geneva,sans-serif">
<p>You don't need to replace existing datastores, the important is you edit the system datastore as "ssh" because you still need to transfer files in each node when you deploy a VM.</p>
</div>
</blockquote>
<div> </div>
<div>So I lose live migration, right?</div>
<div>If I understand correctly ceph cannot be default datastore also.</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-family:Verdana,Geneva,sans-serif">
<p>Next, you should make sure that all your node are able to communicate with the ceph cluster. Issue the command "ceph -s" on all nodes including the front end to be sure that they are connected to ceph.</p>
<p> </p>
</div>
</blockquote>
<div> </div>
<div>... will check...</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-family:Verdana,Geneva,sans-serif"><br>
<p>oneadmin@cloud-node1:~$ onedatastore list</p>
<p><span style="font-family:courier new,courier"> ID NAME SIZE AVAIL CLUSTER IMAGES TYPE DS TM </span></p>
<p><span style="font-family:courier new,courier"> 0 system - - - 0 sys - shared</span></p>
<p><span style="font-family:courier new,courier"> 1 default 7.3G 71% - 1 img fs shared</span></p>
<p><span style="font-family:courier new,courier"> 2 files 7.3G 71% - 0 fil fs ssh</span></p>
<p><span style="font-family:courier new,courier"><strong> 100 cephds 5.5T 59% - 3 img ceph ceph</strong></span></p>
<p><span style="font-size:small;font-family:courier new,courier"></span>Once you add the you have verified that the ceph datastore is active you can upload images on the sunstone GUI. Be aware that conversion of images to RBD format of ceph may take quite some time.</p>
</div>
</blockquote>
<div> </div>
<div>I see in your configuration that system datastore is shared!<br><br></div>
<div>Thanks again,<br>Mario</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</div>
<div>_______________________________________________<br> Users mailing list<br><a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br><a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></div>
</blockquote>
</div>
<br><br clear="all">
<div> </div>
-- <br>
<div dir="ltr">Jaime Melis<br>C12G Labs - Flexible Enterprise Cloud Made Simple<br><a href="http://www.c12g.com" target="_blank">http://www.c12g.com</a> | <a href="mailto:jmelis@c12g.com" target="_blank">jmelis@c12g.com</a><br>
<div> </div>
<div>--</div>
<div> </div>
<div>Confidentiality Warning: The information contained in this e-mail and</div>
<div>any accompanying documents, unless otherwise expressly indicated, is</div>
<div>confidential and privileged, and is intended solely for the person</div>
<div>and/or entity to whom it is addressed (i.e. those identified in the</div>
<div>"To" and "cc" box). They are the property of C12G Labs S.L..</div>
<div>Unauthorized distribution, review, use, disclosure, or copying of this</div>
<div>communication, or any part thereof, is strictly prohibited and may be</div>
<div>unlawful. If you have received this e-mail in error, please notify us</div>
<div>immediately by e-mail at <a href="mailto:abuse@c12g.com" target="_blank">abuse@c12g.com</a> and delete the e-mail and</div>
<div>attachments and any copy from your system. C12G's thanks you for your</div>
<div>cooperation.</div>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div>