Hey Bill,<div><br></div><div>Thanks for getting back to me. </div><div><br></div><div>If I'm understanding you correctly, you're basically using the ssh transfer manager to perform live migrations?</div><div>Do you then create/mount one rbd per host?</div>
<div><br></div><div>E.g., </div><div><br></div><div>host1:</div><div>mount /dev/rbd/rbd/host1-one-system /var/lib/one/datastores/0</div><div><br></div><div>host2:</div><div>mount /dev/rbd/rbd/host2-one-system /var/lib/one/datastores/0</div>
<div><br></div><div>then use the modified ssh drivers to perform the migrations?</div><div><br></div><div>I would definitely be interested in learning how you accomplished that.</div><div><br></div><div>My other thought was to use CephFS for shared storage. This would eliminate the need for a NFS/GlusterFS/CLVM, which is an extra layer of complexity I would like to avoid. As I understand it though, CephFS isn't "ready for prime-time" which gives me pause... </div>
<div><br></div><div>Thanks again,</div><div>Jon A<br><br><div class="gmail_quote">On Tue, Jul 9, 2013 at 7:55 PM, Campbell, Bill <span dir="ltr"><<a href="mailto:bcampbell@axcess-financial.com" target="_blank">bcampbell@axcess-financial.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:10pt;font-family:tahoma,new york,times,serif"><div>Jon,</div><div>I think I understand what you are trying to do, but I think it doesn't quite work that way. Let me try to explain (and please let me know if I don't explain it well enough ;-))</div>
<div><br></div><div>I don't think that you can use Ceph directly as a system datastore. The way the Ceph datastore driver works for migrations is leveraging whatever transfer method you have for the system datastore to perform the migration. For example, if you use the 'shared' system datastore, then it will use that transfer manager's pre and post migration drivers. For 'ssh', the ssh drivers, and so on. The way the Ceph datastore is implemented is as Ceph Block Devices, so unfortunately there is not a way to use it as a simple shared volume.</div>
<div><br></div><div>There are 2 potential solutions for getting live migrations working for your Ceph datastore VMs:</div><div><ul><li>Create a shared NFS volume (or other 'sharable' filesystem, like GFS2, OCFS2, etc., however these are much more complicated to configure and usually not worth the hassle) and have the shared volume mounted to the same location on each hypervisor node. In a previous test deployment, we just exported out the /var/lib/one/vms directory to the hypervisors. At this point, all of the hypervisors should be able to see the deployment files in the same location and you should be able to perform a migration.</li>
<li>Use SSH as the transfer manager for your system datastore, and modify the pre and post-migrate scripts to copy the deployment files from the current VM host to the target VM host. This is the method we use currently in our deployment, as it is one less configuration step that we have to worry about maintaining on each node, and makes expanding our cluster much quicker and easier. I can share with you the pre and post-migrate scripts we use if you like.</li>
</ul><div>Let me know if the above makes sense, and of course if you need any additional help please don't hesitate to bug me. I'm very familiar with the Ceph drivers ;-)</div></div><div><br></div><hr><div style="font-size:12pt;font-style:normal;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal">
<b>From: </b>"Jon" <<a href="mailto:three18ti@gmail.com" target="_blank">three18ti@gmail.com</a>><br><b>To: </b>"Users OpenNebula" <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>><br>
<b>Sent: </b>Tuesday, July 9, 2013 8:05:51 PM<br><b>Subject: </b>[one-users] How to use Ceph/RBD for System Datastore<div><div class="h5"><br><div><br></div><div>Hello All,</div><div><br></div><div>I am using Ceph as my storage back end and would like to know how to configure the system datastore, such that I can live migrate vms.</div>
<div><br></div><div>Following the directions, I thought I could create a datastore, format it, and mount it at /var/lib/one/datastores/0 , however, I discovered, that isn't quite how things work.</div><div><br></div>>> <a href="http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html" target="_blank">http://lists.ceph.com/pipermail/ceph-users-ceph.com/2013-May/001913.html</a><div>
You can read more about that at the above link, but long story short, to mount a shared filesystem it has to be a "clustered" filesystem (I think CephFS is the "clustered filesystem", in this case).</div>
<div><br></div><div>I attempted to modify my system datastore config, however, I was unable to change the DS_MAD parameter, and vm creation errors out telling me there's no /var/lib/one/remotes/tm/ceph/mkswap driver (there isn't)</div>
<div><br></div><div><div>>> oneadmin@red6:~$ onedatastore show 0 </div><div>>> DATASTORE 0 INFORMATION </div><div>>> ID : 0 </div>
<div>>> NAME : system </div><div>>> USER : oneadmin </div><div>>> GROUP : oneadmin </div><div>>> CLUSTER : - </div>
<div>>> TYPE : SYSTEM </div><div>>> DS_MAD : - </div><div>>> TM_MAD : ceph </div><div>>> BASE PATH : /var/lib/one/datastores/0</div>
<div>>> DISK_TYPE : FILE </div><div>>> </div><div>>> PERMISSIONS </div><div>>> OWNER : um- </div>
<div>>> GROUP : u-- </div><div>>> OTHER : --- </div><div>>> </div><div>>> DATASTORE TEMPLATE </div>
<div>>> DISK_TYPE="rbd"</div><div>>> DS_MAD="-"</div><div>>> TM_MAD="ceph"</div><div>>> TYPE="SYSTEM_DS"</div><div>>> </div><div>>> IMAGES </div>
</div><div><br></div><div>Maybe I'm just confused. Can anyone provide some guidance on setting ceph up as the system datastore?</div><div><br></div><div>Thanks,</div><div>Jon A</div><br></div></div>_______________________________________________<br>
Users mailing list<br><a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br><a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</div><div><br></div></div>
<br><div><b>NOTICE: Protect the information in this message in accordance with the company's security policies. If you received this message in error, immediately notify the sender and destroy all copies.</b></div>
<br></div></blockquote></div><br></div>