Hi Gary,<div><br></div><div>sounds great, could you contribute the code at some point?</div><div><br></div><div>cheers,<br>Jaime<br><br><div class="gmail_quote">On Mon, Aug 6, 2012 at 2:52 PM, Gary S. Cuozzo <span dir="ltr"><<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:arial,helvetica,sans-serif">Here's an update on this thread...<br><br>
So I did originally setup a "local" datastore by using NFS from the host to the controller as a shared DS. It worked fine and performed as expected. I was able to get the speed of local storage without having bloated capacity requirements and delays in deployment due to having large images be copied across the network. I also tested rebooting the controller node and the host node and the NFS share would come back up reliably.<br>
<br>Having to setup the NFS share was the main drawback. I figured I could get around that by writing a custom DS/TS driver. That is the route I took. Over the weekend I created a 'localfs' set of drivers and they seem to work great and don't require me to setup the NFS share. This seems to give me exactly what I was looking for.<br>
<br>The only requirement for my driver is that I create the datastore first so that I can get the ID. Then, I mount my local disks (I use an LVM volume) at the correct mount point according to the DS ID.<br><br>I use a DS template variable to specify the hostname of the host so that I can use the driver on multiple hosts. The images get created as hostname:/path/to/image.<br>
<br>So far so good.<br><br>Just FYI,<br>gary<br><br><br><hr><div style="font-size:12pt;font-style:normal;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal"><b>From: </b>"Ruben S. Montero" <<a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a>><div>
<br><b>To: </b>"Gary S. Cuozzo" <<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>><br></div><b>Sent: </b>Thursday, August 2, 2012 12:46:28 PM<div><br><b>Subject: </b>Re: [one-users] Storage architecture question<br>
<br></div>OK, keep us updated with the performance of your final solution<div><br></div><div><br></div><div>Good Luck</div><div><br></div><div>Ruben<br><br><div class="gmail_quote">On Thu, Aug 2, 2012 at 5:58 PM, Gary S. Cuozzo <span dir="ltr"><<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:arial,helvetica,sans-serif">Hi Ruben,<br>Thank you for the reply & information.<br>
<br>I had previously thought about a similar setup to what you have outlined. One drawback I was trying to work around is that I will be duplicating storage requirement as I will have to store images on both my SAN/NAS and the local system. As my SAN/NAS is fully mirrored, I will actually incur a fairly substantial increase in the overhead per GB. I was trying to avoid that.<br>
<br>My other main concern was time to copy the image files to/from the hosts. A few of the images will be TB's and the copies will take a long time I think, even over GB links.<br><br>I do intend to create a separate cluster and datastore for these few hosts. I'll just have to try out a few different setups and see what works best.<br>
<br>You guys have given me some good ideas, thank you.<br><br>gary<br><br><hr><div style="font-size:12pt;font-style:normal;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal"><b>From: </b>"Ruben S. Montero" <<a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a>><div>
<br>
<b>To: </b>"Gary S. Cuozzo" <<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>><br><b>Cc: </b><a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a><br>
</div><b>Sent: </b>Tuesday, July 31, 2012 5:44:39 PM<div><div><br><b>Subject: </b>Re: [one-users] Storage architecture question<br><br>Hi,<div><br></div><div>You can set the Images Datastore of type FS to use the shared TM and your fast NAS, but the system datastore to use the ssh drivers. This will copy the images from the Datastore which uses NFS to the local storage area. </div>
<div><br></div><div>Note 1: This will only work for non-persistent images. Persistent images will be linked and thus used directly from the NAS server.</div><div><br></div><div>Note 2: you cannot mix both setups in the same cluster. This is, if your system datastore is shared the ssh transfer will copy the images to the NFS volume in the remote host.</div>
<div><br></div><div>You can:</div><div><br></div><div>1.- Create a dedicated cluster for this images that uses ssh as the system datastore (so the hosts do not mount the NFS export). You just need to add a few host and the datastore of the high demaning I/O images.The 3.6 scheduler will only use the resources of that cluster for that VMs [2]</div>
<div><br></div><div>2.- Modify the clone/mv/mvds scripts from a TM shared so it copy some of your images to a local disk and link them to the path expected by OpenNebula.</div><div><br></div><div><br></div><div>Cheers</div>
<div><br></div><div>Ruben</div><div><br></div><div>[1]<a href="http://opennebula.org/documentation:rel3.6:system_ds#using_the_ssh_transfer_driver" target="_blank">http://opennebula.org/documentation:rel3.6:system_ds#using_the_ssh_transfer_driver</a></div>
<div><br></div><div>[2]<a href="http://opennebula.org/documentation:rel3.6:cluster_guide" target="_blank">http://opennebula.org/documentation:rel3.6:cluster_guide</a></div><div><br></div><div><br><div class="gmail_quote">
On Mon, Jul 30, 2012 at 7:36 PM, Gary S. Cuozzo <span dir="ltr"><<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Tino,<br>
Thanks for the reply.<br>
<br>
Yes, I think you understand correctly. My goal is to be able to utilize storage which is local to a particular vm host node without incurring the overhead of duplicated storage on the controller node and transfer time from controller to vm host.<br>
<br>
I do understand that the images will only be accessible from the particular vm host which they reside on, but that is ok as it would be the trade-off for local disk performance. I have a great iSCSI/NFS SAN which is used for shared storage, but it will never be able to offer the same level of performance as local storage. So I'm looking to be able to have that local option for the few cases it's required for I/O intensive applications.<br>
<br>
I have not actually had the chance to try it out yet, but I think it will give me what I'm looking for.<br>
<br>
Thanks again,<br>
gary<br>
<div><div><br>
<br>
----- Original Message -----<br>
From: "Tino Vazquez" <<a href="mailto:tinova@opennebula.org" target="_blank">tinova@opennebula.org</a>><br>
To: "Gary S. Cuozzo" <<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>><br>
Cc: <a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a><br>
Sent: Monday, July 30, 2012 12:40:06 PM<br>
Subject: Re: [one-users] Storage architecture question<br>
<br>
Dear Gary,<br>
<br>
I am not sure I understand 100% your desired set-up, but if I grasped<br>
it correctly, I think the problem you may found is the images would<br>
only be local to the node that is exporting the NFS share. Otherwise I<br>
think it will work as you expect.<br>
<br>
Regards,<br>
<br>
-Tino<br>
<br>
--<br>
Constantino Vázquez Blanco, MSc<br>
Project Engineer<br>
OpenNebula - The Open-Source Solution for Data Center Virtualization<br>
<a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | @tinova79 | @OpenNebula<br>
<br>
<br>
On Fri, Jul 27, 2012 at 11:44 PM, Gary S. Cuozzo <<a href="mailto:gary@isgsoftware.net" target="_blank">gary@isgsoftware.net</a>> wrote:<br>
> Hi Users,<br>
> I am running ONE 3.6 and would like to be able to run a combination of<br>
> shared storage (via iSCSI) and local storage (to take advantage of local<br>
> disk performance for certain applications). My question is related to the<br>
> local storage aspect.<br>
><br>
> From what I've seen, I can use a local datastore and the ssh TM to<br>
> accomplish local storage. The drawback that I see is that I need 2x the<br>
> amount of disk space because I need storage for the permanent image on the<br>
> controller node, then storage on the local host for the running image when<br>
> it is deployed. A secondary issue for me is that the images have to be<br>
> transferred between the datastore and the host machine, which will take some<br>
> time with larger images.<br>
><br>
> To get around the problem, I thought I could set the datastore up as a<br>
> shared filesystem, except the sharing would actually be from the host<br>
> machine to the controller machine via NFS. Is there any particular<br>
> reason(s) that would be a bad idea? On the surface it seems like it should<br>
> work just fine, but I'm somewhat new to ONE and want to be sure I'm not<br>
> going down a bad path since I plan to do this with several host machines.<br>
><br>
> Thanks in advance for any advice.<br>
><br>
> gary<br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list<br>
> <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org<span><font color="#888888"><br clear="all"><div><br></div>
-- <br>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect<br>
OpenNebula - The Open Source Solution for Data Center Virtualization<br></font></span></a><span><font color="#888888"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula<br>
<br>
</font></span></div>
</div></blockquote></div></div>
</div></div></div><br></div></div></blockquote></div><div><div><br><br clear="all"><div><br></div>-- <br>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect<br>OpenNebula - The Open Source Solution for Data Center Virtualization<br>
<a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula<br>
</div></div></div>
</div><br></div></div><br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br></div>
<br clear="all"><div><br></div>-- <br>Jaime Melis<br>Project Engineer<br>OpenNebula - The Open Source Toolkit for Cloud Computing<br><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a><br>