<div><br></div><div>Hi, Matt</div><div><br></div><div>I have two solutions for this issue, for reference only:</div><div><br></div><div>1. Try qcow2 snapshot for creating new VMs in KVM environment</div><div>2. NFSv4 might be another option with server-side copy feature</div><div><br></div><div>Both could avoid coping large files between NFS server and client.</div><div><br></div><div>Thanks,</div><div>Lei</div><div><includetail><div><br></div><div><br></div><div style="font-size: 12px;font-family: Arial Narrow;padding:2px 0 2px 0;">------------------ Original ------------------</div><div style="font-size: 12px;background:#efefef;padding:8px;"><div><b>From: </b> "Matt Willsher"<matt.willsher@boxuk.com>;</div><div><b>Date: </b> Fri, Jun 22, 2012 07:06 PM</div><div><b>To: </b> "users@lists.opennebula.org"<users@lists.opennebula.org>; <wbr></div><div></div><div><b>Subject: </b> Re: [one-users] NFS datastore performance during VM instantiation</div></div><div><br></div>
<style><!--
@font-face
{font-family:"Cambria Math";
panose-1:0 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><![endif]--><!--[if gte mso 9]><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D ">Hi Ruben,<o:p></o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D "><o:p> </o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D ">Thanks for your reply. We came to the same conclusion in the end; we need a more suitable file system to store the data. I did try to tune the NFS client without
any improvement and the NFS server (an EMC VNXe3100) doesn¡¯t have anything to tune, though it does use async out of the box. We¡¯ll investigate other options.<o:p></o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D "><o:p> </o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D ">Best regards,<o:p></o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D ">Matt<o:p></o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D "><o:p> </o:p></span></p>
<p class="MsoNormal"><span style=" font-size:11.0pt ; ; ; ;;color:#1F497D "><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US" style=" font-size:10.0pt ; ; ; ; ">From:</span></b><span lang="EN-US" style=" font-size:10.0pt ; ; ; ; "> Ruben S. Montero [mailto:rsmontero@opennebula.org]
<br>
<b>Sent:</b> 22 June 2012 11:41<br>
<b>To:</b> Matt Willsher<br>
<b>Cc:</b> users@lists.opennebula.org<br>
<b>Subject:</b> Re: [one-users] NFS datastore performance during VM instantiation<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hi Matt,<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">NFS is not a valid solution for medium size deployments, specially when there is contention on the network (deploying multiple VMs at the same time, multiple VMs with heavy I/O concurrently). The solution could be based on two basic alternatives
(or a combination of both):<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">1.- Use a better NFS server (SAN) or a better distributed/shared FS (gluster, lustre...)<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">2.- Balance the NFS load by using different servers. Simple configure several datastores mounting volumes from different NFS servers. You may also want to tune the NFS mounting parameters caches, syncs,.....<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Hope it helps <o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Ruben<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Jun 15, 2012 at 12:43 PM, Matt Willsher <<a href="mailto:matt.willsher@boxuk.com" target="_blank">matt.willsher@boxuk.com</a>> wrote:<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Hi,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">I¡¯m currently working on Open Nebula using NFS as a shared data store. I¡¯ve found that during the large copy done during instantiation all other VMs on the same node suffer storage
performance issues and are unable to write to disk. In some circumstances this produces warnings in the VM¡¯s kernel due to time outs. I¡¯ve been over the system configuration, tried different NFS options, sync, async, and others. Nfsiostat is showing avg exe
times to be 40+ seconds during these large sequential writes. There are no errors in either NFS or network stats and the NFS server is responsive when accessed from other clients.
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">My question is, has anyone else come across these sorts of performance issues during VM instantiation when using NFS a shared datastore?
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Thanks,<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Matt
<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
</div>
<p class="MsoNormal"><br clear="all">
<o:p></o:p></p>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists..opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <br>
Ruben S. Montero, PhD<br>
Project co-Lead and Chief Architect<br>
OpenNebula - The Open Source Solution for Data Center Virtualization<br>
<a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">
rsmontero@opennebula.org</a> | @OpenNebula<o:p></o:p></p>
</div>
</div>
<br clear="all"></includetail></div>