There was a post earlier with similar setup details<br><br><a href="http://www.mail-archive.com/users@lists.opennebula.org/msg05546.html">http://www.mail-archive.com/users@lists.opennebula.org/msg05546.html</a><br><br>About the performance I think their FAQ has some answers<br>
<a href="http://www.moosefs.org/moosefs-faq.html">http://www.moosefs.org/moosefs-faq.html</a><br><br>The problem with moosefs is mds is your single point of failure<br><br>HTH<br>Ranga<br><br><div class="gmail_quote">On Wed, Mar 14, 2012 at 6:34 PM, Marshall Grillos <span dir="ltr"><<a href="mailto:mgrillos@optimalpath.com">mgrillos@optimalpath.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div link="blue" vlink="purple" lang="EN-US">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks for the update and information.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">What about the possibility of using a distributed file system (say MooseFS and ucarp for HA) and designating each front-end/controller host as a chunk server?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In that setup, there would only be 2 chunk servers, each with a large array attached. Would the file system remain intact if one chunk server failed (given
the “goal” value was set at 2 or greater) or do you really need additional chunk servers for fault-tolerance to function?
<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">What about the performance of MooseFS across 2 chunk servers with large disk arrays (attached as outlined below utilizing 10GbE)? Does it perform well (we
will be using SATA drives in a Raid 10 configuration)?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Marshall<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Ranga Chakravarthula [mailto:<a href="mailto:rbabu@hexagrid.com" target="_blank">rbabu@hexagrid.com</a>]
<br>
<b>Sent:</b> Wednesday, March 14, 2012 4:08 PM</span></p><div><div class="h5"><br>
<b>To:</b> Marshall Grillos<br>
<b>Cc:</b> <a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a><br>
<b>Subject:</b> Re: [one-users] Shared File System HA<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">It is plain nfsclient to nfsserver behavior. Hypervisor is just acting as NFS client. The OS of the VM is caching the writes in memory and periodically writing to Hard disk. During the failover the NFS client will continue to try to write
but will fail if it cannot connect to the NFS server before the timeout happens. If connection is re-established, all the writes will go thru.<br>
<br>
You need to see the NFS options<u></u><u></u></p>
<pre><b>timeo<br>retrans<br>retry</b><u></u><u></u></pre>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed, Mar 14, 2012 at 12:49 PM, Marshall Grillos <<a href="mailto:mgrillos@optimalpath.com" target="_blank">mgrillos@optimalpath.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">In my design I’m looking at having the shared storage attached to the front-end server and provide
full redundancy for both the front-end and the image repository. This would then be shared to each compute node via NFS.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">StorageArray1 ---DAS--->FrontEnd1---10gb Eth---->BladeChassis1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">|</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">|</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">DRDB/Heartbeat/Pacemaker (between FrontEnd nodes)</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">|</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">|</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">StorageArray2 ---DAS--->FrontEnd2---10gb Eth----> BladeChassis1</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I planned on setting up an active/passive cluster for two front-end servers. These would have completely
separate storage arrays (potentially in separate data centers). Using DRBD (I’m open to other solutions if they provide faster failover) the image repository would be mirrored between the storage devices. In the event of any hardware failure (NIC/Controller/Power
etc) a full failover would occur from Frontend1 to Frontend2 propagating the cluster IP address.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">With this setup, there would be a lag time for the heartbeat/pacemaker to detect the failover and
the failover to occur (possibly upwards of 30 seconds). What will happen to the running VMs when the failover is performed? Is the computing node hypervisor “smart” enough to handle a several second NFS outage?</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I’m definitely open to other solutions GlusterFS etc if they provide a smoother failover transition
given my existing hardware configuration.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Thanks,</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Marshall
</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Ranga Chakravarthula [mailto:<a href="mailto:rbabu@hexagrid.com" target="_blank">rbabu@hexagrid.com</a>]
<br>
<b>Sent:</b> Wednesday, March 14, 2012 10:57 AM<br>
<b>To:</b> Marshall Grillos<br>
<b>Cc:</b> <a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a><br>
<b>Subject:</b> Re: [one-users] Shared File System HA</span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">If you are looking at HA at storage level, it would be better you have Heartbeat/Failover on the NFS resource than failing over to secondary front-end server. Anyway your NFS is mounted
on the compute nodes and if one storage goes down, heartbeat will failover to another storage. Your frontend doesn't have to part of this.<u></u><u></u></p>
<div>
<p class="MsoNormal">On Wed, Mar 14, 2012 at 10:26 AM, Marshall Grillos <<a href="mailto:mgrillos@optimalpath.com" target="_blank">mgrillos@optimalpath.com</a>> wrote:<u></u><u></u></p>
<div>
<div>
<p class="MsoNormal">I am debating the differences between Shared and Non-shared file systems for an OpenNebula deployment.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">One concern with the shared file system is High Availability. I am setting up the OpenNebula front-end with connectivity to a storage device. To avoid the event of a storage device
failure (RAID controller, Power, etc) I am looking into setting up a secondary front-end server with attached storage. I would use NFS to share the storage to each VM Host and setup DRDB for block level replication between each cluster node. In the event
of a storage failure, a failover would occur utilizing heartbeat/pacemaker to the secondary front-end server.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">If anyone has tested a similar setup how do the VMs handle the minimal outage required for the failover to occur (the several seconds required to failover to the secondary front-end)?
For a certain duration, wouldn’t the NFS mount be unavailable due to the failover mechanism?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Marshall<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><u></u><u></u></p>
</div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div></div></div>
</div>
</blockquote></div><br>