<div dir="ltr">Hi<div><br></div><div>In general, we cannot assume that there will be access to the host in a failure state. The current delete/recreate mechanism is based on the following:</div><div><br></div><div>* If there are a shared FS (a reliable one) then recreation would not affect if the VM disks are persistent. Changes will be available when the VM is recreated, even if the disk is deleted (just the symbolic link is removed)</div>
<div><br></div><div>* If the disk is not persistent, you will lost the changes, even if you have a shared FS. However, a HA service should not be based on non-persistent disks, as OpenNebula assumes that the changes can be disposed.</div>
<div><br></div><div>* If there is no shared FS, and the host is down, then there is no hope to get the disk out of it.</div><div><br></div><div>With this in mind the current OPenNebula behavior should work, provided that *YOU HAVE* a working fencing mechanism for the physical hosts. </div>
<div><br></div><div>You need to fence the falling host in the ha-hook, to prevent a split brain condition on the shared disks.</div><div><br></div><div>Cheers</div><div><br></div><div>Ruben</div><div><br></div><div><br></div>
<div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Dec 5, 2013 at 3:45 PM, Dmitri Chebotarov <span dir="ltr"><<a href="mailto:dchebota@gmu.edu" target="_blank">dchebota@gmu.edu</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>
<div>Hi,</div>
<div><br>
</div>
<div>Did you ever figured out how to “move” VM in case when VM host goes down? </div>
<div>I ran into the same issue last night. </div>
<div>RHEL6 cluster, same OS/KVM version. One of them VM hosts went down (error) and VMs running on that host were recreated on available hosts.</div>
<div>Recreated VMs lost work progress.</div>
<div><br>
</div>
<div>RHEL6 cluster is using shared NFS storage for system and “data” datastores.</div>
<div>Once host died, ONE attempts to connect to the dead host to access system datastore, which is already mounted on ONE controller under the same path (log below).</div>
<div>This is how system datastore configured:</div>
<div><br>
</div>
<div>TYPE: SYSTEM_DS</div>
<div>DISK_TYPE: file</div>
<div>TM_MAD: shared.</div>
<div><br>
</div>
<div>It’s mounted on all cluster nodes and ONED controller under the same path (/var/lib/one).</div>
<div>I’m probably missing something in system datastore configuration, which would tell ONED to access it locally, not via dead VM host…</div>
<div><br>
</div>
<div>Shouldn’t ONED start VMs on available host using existing config/disk files in system datastore?</div>
<div>And not delete/recreated it?</div>
<div><br>
</div>
<div>Thank you.</div>
<div><br>
</div>
<div><span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:29 2013 [VMM][I]: Command execution fail: /var/tmp/one/vnm/ovswitch/clean </span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:29 2013 [VMM][I]: ssh: connect to host BC4-10 port 22: No route to host</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:29 2013 [VMM][I]: ExitSSHCode: 255</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(255,141,129);font-family:'Open Sans',sans-serif;line-height:28px">Thu Dec 5 04:49:29 2013 [VMM][E]: Error connecting to BC4-10</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:29 2013 [VMM][I]: Failed to execute network driver operation: clean.</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:32 2013 [VMM][I]: Command execution fail: /var/lib/one/remotes/tm/qcow2/delete BC4-10:/var/lib/one//datastores/111/11251/disk.0
11251 107</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:32 2013 [VMM][I]: delete: Deleting /var/lib/one/datastores/111/11251/disk.0</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(255,141,129);font-family:'Open Sans',sans-serif;line-height:28px">Thu Dec 5 04:49:32 2013 [VMM][E]: delete: Command "rm -rf /var/lib/one/datastores/111/11251/disk.0" failed: ssh: connect
to host BC4-10 port 22: No route to host</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(255,141,129);font-family:'Open Sans',sans-serif;line-height:28px">Thu Dec 5 04:49:32 2013 [VMM][E]: Error deleting /var/lib/one/datastores/111/11251/disk.0</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:32 2013 [VMM][I]: ExitCode: 255</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:32 2013 [VMM][I]: Failed to execute transfer manager driver operation: tm_delete.</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:35 2013 [VMM][I]: Command execution fail: /var/lib/one/remotes/tm/shared/delete BC4-10:/var/lib/one//datastores/111/11251
11251 111</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:35 2013 [VMM][I]: delete: Deleting /var/lib/one/datastores/111/11251</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(255,141,129);font-family:'Open Sans',sans-serif;line-height:28px">Thu Dec 5 04:49:35 2013 [VMM][E]: delete: Command "rm -rf /var/lib/one/datastores/111/11251" failed: ssh: connect
to host BC4-10 port 22: No route to host</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(255,141,129);font-family:'Open Sans',sans-serif;line-height:28px">Thu Dec 5 04:49:35 2013 [VMM][E]: Error deleting /var/lib/one/datastores/111/11251</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:35 2013 [VMM][I]: ExitCode: 255</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:35 2013 [VMM][I]: Failed to execute transfer manager driver operation: tm_delete.</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:35 2013 [VMM][I]: Host successfully cleaned.</span><br style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px">
<span style="color:rgb(242,242,242);font-family:'Open Sans',sans-serif;line-height:28px;background-color:rgb(111,111,111)">Thu Dec 5 04:49:35 2013 [DiM][I]: New VM state is PENDING</span></div>
<div>— <br>
Thank you,<br>
<br>
Dmitri Chebotarov<br>
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging<br>
223 Aquia Building, Ffx, MSN: 1B5<br>
Phone: <a href="tel:%28703%29%20993-6175" value="+17039936175" target="_blank">(703) 993-6175</a> | Fax: <a href="tel:%28703%29%20993-3404" value="+17039933404" target="_blank">(703) 993-3404</a><br style="font-family:Helvetica;font-size:medium">
<br style="font-family:Helvetica;font-size:medium">
</div>
</div>
<div><br>
</div>
<span>
<div style="border-right:medium none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium none;font-family:Calibri;border-top:#b5c4df 1pt solid;padding-bottom:0in;border-left:medium none">
<span style="font-weight:bold">From: </span>Carlos Martín Sánchez <<a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a>><br>
<span style="font-weight:bold">Date: </span>Wednesday, September 11, 2013 at 5:44
<br>
<span style="font-weight:bold">To: </span>Romany Nageh <<a href="mailto:engromanynageh@gmail.com" target="_blank">engromanynageh@gmail.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>" <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [one-users] Fwd: how running vms moved(not recreate) on another host on host error<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Hi,
<div><br>
</div>
<div>What do you exactly mean by "move"? If you are referring to migration, that's not possible, once a host goes down, the VM state is lost.</div>
<div><br>
</div>
<div>Regards</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div dir="ltr">--<br>
Join us at <a href="http://opennebulaconf.com" target="_blank">OpenNebulaConf2013</a> in Berlin, 24-26 September, 2013<br>
--
<div>Carlos Martín, MSc<br>
Project Engineer<br>
OpenNebula - The Open-source Solution for Data Center Virtualization
<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> |
<a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> |
<a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="mailto:cmartin@opennebula.org" style="color:rgb(42,93,176)" target="_blank"></a></span></div>
</div>
</div>
</div>
<br>
<br>
<div class="gmail_quote">On Tue, Sep 10, 2013 at 11:47 PM, Romany Nageh <span dir="ltr">
<<a href="mailto:engromanynageh@gmail.com" target="_blank">engromanynageh@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<p>HI <br>
i am using opennebula 4.2 how to handle vms running on specific host to move (not recreate) to another host when host error(down)</p>
</div>
<p></p>
<div>please could any on help me ?<br>
</div>
Thanks
<p></p>
<div>
<div>
<div class="gmail_quote">---------- Forwarded message ----------<br>
From: "Romany Nageh" <<a href="mailto:engromanynageh@gmail.com" target="_blank">engromanynageh@gmail.com</a>><br>
Date: Sep 9, 2013 9:46 PM<br>
Subject: how running vms moved(not recreate) on another host on host error<br>
To: <<a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a>>, "Carlos Martín Sánchez" <<a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a>><br>
<br type="attribution">
<div dir="ltr">
<div>HI<br>
i am using opennebula 4.2<br>
</div>
<div>how to handle vms running on specific host to move (not recreate) to another host when host error(down)<br>
</div>
<div><br>
please could any on help me ?<br>
<br>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span>
</div>
<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div dir="ltr"><div><div>-- <br></div></div>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect<div>OpenNebula - Flexible Enterprise Cloud Made Simple<br>
<a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula</div></div>
</div>