<div dir="ltr">Hi Matthew,<div><br></div><div>The stock ha-restart scripts needs to include a proper fencing mechanism for the vm hosts. This is needed to prevent the split-brain conditions described in your email.</div><div>
<br></div><div>Simply include the fencing command in the hook (you have the hostname of the target host in the script, so it should be straight-forward). This will typically reboot the host, shutdown any VM in it.</div><div>
<br></div><div>Cheers</div><div><br></div><div>Ruben</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jan 9, 2014 at 5:39 PM, Matthew Richardson <span dir="ltr"><<a href="mailto:m.richardson@ed.ac.uk" target="_blank">m.richardson@ed.ac.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I'm running a ONE 4.2 pool, and had some issues with it earlier today.<br>
<br>
I had some vm hosts lock up due to networking issues, where the vm hosts<br>
could see the rest of the world, but not be reached by the ONE server.<br>
<br>
As a result, the ONE server called a hook script:<br>
<br>
VM_HOOK = [ name = "on_crash_boot", on = "UNKNOWN", command =<br>
"/usr/bin/env onevm boot", arguments = "$ID" ]<br>
<br>
This resulted in an attempted cleanup (which appears to fail due to the<br>
ongoing network problems) followed by a restart elsewhere. However, the<br>
failed cleanup meant that I then had 2 instances of the same guest<br>
running on 2 vm hosts, which led to mac address conflicts on the network.<br>
<br>
Is this a bug in ONE's handling of cleanup failure, or is there<br>
something else I should be doing in my hook script to ensure that it is<br>
safe to call onevm boot?<br>
<br>
Any advice appreciated! (other than to take better care of the network :) )<br>
<br>
thanks,<br>
<br>
Matthew<br>
<br>
<br>
oned.log starts as follows:<br>
<br>
Thu Jan 9 08:13:07 2014 [InM][I]: Command execution fail: 'if [ -x<br>
"/var/tmp/one/im/run_probes" ]; then /var/tmp/one/im/run_probes kvm 2<br>
vmhost3; else exit 42; fi'<br>
Thu Jan 9 08:13:07 2014 [InM][I]: Connection closed by 192.168.12.16<br>
Thu Jan 9 08:13:07 2014 [InM][I]: ExitCode: 255<br>
Thu Jan 9 08:13:07 2014 [ONE][E]: Error monitoring Host vmhost3 (2): -<br>
Thu Jan 9 08:13:07 2014 [ReM][D]: Req:3296 UID:0 VirtualMachineAction<br>
invoked, "boot", 14<br>
Thu Jan 9 08:13:07 2014 [DiM][D]: Restarting VM 14<br>
Thu Jan 9 08:13:07 2014 [ReM][D]: Req:3296 UID:0 VirtualMachineAction<br>
result SUCCESS, 14<br>
Thu Jan 9 08:13:07 2014 [HKM][D]: Message received: EXECUTE SUCCESS 14<br>
on_crash_boot:<br>
<br>
Thu Jan 9 08:13:08 2014 [ReM][D]: Req:3328 UID:0 VirtualMachineInfo<br>
invoked, 14<br>
Thu Jan 9 08:13:08 2014 [ReM][D]: Req:3328 UID:0 VirtualMachineInfo<br>
result SUCCESS, "<VM><ID>14</ID><UID>..."<br>
<br>
Thu Jan 9 08:13:08 2014 [ReM][D]: Req:9328 UID:0 VirtualMachineAction<br>
invoked, "delete-recreate", 14<br>
Thu Jan 9 08:13:08 2014 [ReM][D]: Req:9328 UID:0 VirtualMachineAction<br>
result SUCCESS, 14<br>
<br>
Thu Jan 9 08:13:08 2014 [VMM][D]: Message received: LOG I 14 Driver<br>
command for 14 cancelled<br>
<br>
<br>
<br>
The (slightly redacted) guest log (14.log) is as follows:<br>
<br>
Thu Jan 9 07:44:53 2014 [LCM][I]: New VM state is RUNNING<br>
Thu Jan 9 08:13:07 2014 [LCM][I]: New VM state is UNKNOWN<br>
Thu Jan 9 08:13:07 2014 [LCM][I]: New VM state is BOOT_UNKNOWN<br>
Thu Jan 9 08:13:07 2014 [HKM][I]: Success executing Hook: on_crash_boot: .<br>
Thu Jan 9 08:13:07 2014 [VMM][I]: Generating deployment file:<br>
/var/lib/one/vms/14/deployment.4917<br>
Thu Jan 9 08:13:08 2014 [LCM][I]: New VM state is CLEANUP.<br>
Thu Jan 9 08:13:08 2014 [VMM][I]: Driver command for 14 cancelled<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: Command execution fail:<br>
/var/tmp/one/vmm/kvm/cancel 'one-14' 'vmhost3' 14 vmhost3<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: Connection closed by 192.168.12.16<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: ExitSSHCode: 255<br>
Thu Jan 9 08:18:52 2014 [VMM][E]: Error connecting to vmhost3<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: Failed to execute virtualization<br>
driver operation: cancel.<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: Command execution fail:<br>
/var/tmp/one/vnm/dummy/clean <...snip...><br>
Thu Jan 9 08:18:52 2014 [VMM][I]: Connection closed by 192.168.12.16<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: ExitSSHCode: 255<br>
Thu Jan 9 08:18:52 2014 [VMM][E]: Error connecting to vmhost3<br>
Thu Jan 9 08:18:52 2014 [VMM][I]: Failed to execute network driver<br>
operation: clean.<br>
Thu Jan 9 08:19:01 2014 [VMM][I]: Successfully execute transfer manager<br>
driver operation: tm_delete.<br>
Thu Jan 9 08:19:02 2014 [VMM][I]: Successfully execute transfer manager<br>
driver operation: tm_delete.<br>
Thu Jan 9 08:19:02 2014 [VMM][I]: Host successfully cleaned.<br>
Thu Jan 9 08:19:03 2014 [DiM][I]: New VM state is PENDING<br>
Thu Jan 9 08:20:54 2014 [DiM][I]: New VM state is ACTIVE.<br>
Thu Jan 9 08:20:54 2014 [LCM][I]: New VM state is PROLOG.<br>
Thu Jan 9 08:20:54 2014 [VM][I]: Virtual Machine has no context<br>
Thu Jan 9 08:20:54 2014 [LCM][I]: New VM state is BOOT<br>
Thu Jan 9 08:20:54 2014 [VMM][I]: Generating deployment file:<br>
/var/lib/one/vms/14/deployment.4918<br>
Thu Jan 9 08:20:56 2014 [VMM][I]: ExitCode: 0<br>
Thu Jan 9 08:20:56 2014 [VMM][I]: Successfully execute network driver<br>
operation: pre.<br>
Thu Jan 9 08:20:56 2014 [VMM][I]: ExitCode: 0<br>
Thu Jan 9 08:20:56 2014 [VMM][I]: Successfully execute virtualization<br>
driver operation: deploy.<br>
Thu Jan 9 08:20:56 2014 [VMM][I]: ExitCode: 0<br>
Thu Jan 9 08:20:56 2014 [VMM][I]: Successfully execute network driver<br>
operation: post.<br>
Thu Jan 9 08:20:56 2014 [LCM][I]: New VM state is RUNNING<br>
<span><font color="#888888"><br>
<br>
<br>
--<br>
The University of Edinburgh is a charitable body, registered in<br>
Scotland, with registration number SC005336.<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org<br clear="all"><div><br></div>-- <br></a><div dir="ltr"><a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank"><div>
<div>-- <br></div></div>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect</a><div><a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">OpenNebula - Flexible Enterprise Cloud Made Simple<br>
</a><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula</div></div>
</font></span></blockquote></div></div></div>