<div dir="ltr">I don't know why I'm offering more solutions to you, but anyway.<div><br></div><div>From what I can see the state trace for a successful migrate is RUNNING->MIGRATE->RUNNING, and that it is the same for a failed migrate. You could write a hook that triggers on state RUNNING and checks for previous state MIGRATE. The issue is you would have to maintain some state info in your add-on by writing a "start migration" flag in your pre-migrate script that gets cleaned up by your post-migrate script, when migrate succeeds, or the proposed hook when the migrate fails. Alternatively you could forgo maintaining your own state and use the hook to parse the VM's log for migration failed messages.</div>
<div><br></div><div>A permanent fix would be for OpenNebula to add a new state MIGRATE_FAILED that a developer could use to distinguish between a successful and failed migrate. The failed migrate state trace would then be RUNNING->MIGRATE->MIGRATE_FAILED->RUNNING</div>
<div><br></div><div>Regards,</div><div>David</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Jan 20, 2014 at 3:23 PM, Gareth Bult <span dir="ltr"><<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div style="font-size:12pt;font-family:trebuchet ms,sans-serif"><div><div class="im"><ol style="font-family:Helvetica,Arial,sans-serif">
<li>OK, so you are quite defensive and rude about this for some reason</li></ol></div>You are suggesting modifying and maintaining (currently) 5 migrate scripts, then redistributing (and keeping up-to-date)</div><div>90 other scripts, just to cope with a current deficiency with the current mechanism available for add-ons with regards to</div>
<div>hooking into the current infrastructure. </div><div><br></div><div>That was me being restrained.</div><div class="im"><div><br></div><div><span style="font-family:Helvetica,Arial,sans-serif">>I know it's not as elegant as the four line solution but point 2.</span></div>
<div><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div></div><div><span style="font-family:Helvetica,Arial,sans-serif">To put it mildly. </span></div><div><span style="font-family:Helvetica,Arial,sans-serif"><br>
</span></div><div><span style="font-family:Helvetica,Arial,sans-serif">I'm *not* trying to insist my solution is included, but there needs to be "a" solution and generating lots more</span></div><div><span style="font-family:Helvetica,Arial,sans-serif">duplicate code to install and redistribute simply isn't "it" and I'm more than a little shocked anyone would suggest it!</span></div>
<div class="im"><div><span style="font-family:Helvetica,Arial,sans-serif"><br></span></div><div><br></div><div><span name="x"></span><div><div><span size="2" style="font-size:small">--</span></div><div><table><tbody><tr><td>
<img src="http://linuxforums.org.uk/index.php?action=dlattach;attach=262;type=avatar"></td><td><div><b style="font-family:verdana,Helvetica,sans-serif;font-size:11px;line-height:14px">Gareth Bult</b><br><span color="#666666" style="color:#666666"><span face="verdana, Helvetica, sans-serif" style="font-family:verdana,Helvetica,sans-serif"><span style="font-size:11px;line-height:14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span color="#666666" style="color:#666666"><br>
</span><span color="#666666" style="color:#666666"><span style="color:#666666;font-family:verdana,Helvetica,sans-serif;font-size:11px;line-height:14px">See the status of my current project at <a href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div>
<div><span color="#666666" style="color:#666666"><span face="verdana, Helvetica, sans-serif" style="font-family:verdana,Helvetica,sans-serif"><span style="font-size:11px;line-height:14px"><br></span></span></span></div></td>
</tr></tbody></table></div></div><span name="x"></span><br></div><hr></div><div style="font-size:12pt;font-style:normal;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal"><div class="im"><b>From: </b>"David Macleod" <<a href="mailto:dmacleod@csir.co.za" target="_blank">dmacleod@csir.co.za</a>><br>
<b>To: </b>"Gareth Bult" <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>><br><b>Cc: </b>"Jaime Melis" <<a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a>>, <a href="mailto:dev@lists.opennebula.org" target="_blank">dev@lists.opennebula.org</a><br>
</div><b>Sent: </b>Monday, 20 January, 2014 12:50:14 PM<div><div class="h5"><br><b>Subject: </b>Re: [one-dev] Live migration / recovery - suggestions<br><div><br></div><div dir="ltr"><ol><li>OK, so you are quite defensive and rude about this for some reason.<br>
</li><li>The OpenNebula project has just said they don't want to change their code base to suit you, so your four line solution is off the table.<br>
</li><li>You don't need to write the vmm drivers from scratch. Just copy the three provided with ONE call them kvm-vdc, xen-vdc and vmware-vdc or something, then make your customization to them. I know it's not as elegant as the four line solution but point 2.</li>
</ol></div><div class="gmail_extra"><br><div><br></div><div class="gmail_quote">On Mon, Jan 20, 2014 at 2:23 PM, Gareth Bult <span dir="ltr"><<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="font-size:12pt;font-family:trebuchet ms,sans-serif"><div>Once I switch to a different vmm (say "vdc" rather than "kvm") , how will the system know to fall back on</div>
<div>the hypervisor in use for all the other scripts that I don't want to modify?</div><div><div><br></div><div><span></span><div><div><span style="font-size:small">--</span></div><div><table>
<tbody><tr><td><img src="http://linuxforums.org.uk/index.php?action=dlattach;attach=262;type=avatar"></td><td><div><b style="font-family:verdana,Helvetica,sans-serif;font-size:11px;line-height:14px">Gareth Bult</b><br><span style="color:#666666"><span style="font-family:verdana,Helvetica,sans-serif"><span style="font-size:11px;line-height:14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span style="color:#666666"><br>
</span><span style="color:#666666"><span style="color:#666666;font-family:verdana,Helvetica,sans-serif;font-size:11px;line-height:14px">See the status of my current project at <a href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div>
<div><span style="color:#666666"><span style="font-family:verdana,Helvetica,sans-serif"><span style="font-size:11px;line-height:14px"><br></span></span></span></div></td>
</tr></tbody></table></div></div><span></span><br></div><hr></div><div style="font-size:12pt;font-style:normal;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal"><div><b>From: </b>"David Macleod" <<a href="mailto:dmacleod@csir.co.za" target="_blank">dmacleod@csir.co.za</a>><br>
<b>To: </b>"Gareth Bult" <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>><br><b>Cc: </b>"Jaime Melis" <<a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a>>, <a href="mailto:dev@lists.opennebula.org" target="_blank">dev@lists.opennebula.org</a><br>
</div><b>Sent: </b>Monday, 20 January, 2014 11:57:31 AM<div><br><b>Subject: </b>Re: [one-dev] Live migration / recovery - suggestions<br><div><br></div></div>No, you should write a basic framework around the migrate script that supports the three hypervisors that OpenNebula uses.<br>
<div><br></div>>>> Gareth Bult <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>> 2014/01/20 02:15 PM >>><div><div><br><div style="font-size:12pt;font-family:trebuchet ms,sans-serif">
<div>Erm, yeah, Ok, should I include filters for the coffee machine too ?!<br></div><div><br></div><div><span></span><div><div><span style="FONT-SIZE:small">--</span></div><div><table><tbody><tr><td><img src="https://mail.linux.co.uk/service/home/~/?auth=co&id=504676&part=4"></td>
<td><div><b style="FONT-SIZE:11px;FONT-FAMILY:verdana,Helvetica,sans-serif;LINE-HEIGHT:14px">Gareth Bult</b><br><span style="COLOR:#666666"><span style="FONT-FAMILY:verdana,Helvetica,sans-serif"><span style="FONT-SIZE:11px;LINE-HEIGHT:14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span style="COLOR:#666666"><br>
</span><span style="COLOR:#666666"><span style="FONT-SIZE:11px;FONT-FAMILY:verdana,Helvetica,sans-serif;COLOR:#666666;LINE-HEIGHT:14px">See the status of my current project at <a href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div>
<div><span style="COLOR:#666666"><span style="FONT-FAMILY:verdana,Helvetica,sans-serif"><span style="FONT-SIZE:11px;LINE-HEIGHT:14px"><br></span></span></span></div></td></tr></tbody></table></div></div><span></span><br>
</div>
<hr><div style="font-style:normal;font-size:12pt;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal"><b>From: </b>"David Macleod" <<a href="mailto:dmacleod@csir.co.za" target="_blank">dmacleod@csir.co.za</a>><br>
<b>To: </b>"Gareth Bult" <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>><br><b>Cc: </b>"Jaime Melis" <<a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a>>, <a href="mailto:dev@lists.opennebula.org" target="_blank">dev@lists.opennebula.org</a><br>
<b>Sent: </b>Monday, 20 January, 2014 11:50:26 AM<br><b>Subject: </b>Re: [one-dev] Live migration / recovery - suggestions<br><div><br></div>Then distribute a vmm driver per hypervisor...<br><div><br></div>>>> Gareth Bult <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>> 2014/01/20 02:01 PM >>><br>
<div style="font-size:12pt;font-family:trebuchet ms,sans-serif"><div>.. Erm, because the vmm driver is per-hypervisor .. and this product is technically<br></div><div>hypervisor agnostic .. ??</div><div><br></div><div><br>
</div><div><span></span><div><div><span style="FONT-SIZE:small">--</span></div><div><table><tbody><tr><td><img src="https://mail.linux.co.uk/service/home/~/?auth=co&id=504676&part=2"></td><td><div><b style="FONT-SIZE:11px;FONT-FAMILY:verdana,Helvetica,sans-serif;LINE-HEIGHT:14px">Gareth Bult</b><br>
<span style="COLOR:#666666"><span style="FONT-FAMILY:verdana,Helvetica,sans-serif"><span style="FONT-SIZE:11px;LINE-HEIGHT:14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span style="COLOR:#666666"><br>
</span><span style="COLOR:#666666"><span style="FONT-SIZE:11px;FONT-FAMILY:verdana,Helvetica,sans-serif;COLOR:#666666;LINE-HEIGHT:14px">See the status of my current project at <a href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div>
<div><span style="COLOR:#666666"><span style="FONT-FAMILY:verdana,Helvetica,sans-serif"><span style="FONT-SIZE:11px;LINE-HEIGHT:14px"><br></span></span></span></div></td></tr></tbody></table></div></div><span></span><br>
</div>
<hr><div style="font-style:normal;font-size:12pt;font-family:Helvetica,Arial,sans-serif;text-decoration:none;font-weight:normal"><b>From: </b>"David Macleod" <<a href="mailto:dmacleod@csir.co.za" target="_blank">dmacleod@csir.co.za</a>><br>
<b>To: </b>"Gareth Bult" <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>>, "Jaime Melis" <<a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a>><br>
<b>Cc: </b><a href="mailto:dev@lists.opennebula.org" target="_blank">dev@lists.opennebula.org</a><br><b>Sent: </b>Monday, 20 January, 2014 11:32:33 AM<br><b>Subject: </b>Re: [one-dev] Live migration / recovery - suggestions<br>
<div><br></div><div>Hi Gareth, Jaime</div><div> </div><div>Why can't the recovery be handled by the migrate script? You can distribute your own custom vmm driver so that it doesn't get overwritten by the updates. That's how I wrote my add-on, I also have the need to recover from a migrate fail.</div>
<div> </div><div>Regards,</div><div>David</div><div><br>>>> Jaime Melis <<a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a>> 2014/01/20 12:43 PM >>><br></div><div dir="ltr">
<div>Hi Gareth,</div><div><br></div><div>we've been studying your proposal, and even though we agree with what you say we</div><div>aren't 100% convinced with this solution. The issues with the proposal are the</div>
<div>following:</div><div><br></div><div>- As long as it's possible, we'd like to keep separate the main opennebula code</div><div>and the addons. In this case it means that we are not very comfortable with</div>
<div>
point 3: adding addon-specific code to the main repository.</div><div><br></div><div>- The proposed solution only solves the "migrate" issue, but other addons will</div><div>have potentially issues with other scripts, and not necessarily with the</div>
<div>"CleanUp" part of the "ssh_exec_and_log". We would like to find a more general</div><div>solution.</div><div><br></div><div>We are still thinking about this, we definitely want to solve this issue, so if</div>
<div>you (or anyone else) has any ideas, please let us know.</div><div><br></div><div>cheers,</div><div>Jaime</div></div><div class="gmail_extra"><br><div><br></div><div class="gmail_quote">On Tue, Jan 14, 2014 at 2:23 PM, Gareth Bult <span dir="ltr"><<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT:1ex;MARGIN:0px 0px 0px 0.8ex;BORDER-LEFT:#ccc 1px solid"><div style="FONT-SIZE:12pt;FONT-FAMILY:trebuchet ms,sans-serif"><div>Hey Guys, I've done a little work on the migration script - <span style="FONT-SIZE:12pt">this is what I've done here ..</span></div>
<div><span style="FONT-SIZE:12pt">- be nice if something similar could be implemented @ source .. ?</span></div><div><br></div><div>1. ssh_exec_and_log (generic change - this could be useful elsewhere..) modify as follows;</div>
<div><p><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">function ssh_exec_and_log</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">{</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">message=$2</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif"><span style="COLOR:rgb(255,0,0)">cleanup=$3 # ++</span></span></p>
<p><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">EXEC_LOG_ERR=`$1 2>&1 1>/dev/null`</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">EXEC_LOG_RC=$?</span></p>
<p><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">if [ $EXEC_LOG_RC -ne 0 ]; then</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">log_error "Command \"$1\" failed: $EXEC_LOG_ERR"</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif"><span style="COLOR:rgb(255,0,0)">if [ ! -z $cleanup ]; then # ++</span></span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif;COLOR:rgb(255,0,0)">$cleanup # ++</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif;COLOR:rgb(255,0,0)">fi # ++<br></span></p><p><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">if [ -n "$2" ]; then</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">error_message "$2"</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">else</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">error_message "Error executing $1: $EXEC_LOG_ERR"</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">fi</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">return $EXEC_LOG_RC</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">fi</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">}</span></p>
</div><div>i.e. allow a third parameter which is a function to call if the exec fails.</div><div><br></div><div>2. migrate (for my vdc code), add <span style="FONT-SIZE:12pt">"CleanUp" as a last parameter on the exec_and_log on the last line</span></div>
<div><p><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">ssh_exec_and_log "virsh --connect $LIBVIRT_URI migrate --live $deploy_id $QEMU_PROTOCOL://$dest_host/system" \</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">"Could not migrate $deploy_id to $dest_host" <span style="COLOR:rgb(255,0,0)">CleanUp</span></span></p></div><div>3. Then add the following function to migrate;</div>
<div><p><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">function CleanUp</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">{</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">VDC=$(dirname $0)/../../../vdc-nebula</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">if [ -d "${VDC}" ]; then</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">${VDC}/remotes/tm/vdc/postmigrate_fail ${deploy_id} ${dest_host}</span><br>
<span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">fi</span><br><span style="FONT-FAMILY:'courier new',courier,monaco,monospace,sans-serif">}</span></p><p><span style="FONT-SIZE:12pt">Cleanup could be extended for other storage options ... ??<br>
I guess ideally you would pass the driver through and CleanUp would become completely generic and postmigrate_fail<br>would become just another standard routine??</span></p><p><br></p></div><div><div></div><span></span></div>
</div></blockquote></div></div></div></div></div></div></div></div></div></div></blockquote></div></div></div></div></div></div><div class="im"><font face="Verdana,Arial,Helvetica,Trebuchet MS" size="1">
<br>--
<br>This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
<br>The full disclaimer details can be found at <a href="http://www.csir.co.za/disclaimer.html" target="_blank">http://www.csir.co.za/disclaimer.html</a>.
<p>
<br>This message has been scanned for viruses and dangerous content by <a href="http://www.mailscanner.info/" target="_blank"><b>MailScanner</b></a>,
<br>and is believed to be clean.
</p><p>
<br>Please consider the environment before printing this email.
</p><p></p></font>
</div></div>
</blockquote></div><br></div>