<html><body><div style="font-family: trebuchet ms,sans-serif; font-size: 12pt; color: #000000"><div>.. Erm, because the vmm driver is per-hypervisor .. and this product is technically<br></div><div>hypervisor agnostic .. ??</div><div><br></div><div><br></div><div><span name="x"></span><div><div><span size="2" style="font-size: small;" data-mce-style="font-size: small;">--</span></div><div><table class="mceItemTable"><tbody><tr><td><img src="http://linuxforums.org.uk/index.php?action=dlattach;attach=262;type=avatar" data-mce-src="http://linuxforums.org.uk/index.php?action=dlattach;attach=262;type=avatar" src="http://linuxforums.org.uk/index.php?action=dlattach;attach=262;type=avatar"></td><td><div><b style="font-family: verdana, Helvetica, sans-serif; font-size: 11px; line-height: 14px;" data-mce-style="font-family: verdana, Helvetica, sans-serif; font-size: 11px; line-height: 14px;">Gareth Bult</b><br><span color="#666666" style="color: #666666;" data-mce-style="color: #666666;"><span face="verdana, Helvetica, sans-serif" style="font-family: verdana, Helvetica, sans-serif;" data-mce-style="font-family: verdana, Helvetica, sans-serif;"><span style="font-size: 11px; line-height: 14px;" data-mce-style="font-size: 11px; line-height: 14px;">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span color="#666666" style="color: #666666;" data-mce-style="color: #666666;"><br></span><span color="#666666" style="color: #666666;" data-mce-style="color: #666666;"><span style="color: #666666; font-family: verdana, Helvetica, sans-serif; font-size: 11px; line-height: 14px;" data-mce-style="color: #666666; font-family: verdana, Helvetica, sans-serif; font-size: 11px; line-height: 14px;">See the status of my current project at <a data-mce-href="http://vdc-store.com" href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div><div><span color="#666666" style="color: #666666;" data-mce-style="color: #666666;"><span face="verdana, Helvetica, sans-serif" style="font-family: verdana, Helvetica, sans-serif;" data-mce-style="font-family: verdana, Helvetica, sans-serif;"><span style="font-size: 11px; line-height: 14px;" data-mce-style="font-size: 11px; line-height: 14px;"><br></span></span></span></div></td></tr></tbody></table></div></div><span name="x"></span><br></div><hr id="zwchr"><div style="color:#000;font-weight:normal;font-style:normal;text-decoration:none;font-family:Helvetica,Arial,sans-serif;font-size:12pt;"><b>From: </b>"David Macleod" <dmacleod@csir.co.za><br><b>To: </b>"Gareth Bult" <gareth@linux.co.uk>, "Jaime Melis" <jmelis@opennebula.org><br><b>Cc: </b>dev@lists.opennebula.org<br><b>Sent: </b>Monday, 20 January, 2014 11:32:33 AM<br><b>Subject: </b>Re: [one-dev] Live migration / recovery - suggestions<br><div><br></div>
<div>Hi Gareth, Jaime</div>
<div> </div>
<div>Why can't the recovery be handled by the migrate script? You can distribute your own custom vmm driver so that it doesn't get overwritten by the updates. That's how I wrote my add-on, I also have the need to recover from a migrate fail.</div>
<div> </div>
<div>Regards,</div>
<div>David</div>
<div><br>>>> Jaime Melis <jmelis@opennebula.org> 2014/01/20 12:43 PM >>><br></div>
<div dir="ltr">
<div>Hi Gareth,</div>
<div><br></div>
<div>we've been studying your proposal, and even though we agree with what you say we</div>
<div>aren't 100% convinced with this solution. The issues with the proposal are the</div>
<div>following:</div>
<div><br></div>
<div>- As long as it's possible, we'd like to keep separate the main opennebula code</div>
<div>and the addons. In this case it means that we are not very comfortable with</div>
<div>point 3: adding addon-specific code to the main repository.</div>
<div><br></div>
<div>- The proposed solution only solves the "migrate" issue, but other addons will</div>
<div>have potentially issues with other scripts, and not necessarily with the</div>
<div>"CleanUp" part of the "ssh_exec_and_log". We would like to find a more general</div>
<div>solution.</div>
<div><br></div>
<div>We are still thinking about this, we definitely want to solve this issue, so if</div>
<div>you (or anyone else) has any ideas, please let us know.</div>
<div><br></div>
<div>cheers,</div>
<div>Jaime</div></div>
<div class="gmail_extra"><br><div><br></div>
<div class="gmail_quote">On Tue, Jan 14, 2014 at 2:23 PM, Gareth Bult <span dir="ltr"><<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div style="FONT-SIZE: 12pt; FONT-FAMILY: trebuchet ms,sans-serif">
<div>Hey Guys, I've done a little work on the migration script - <span style="FONT-SIZE: 12pt">this is what I've done here ..</span></div>
<div><span style="FONT-SIZE: 12pt">- be nice if something similar could be implemented @ source .. ?</span></div>
<div><br></div>
<div>1. ssh_exec_and_log (generic change - this could be useful elsewhere..) modify as follows;</div>
<div>
<p><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">function ssh_exec_and_log</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">{</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">message=$2</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif"><span style="COLOR: rgb(255,0,0)">cleanup=$3 # ++</span></span></p>
<p><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">EXEC_LOG_ERR=`$1 2>&1 1>/dev/null`</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">EXEC_LOG_RC=$?</span></p>
<p><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">if [ $EXEC_LOG_RC -ne 0 ]; then</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">log_error "Command \"$1\" failed: $EXEC_LOG_ERR"</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif"><span style="COLOR: rgb(255,0,0)">if [ ! -z $cleanup ]; then # ++</span></span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif; COLOR: rgb(255,0,0)">$cleanup # ++</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif; COLOR: rgb(255,0,0)">fi # ++<br></span></p>
<p><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">if [ -n "$2" ]; then</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">error_message "$2"</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">else</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">error_message "Error executing $1: $EXEC_LOG_ERR"</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">fi</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">return $EXEC_LOG_RC</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">fi</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">}</span></p></div>
<div>i.e. allow a third parameter which is a function to call if the exec fails.</div>
<div><br></div>
<div>2. migrate (for my vdc code), add <span style="FONT-SIZE: 12pt">"CleanUp" as a last parameter on the exec_and_log on the last line</span></div>
<div>
<p><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">ssh_exec_and_log "virsh --connect $LIBVIRT_URI migrate --live $deploy_id $QEMU_PROTOCOL://$dest_host/system" \</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">"Could not migrate $deploy_id to $dest_host" <span style="COLOR: rgb(255,0,0)">CleanUp</span></span></p></div>
<div>3. Then add the following function to migrate;</div>
<div>
<p><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">function CleanUp</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">{</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">VDC=$(dirname $0)/../../../vdc-nebula</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">if [ -d "${VDC}" ]; then</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">${VDC}/remotes/tm/vdc/postmigrate_fail ${deploy_id} ${dest_host}</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">fi</span><br><span style="FONT-FAMILY: 'courier new',courier,monaco,monospace,sans-serif">}</span></p>
<p><span style="FONT-SIZE: 12pt">Cleanup could be extended for other storage options ... ??<br>I guess ideally you would pass the driver through and CleanUp would become completely generic and postmigrate_fail<br>would become just another standard routine??</span></p>
<p><span style="FONT-SIZE: 12pt"></span></p></div>
<div>hth</div>
<div>Gareth.</div>
<div><br></div>
<div><span></span>
<div>
<div><span style="FONT-SIZE: small">--</span></div>
<div>
<table class="mceItemTable">
<tbody>
<tr>
<td><img src="cid:GGDIPTHKGGEU.avatar"></td>
<td>
<div><b style="FONT-SIZE: 11px; FONT-FAMILY: verdana,Helvetica,sans-serif; LINE-HEIGHT: 14px">Gareth Bult</b><br><span style="COLOR: #666666"><span style="FONT-FAMILY: verdana,Helvetica,sans-serif"><span style="FONT-SIZE: 11px; LINE-HEIGHT: 14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span style="COLOR: #666666"><br></span><span style="COLOR: #666666"><span style="FONT-SIZE: 11px; FONT-FAMILY: verdana,Helvetica,sans-serif; COLOR: #666666; LINE-HEIGHT: 14px">See the status of my current project at <a href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div>
<div><span style="COLOR: #666666"><span style="FONT-FAMILY: verdana,Helvetica,sans-serif"><span style="FONT-SIZE: 11px; LINE-HEIGHT: 14px"><br></span></span></span></div></td></tr></tbody></table></div></div><span></span><br></div>
<hr>
<div style="FONT-SIZE: 12pt; TEXT-DECORATION: none; FONT-FAMILY: Helvetica,Arial,sans-serif; FONT-WEIGHT: normal; FONT-STYLE: normal"><b>From: </b>"Ruben S. Montero" <<a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a>><br><b>To: </b>"Gareth Bult" <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>><br><b>Cc: </b><a href="mailto:dev@lists.opennebula.org" target="_blank">dev@lists.opennebula.org</a><br><b>Sent: </b>Monday, 13 January, 2014 3:41:50 PM<br><b>Subject: </b>Re: [one-dev] Live migration / recovery<br>
<div><br></div>
<div dir="ltr">Totally agree, in fact we've seen this in the past.
<div><br></div>
<div>This is fairly easy to add, one_vmm_exec.rb includes a pseudo-dsl to specify the actions, each action includes a fail action. Looking at the code we have also to roll-back the networking configuration on the target host.</div>
<div><br></div>
<div>Added a new issue for this. In the meantime we can just use the previous workaround.</div>
<div><br></div>
<div><a href="http://dev.opennebula.org/issues/2633" target="_blank">http://dev.opennebula.org/issues/2633</a><br></div>
<div><br></div></div>
<div class="gmail_extra"><br>
<div><br></div>
<div class="gmail_quote">On Mon, Jan 13, 2014 at 2:59 PM, Gareth Bult <span dir="ltr"><<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div style="FONT-SIZE: 12pt; FONT-FAMILY: trebuchet ms,sans-serif">
<div>Ok, this almost seems too easy ... :)<br></div>
<div><br></div>
<div>What I was trying to avoid was more tweaking of ON files when installing VDC.</div>
<div><br></div>
<div>The issue I see is that each time someone upgraded ON, they will potentially have a chain of</div>
<div>small patches to apply ... if I supply a few small patches based on "if <vdc installed> - do something extra"</div>
<div>would you include these in the stock scripts??</div>
<div><br></div>
<div>Gareth.</div>
<div><br></div>
<div><span></span>
<div>
<div><span style="FONT-SIZE: small">--</span></div>
<div>
<table class="mceItemTable">
<tbody>
<tr>
<td><img></td>
<td>
<div><b style="FONT-SIZE: 11px; FONT-FAMILY: verdana,Helvetica,sans-serif; LINE-HEIGHT: 14px">Gareth Bult</b><br><span style="COLOR: #666666"><span style="FONT-FAMILY: verdana,Helvetica,sans-serif"><span style="FONT-SIZE: 11px; LINE-HEIGHT: 14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span style="COLOR: #666666"><br></span><span style="COLOR: #666666"><span style="FONT-SIZE: 11px; FONT-FAMILY: verdana,Helvetica,sans-serif; COLOR: #666666; LINE-HEIGHT: 14px">See the status of my current project at <a href="http://vdc-store.com" target="_blank">http://vdc-store.com</a></span></span></div>
<div><span style="COLOR: #666666"><span style="FONT-FAMILY: verdana,Helvetica,sans-serif"><span style="FONT-SIZE: 11px; LINE-HEIGHT: 14px"><br></span></span></span></div></td></tr></tbody></table></div></div><span></span><br></div>
<hr>
<div style="FONT-SIZE: 12pt; TEXT-DECORATION: none; FONT-FAMILY: Helvetica,Arial,sans-serif; FONT-WEIGHT: normal; FONT-STYLE: normal"><b>From: </b>"Ruben S. Montero" <<a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a>><br><b>To: </b>"Gareth Bult" <<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>><br><b>Cc: </b><a href="mailto:dev@lists.opennebula.org" target="_blank">dev@lists.opennebula.org</a><br><b>Sent: </b>Sunday, 12 January, 2014 10:36:30 PM<br><b>Subject: </b>Re: [one-dev] Live migration / recovery<br>
<div><br></div>
<div dir="ltr">Hi Gareth
<div><br></div>
<div>As the migrate script can be easily updated we do not provide any hook for that. I'd go to kvm/migrate, and do a simple if [ $? ... after the virsh command to kill the cache on the target host.</div>
<div><br></div>
<div>Cheers</div>
<div><br></div>
<div>Ruben </div>
<div class="gmail_extra"><br>
<div><br></div>
<div class="gmail_quote">On Thu, Dec 19, 2013 at 12:54 PM, Gareth Bult <span dir="ltr"><<a href="mailto:gareth@linux.co.uk" target="_blank">gareth@linux.co.uk</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="PADDING-LEFT: 1ex; MARGIN: 0px 0px 0px 0.8ex; BORDER-LEFT: #ccc 1px solid">
<div>
<div style="FONT-SIZE: 12pt; FONT-FAMILY: trebuchet ms,sans-serif">
<div>Hi,</div>
<div><br></div>
<div>I implemented live migration for the VDC driver a few weeks back and on the whole it seems to work</div>
<div>quite well. The "premigrate" script creates a cache instance on the target server and puts the source</div>
<div>cache into proxy mode, then the "postmigrate" script kills the original cache instance.</div>
<div><br></div>
<div>Problem :: if the migration fails, I'm left with a running cache on both the source and target servers, with</div>
<div>the source cache in proxy mode. I have a 1-line CLI command to revert the issue, but I need to hook into</div>
<div>the system in order to call it. </div>
<div><br></div>
<div><span style="FONT-SIZE: 12pt">How to do I </span><span style="FONT-SIZE: 12pt">do this, I guess effectively I need something like "postmigrate_fail" .. ???</span></div>
<div><br></div>
<div>tia</div>
<div>Gareth.</div>
<div><br></div>
<div><span></span>
<div>
<div><span style="FONT-SIZE: small">--</span></div>
<div><span><span style="COLOR: #888888"></span></span>
<table class="mceItemTable">
<tbody>
<tr>
<td><img></td>
<td><b style="FONT-SIZE: 11px; FONT-FAMILY: verdana,Helvetica,sans-serif; LINE-HEIGHT: 14px">Gareth Bult</b><br><span style="COLOR: #666666"><span style="FONT-FAMILY: verdana,Helvetica,sans-serif"><span style="FONT-SIZE: 11px; LINE-HEIGHT: 14px">“The odds of hitting your target go up dramatically when you aim at it.”</span></span></span><span style="COLOR: #666666"><br></span><span style="COLOR: #666666"><br></span><span style="COLOR: #666666"><br></span></td></tr></tbody></table></div></div><span></span><br></div></div></div><br>_______________________________________________<br>Dev mailing list<br><a href="mailto:Dev@lists.opennebula.org" target="_blank">Dev@lists.opennebula.org</a><br><a href="http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org<span><span style="COLOR: #888888"><br clear="all">
<div><br></div>-- <br></span></span></a><span><span style="COLOR: #888888">
<div dir="ltr"><a href="http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org" target="_blank">
<div>
<div>-- <br></div></div>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect</a>
<div><a href="http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org" target="_blank">OpenNebula - Flexible Enterprise Cloud Made Simple<br></a><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula</div></div></span></span></blockquote></div></div></div></div>
<div><br></div></div></div></blockquote></div><span class="HOEnZb"><span color="#888888" data-mce-style="color: #888888;" style="color: #888888;"><br><br clear="all">
<div><br></div>-- <br>
<div dir="ltr">
<div>
<div>-- <br></div></div>Ruben S. Montero, PhD<br>Project co-Lead and Chief Architect
<div>OpenNebula - Flexible Enterprise Cloud Made Simple<br><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:rsmontero@opennebula.org" target="_blank">rsmontero@opennebula.org</a> | @OpenNebula</div></div></span></span></div></div>
<div><br></div></div></div><br>_______________________________________________<br>Dev mailing list<br><a href="mailto:Dev@lists.opennebula.org" target="_blank">Dev@lists.opennebula.org</a><br><a href="http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/dev-opennebula.org</a><br><div><br></div></blockquote></div><br><br clear="all">
<div><br></div>-- <br>
<div dir="ltr">
<div>Jaime Melis<br>Project Engineer<br>OpenNebula - Flexible Enterprise Cloud Made Simple<br><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:jmelis@opennebula.org" target="_blank">jmelis@opennebula.org</a></div></div></div><span size="1" face="Verdana,Arial,Helvetica,Trebuchet MS" data-mce-style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;" style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;"><br>-- <br>This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard. <br>The full disclaimer details can be found at <a href="http://www.csir.co.za/disclaimer.html" target="_blank">http://www.csir.co.za/disclaimer.html</a>.
<p><br>This message has been scanned for viruses and dangerous content by <a href="http://www.mailscanner.info/" target="_blank"><b>MailScanner</b></a>, <br>and is believed to be clean.
</p></span><p><span size="1" face="Verdana,Arial,Helvetica,Trebuchet MS" data-mce-style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;" style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;"><br>Please consider the environment before printing this email. </span></p><span face="Verdana,Arial,Helvetica,Trebuchet MS" size="1" data-mce-style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;" style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;">
<br>--
<br>This message is subject to the CSIR's copyright terms and conditions, e-mail legal notice, and implemented Open Document Format (ODF) standard.
<br>The full disclaimer details can be found at <a href="http://www.csir.co.za/disclaimer.html" target="_blank">http://www.csir.co.za/disclaimer.html</a>.
<p>
<br>This message has been scanned for viruses and dangerous content by <a href="http://www.mailscanner.info/" target="_blank"><b>MailScanner</b></a>,
<br>and is believed to be clean.
</p></span><p><span face="Verdana,Arial,Helvetica,Trebuchet MS" size="1" data-mce-style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;" style="font-family: Verdana, Arial, Helvetica, 'Trebuchet MS'; font-size: xx-small;">
<br>Please consider the environment before printing this email.
</span>
</p></div><div><br></div></div></body></html>