[one-dev] Live migration / recovery - suggestions

Mon Jan 20 04:50:14 PST 2014

   1. OK, so you are quite defensive and rude about this for some reason.
   2. The OpenNebula project has just said they don't want to change their
   code base to suit you, so your four line solution is off the table.
   3. You don't need to write the vmm drivers from scratch. Just copy the
   three provided with ONE call them kvm-vdc, xen-vdc and vmware-vdc or
   something, then make your customization to them. I know it's not as elegant
   as the four line solution but point 2.

On Mon, Jan 20, 2014 at 2:23 PM, Gareth Bult <gareth at linux.co.uk> wrote:

> Once I switch to a different vmm (say "vdc" rather than "kvm") , how will
> the system know to fall back on
> the hypervisor in use for all the other scripts that I don't want to
> modify?
>
> --
> *Gareth Bult*
> “The odds of hitting your target go up dramatically when you aim at it.”
> See the status of my current project at http://vdc-store.com
>
>
> ------------------------------
> *From: *"David Macleod" <dmacleod at csir.co.za>
> *To: *"Gareth Bult" <gareth at linux.co.uk>
> *Cc: *"Jaime Melis" <jmelis at opennebula.org>, dev at lists.opennebula.org
> *Sent: *Monday, 20 January, 2014 11:57:31 AM
>
> *Subject: *Re: [one-dev] Live migration / recovery - suggestions
>
> No, you should write a basic framework around the migrate script that
> supports the three hypervisors that OpenNebula uses.
>
> >>> Gareth Bult <gareth at linux.co.uk> 2014/01/20 02:15 PM >>>
>
> Erm, yeah, Ok, should I include filters for the coffee machine too ?!
>
> --
> *Gareth Bult*
> “The odds of hitting your target go up dramatically when you aim at it.”
> See the status of my current project at http://vdc-store.com
>
>
> ------------------------------
> *From: *"David Macleod" <dmacleod at csir.co.za>
> *To: *"Gareth Bult" <gareth at linux.co.uk>
> *Cc: *"Jaime Melis" <jmelis at opennebula.org>, dev at lists.opennebula.org
> *Sent: *Monday, 20 January, 2014 11:50:26 AM
> *Subject: *Re: [one-dev] Live migration / recovery - suggestions
>
> Then distribute a vmm driver per hypervisor...
>
> >>> Gareth Bult <gareth at linux.co.uk> 2014/01/20 02:01 PM >>>
> .. Erm, because the vmm driver is per-hypervisor .. and this product is
> technically
> hypervisor agnostic .. ??
>
>
> --
> *Gareth Bult*
> “The odds of hitting your target go up dramatically when you aim at it.”
> See the status of my current project at http://vdc-store.com
>
>
> ------------------------------
> *From: *"David Macleod" <dmacleod at csir.co.za>
> *To: *"Gareth Bult" <gareth at linux.co.uk>, "Jaime Melis" <
> jmelis at opennebula.org>
> *Cc: *dev at lists.opennebula.org
> *Sent: *Monday, 20 January, 2014 11:32:33 AM
> *Subject: *Re: [one-dev] Live migration / recovery - suggestions
>
> Hi Gareth, Jaime
>
> Why can't the recovery be handled by the migrate script? You can
> distribute your own custom vmm driver so that it doesn't get overwritten by
> the updates. That's how I wrote my add-on, I also have the need to recover
> from a migrate fail.
>
> Regards,
> David
>
> >>> Jaime Melis <jmelis at opennebula.org> 2014/01/20 12:43 PM >>>
> Hi Gareth,
>
> we've been studying your proposal, and even though we agree with what you
> say we
> aren't 100% convinced with this solution. The issues with the proposal are
> the
> following:
>
> - As long as it's possible, we'd like to keep separate the main opennebula
> code
> and the addons. In this case it means that we are not very comfortable with
> point 3: adding addon-specific code to the main repository.
>
> - The proposed solution only solves the "migrate" issue, but other addons
> will
> have potentially issues with other scripts, and not necessarily with the
> "CleanUp" part of the "ssh_exec_and_log". We would like to find a more
> general
> solution.
>
> We are still thinking about this, we definitely want to solve this issue,
> so if
> you (or anyone else) has any ideas, please let us know.
>
> cheers,
> Jaime
>
>
> On Tue, Jan 14, 2014 at 2:23 PM, Gareth Bult <gareth at linux.co.uk> wrote:
>
>> Hey Guys, I've done a little work on the migration script - this is what
>> I've done here ..
>> - be nice if something similar could be implemented @ source .. ?
>>
>> 1. ssh_exec_and_log (generic change - this could be useful elsewhere..)
>> modify as follows;
>>
>> function ssh_exec_and_log
>> {
>> message=$2
>> cleanup=$3 # ++
>>
>> EXEC_LOG_ERR=`$1 2>&1 1>/dev/null`
>> EXEC_LOG_RC=$?
>>
>> if [ $EXEC_LOG_RC -ne 0 ]; then
>> log_error "Command \"$1\" failed: $EXEC_LOG_ERR"
>> if [ ! -z $cleanup ]; then # ++
>> $cleanup # ++
>> fi # ++
>>
>> if [ -n "$2" ]; then
>> error_message "$2"
>> else
>> error_message "Error executing $1: $EXEC_LOG_ERR"
>> fi
>> return $EXEC_LOG_RC
>> fi
>> }
>> i.e. allow a third parameter which is a function to call if the exec
>> fails.
>>
>> 2. migrate (for my vdc code), add "CleanUp" as a last parameter on the
>> exec_and_log on the last line
>>
>> ssh_exec_and_log "virsh --connect $LIBVIRT_URI migrate --live $deploy_id
>> $QEMU_PROTOCOL://$dest_host/system" \
>> "Could not migrate $deploy_id to $dest_host" CleanUp
>> 3. Then add the following function to migrate;
>>
>> function CleanUp
>> {
>> VDC=$(dirname $0)/../../../vdc-nebula
>> if [ -d "${VDC}" ]; then
>> ${VDC}/remotes/tm/vdc/postmigrate_fail ${deploy_id} ${dest_host}
>> fi
>> }
>>
>> Cleanup could be extended for other storage options ... ??
>> I guess ideally you would pass the driver through and CleanUp would
>> become completely generic and postmigrate_fail
>> would become just another standard routine??
>>
>>
>> hth
>> Gareth.
>>
>> --
>> *Gareth Bult*
>> “The odds of hitting your target go up dramatically when you aim at it.”
>> See the status of my current project at http://vdc-store.com
>>
>>
>> ------------------------------
>> *From: *"Ruben S. Montero" <rsmontero at opennebula.org>
>> *To: *"Gareth Bult" <gareth at linux.co.uk>
>> *Cc: *dev at lists.opennebula.org
>> *Sent: *Monday, 13 January, 2014 3:41:50 PM
>> *Subject: *Re: [one-dev] Live migration / recovery
>>
>> Totally agree, in fact we've seen this in the past.
>>
>> This is fairly easy to add, one_vmm_exec.rb includes a pseudo-dsl to
>> specify the actions, each action includes a fail action. Looking at the
>> code we have also to roll-back the networking configuration on the target
>> host.
>>
>> Added a new issue for this. In the meantime we can just use the previous
>> workaround.
>>
>> http://dev.opennebula.org/issues/2633
>>
>>
>>
>> On Mon, Jan 13, 2014 at 2:59 PM, Gareth Bult <gareth at linux.co.uk> wrote:
>>
>>> Ok, this almost seems too easy ... :)
>>>
>>> What I was trying to avoid was more tweaking of ON files when installing
>>> VDC.
>>>
>>> The issue I see is that each time someone upgraded ON, they will
>>> potentially have a chain of
>>> small patches to apply ... if I supply a few small patches based on "if
>>> <vdc installed> - do something extra"
>>> would you include these in the stock scripts??
>>>
>>> Gareth.
>>>
>>> --
>>> *Gareth Bult*
>>> “The odds of hitting your target go up dramatically when you aim at it.”
>>> See the status of my current project at http://vdc-store.com
>>>
>>>
>>> ------------------------------
>>> *From: *"Ruben S. Montero" <rsmontero at opennebula.org>
>>> *To: *"Gareth Bult" <gareth at linux.co.uk>
>>> *Cc: *dev at lists.opennebula.org
>>> *Sent: *Sunday, 12 January, 2014 10:36:30 PM
>>> *Subject: *Re: [one-dev] Live migration / recovery
>>>
>>> Hi Gareth
>>>
>>> As the migrate script can be easily updated we do not provide any hook
>>> for that. I'd go to kvm/migrate, and do a simple if [ $? ... after the
>>> virsh command to kill the cache on the target host.
>>>
>>> Cheers
>>>
>>> Ruben
>>>
>>>
>>> On Thu, Dec 19, 2013 at 12:54 PM, Gareth Bult <gareth at linux.co.uk>wrote:
>>>
>>>> Hi,
>>>>
>>>> I implemented live migration for the VDC driver a few weeks back and on
>>>> the whole it seems to work
>>>> quite well. The "premigrate" script creates a cache instance on the
>>>> target server and puts the source
>>>> cache into proxy mode, then the "postmigrate" script kills the original
>>>> cache instance.
>>>>
>>>> Problem :: if the migration fails, I'm left with a running cache on
>>>> both the source and target servers, with
>>>> the source cache in proxy mode. I have a 1-line CLI command to revert
>>>> the issue, but I need to hook into
>>>> the system in order to call it.
>>>>
>>>> How to do I do this, I guess effectively I need something like
>>>> "postmigrate_fail" .. ???
>>>>
>>>> tia
>>>> Gareth.
>>>>
>>>
> --
> This message is subject to the CSIR's copyright terms and conditions,
> e-mail legal notice, and implemented Open Document Format (ODF) standard.
> The full disclaimer details can be found at
> http://www.csir.co.za/disclaimer.html.
>
>
> This message has been scanned for viruses and dangerous content by
> *MailScanner* <http://www.mailscanner.info/>,
> and is believed to be clean.
>
>
> Please consider the environment before printing this email.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/dev-opennebula.org/attachments/20140120/63c2e542/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 5338 bytes
Desc: not available
URL: <http://lists.opennebula.org/pipermail/dev-opennebula.org/attachments/20140120/63c2e542/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 5338 bytes
Desc: not available
URL: <http://lists.opennebula.org/pipermail/dev-opennebula.org/attachments/20140120/63c2e542/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 5435 bytes
Desc: not available
URL: <http://lists.opennebula.org/pipermail/dev-opennebula.org/attachments/20140120/63c2e542/attachment-0005.png>