[one-users] Nebula 4.0.1 Xen 4.1.4 Debian Wheezy - MIGRATION problem..

[Jacek Jarosiewicz]

Hi,

I have a problem with migrating VMs between hosts. Both cold and live 
migration.

Cold migration log is:
Wed Jun 12 12:32:24 2013 [LCM][I]: New VM state is RUNNING
Wed Jun 12 12:32:41 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 12:39:56 2013 [LCM][I]: New VM state is SAVE_MIGRATE
Wed Jun 12 12:40:40 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 12:40:40 2013 [VMM][I]: Successfully execute virtualization 
driver operation: save.
Wed Jun 12 12:40:40 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 12:40:40 2013 [VMM][I]: Successfully execute network driver 
operation: clean.
Wed Jun 12 12:40:40 2013 [LCM][I]: New VM state is PROLOG_MIGRATE
Wed Jun 12 12:40:40 2013 [TM][I]: ExitCode: 0
Wed Jun 12 12:41:18 2013 [LCM][E]: monitor_done_action, VM in a wrong state
Wed Jun 12 12:46:29 2013 [LCM][E]: monitor_done_action, VM in a wrong state
Wed Jun 12 12:51:40 2013 [LCM][E]: monitor_done_action, VM in a wrong state
Wed Jun 12 12:56:09 2013 [TM][I]: mv: Moving 
nebula1:/var/lib/one/datastores/0/29 to nebula0:/var/lib/one/datastores/0/29
Wed Jun 12 12:56:09 2013 [TM][I]: ExitCode: 0
Wed Jun 12 12:56:09 2013 [LCM][I]: New VM state is BOOT
Wed Jun 12 12:56:09 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 12:56:09 2013 [VMM][I]: Successfully execute network driver 
operation: pre.
Wed Jun 12 12:56:32 2013 [VMM][I]: Command execution fail: 
/var/tmp/one/vmm/xen4/restore /var/lib/one//datastores/0/29/checkpoint 
nebula0 29 nebula0
Wed Jun 12 12:56:32 2013 [VMM][E]: restore: Command "sudo /usr/sbin/xm 
restore /var/lib/one//datastores/0/29/checkpoint" failed: Error: 
/usr/lib/xen-4.1/bin/xc_restore 23 12 1 2 0 0 0 0 failed
Wed Jun 12 12:56:32 2013 [VMM][E]: Could not restore from 
/var/lib/one//datastores/0/29/checkpoint
Wed Jun 12 12:56:32 2013 [VMM][I]: ExitCode: 1
Wed Jun 12 12:56:32 2013 [VMM][I]: Failed to execute virtualization 
driver operation: restore.
Wed Jun 12 12:56:32 2013 [VMM][E]: Error restoring VM: Could not restore 
from /var/lib/one//datastores/0/29/checkpoint
Wed Jun 12 12:56:33 2013 [DiM][I]: New VM state is FAILED

and in xend.log i see:

[2013-06-12 12:56:32 24698] ERROR (XendCheckpoint:357) 
/usr/lib/xen-4.1/bin/xc_restore 23 12 1 2 0 0 0 0 failed
Traceback (most recent call last):
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendCheckpoint.py", 
line 309, in restore
     forkHelper(cmd, fd, handler.handler, True)
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendCheckpoint.py", 
line 411, in forkHelper
     raise XendError("%s failed" % string.join(cmd))
XendError: /usr/lib/xen-4.1/bin/xc_restore 23 12 1 2 0 0 0 0 failed
[2013-06-12 12:56:32 24698] ERROR (XendDomain:1194) Restore failed
Traceback (most recent call last):
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", 
line 1178, in domain_restore_fd
     dominfo = XendCheckpoint.restore(self, fd, paused=paused, 
relocating=relocating)
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendCheckpoint.py", 
line 358, in restore
     raise exn
XendError: /usr/lib/xen-4.1/bin/xc_restore 23 12 1 2 0 0 0 0 failed


..and with live migration i see:

Wed Jun 12 12:27:16 2013 [LCM][I]: New VM state is RUNNING
Wed Jun 12 12:27:32 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:34:26 2013 [LCM][I]: New VM state is MIGRATE
Wed Jun 12 13:34:26 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:34:26 2013 [VMM][I]: Successfully execute transfer manager 
driver operation: tm_premigrate.
Wed Jun 12 13:34:26 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:34:26 2013 [VMM][I]: Successfully execute network driver 
operation: pre.
Wed Jun 12 13:37:34 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:37:34 2013 [VMM][I]: Successfully execute virtualization 
driver operation: migrate.
Wed Jun 12 13:37:34 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:37:34 2013 [VMM][I]: Successfully execute network driver 
operation: clean.
Wed Jun 12 13:37:34 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:37:34 2013 [VMM][I]: Successfully execute network driver 
operation: post.
Wed Jun 12 13:37:34 2013 [VMM][I]: ExitCode: 0
Wed Jun 12 13:37:34 2013 [VMM][I]: Successfully execute transfer manager 
driver operation: tm_postmigrate.
Wed Jun 12 13:37:35 2013 [LCM][I]: New VM state is RUNNING

and in xend.log:

[2013-06-12 13:37:39 9651] ERROR (XendCheckpoint:357) Device 51712 (vbd) 
could not be connected. /var/lib/one//datastores/0/28/disk.0 does not exist.
Traceback (most recent call last):
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendCheckpoint.py", 
line 346, in restore
     dominfo.waitForDevices() # Wait for backends to set up
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomainInfo.py", 
line 1237, in waitForDevices
     self.getDeviceController(devclass).waitForDevices()
   File 
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/DevController.py", 
line 140, in waitForDevices
     return map(self.waitForDevice, self.deviceIDs())
   File 
"/usr/lib/xen-4.1/bin/../lib/python/xen/xend/server/DevController.py", 
line 165, in waitForDevice
     "%s" % (devid, self.deviceClass, err))
VmError: Device 51712 (vbd) could not be connected. 
/var/lib/one//datastores/0/28/disk.0 does not exist.
[2013-06-12 13:37:39 9651] ERROR (XendDomain:1194) Restore failed
Traceback (most recent call last):
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendDomain.py", 
line 1178, in domain_restore_fd
     dominfo = XendCheckpoint.restore(self, fd, paused=paused, 
relocating=relocating)
   File "/usr/lib/xen-4.1/bin/../lib/python/xen/xend/XendCheckpoint.py", 
line 358, in restore
     raise exn
VmError: Device 51712 (vbd) could not be connected. 
/var/lib/one//datastores/0/28/disk.0 does not exist.

any help would be appreciated..

Cheers,
J

-- 
Jacek Jarosiewicz