[one-users] KVM drivers fail to load? [MAD did not answer INIT command]

[biro lehel]

Dear all,

In my ONE system, sometimes, after the front-end's startup, I've noticed that the KVM drivers (vmm_kvm and im_kvm) are not able to load. At first I didn't understand what could be causing such ONE malfunction, but soon I tracked back all related errors and found the answer in the very initial part of oned.log, which I copied at the end of this mail.

It seems to be a strange issue, since it apparently happens rather randomly, sometimes a reboot of the front-end makes the issue disappear, but sometimes the error just sticks.

I'm aware that it might possibly not be an ONE issue. However, I would be extremely grateful if you could provide some hints or point-outs to where should I at least be looking for the cause of this problem. Is this a front-end issue? Is this an issue with the respective host? Both have KVM installed and I've performed virtualizations on both of them (with virsh+qemu) since my ONE infrastructure is up. Are there any logs which I could check to possibly find more detail?

Here is the initial part of oned.log, highlighted with the errors:

Wed Apr 18 16:24:58 2012 [ONE][I]: Starting OpenNebula 3.2.0
----------------------------------------
     OpenNebula Configuration File      
----------------------------------------
AUTH_MAD=ARGUMENTS=--authn ssh,x509,ldap,server_cipher,server_x509,EXECUTABLE=one_auth_mad
DB=BACKEND=sqlite
DEBUG_LEVEL=3
DEFAULT_DEVICE_PREFIX=hd
DEFAULT_IMAGE_TYPE=OS
ENABLE_OTHER_PERMISSIONS=YES
HM_MAD=EXECUTABLE=one_hm
HOST_MONITORING_INTERVAL=600
HOST_PER_INTERVAL=15
IMAGE_MAD=ARGUMENTS=fs -t 15,EXECUTABLE=one_image
IM_MAD=ARGUMENTS=-r 0 -t 15 kvm,EXECUTABLE=one_im_ssh,NAME=im_kvm
MAC_PREFIX=02:00
MANAGER_TIMER=15
NETWORK_SIZE=254
PORT=2633
SCRIPTS_REMOTE_DIR=/var/tmp/one
SESSION_EXPIRATION_TIME=900
TM_MAD=ARGUMENTS=tm_shared/tm_shared.conf,EXECUTABLE=one_tm,NAME=tm_shared
VM_DIR=/var/lib/one/
VM_MAD=ARGUMENTS=-t 15 -r 0 kvm,DEFAULT=vmm_exec/vmm_exec_kvm.conf,EXECUTABLE=one_vmm_exec,NAME=vmm_kvm,TYPE=kvm
VM_PER_INTERVAL=5
VM_POLLING_INTERVAL=600
VNC_BASE_PORT=5900
----------------------------------------
Wed Apr 18 16:24:58 2012 [ONE][I]: Log level:3 [0=ERROR,1=WARNING,2=INFO,3=DEBUG]
Wed Apr 18 16:24:58 2012 [ONE][I]: Checking database version.
Wed Apr 18 16:24:58 2012 [VMM][I]: Starting Virtual Machine Manager...
Wed Apr 18 16:24:58 2012 [VMM][I]: Virtual Machine Manager started.
Wed Apr 18 16:24:58 2012 [LCM][I]: Starting Life-cycle Manager...
Wed Apr 18 16:24:58 2012 [LCM][I]: Life-cycle Manager started.
Wed Apr 18 16:24:58 2012 [InM][I]: Starting Information Manager...
Wed Apr 18 16:24:58 2012 [InM][I]: Information Manager started.
Wed Apr 18 16:24:58 2012 [TrM][I]: Starting Transfer Manager...
Wed Apr 18 16:24:58 2012 [TrM][I]: Transfer Manager started.
Wed Apr 18 16:24:58 2012 [DiM][I]: Starting Dispatch Manager...
Wed Apr 18 16:24:58 2012 [DiM][I]: Dispatch Manager started.
Wed Apr 18 16:24:58 2012 [HKM][I]: Starting Hook Manager...
Wed Apr 18 16:24:58 2012 [HKM][I]: Hook Manager started.
Wed Apr 18 16:24:58 2012 [AuM][I]: Starting Auth Manager...
Wed Apr 18 16:24:58 2012 [AuM][I]: Authorization Manager started.
Wed Apr 18 16:24:58 2012 [ImM][I]: Starting Image Manager...
Wed Apr 18 16:24:58 2012 [ImM][I]: Image Manager started.
Wed Apr 18 16:24:58 2012 [ReM][I]: Starting Request Manager...
Wed Apr 18 16:24:58 2012 [ReM][I]: Request Manager started.
Wed Apr 18 16:24:58 2012 [ReM][I]: Starting XML-RPC server, port 2633 ...
Wed Apr 18 16:25:00 2012 [VMM][I]: Loading Virtual Machine Manager drivers.
Wed Apr 18 16:25:00 2012 [VMM][I]:     Loading driver: vmm_kvm (KVM)
Wed Apr 18 16:25:05 2012 [MAD][E]: MAD did not answer INIT command
Wed Apr 18 16:25:05 2012 [InM][I]: Loading Information Manager drivers.
Wed Apr 18 16:25:05 2012 [InM][I]:     Loading driver: im_kvm
Wed Apr 18 16:25:10 2012 [MAD][E]: MAD did not answer INIT command
Wed Apr 18 16:25:10 2012 [TM][I]: Loading Transfer Manager drivers.
Wed Apr 18 16:25:11 2012 [VMM][I]:     Loading driver: tm_shared
Wed Apr 18 16:25:12 2012 [TM][I]:     Driver tm_shared loaded.
Wed Apr 18 16:25:12 2012 [HKM][I]: Loading Hook Manager driver.
Wed Apr 18 16:25:13 2012 [HKM][I]:     Hook Manager loaded
Wed Apr 18 16:25:13 2012 [ImM][I]: Loading Image Manager driver.
Wed Apr 18 16:25:13 2012 [InM][I]: Monitoring host pcls (6)
Wed Apr 18 16:25:13 2012 [InM][E]: Could not find information driver im_kvm


Sometimes, only vmm_kvm fails to load, while im_kvm loads successfully, which is another twist to the story.

Any suggestion would be highly appreciated.

Thank you,
Lehel.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120419/02413880/attachment-0002.htm>