<b>Here is the vm.log for 76:</b><br>Wed Sep 30 08:57:50 2009 [VMM][D]: Monitor Information:<br> CPU : -1<br> Memory: 1048576<br> Net_TX: -1<br> Net_RX: -1<br>Wed Sep 30 08:57:56 2009 [DiM][I]: New VM state is DONE<br>
Wed Sep 30 08:57:56 2009 [VMM][W]: Ignored: LOG - 76 Driver command for 76 cancelled<br><br>Wed Sep 30 08:57:56 2009 [VMM][W]: Ignored: CANCEL SUCCESS 76 -<br><br>Wed Sep 30 08:58:00 2009 [TM][W]: Ignored: LOG - 76 tm_delete.sh: Deleting /srv/cloud/one/var/76/images<br>
<br>Wed Sep 30 08:58:00 2009 [TM][W]: Ignored: LOG - 76 tm_delete.sh: Executed "rm -rf /srv/cloud/one/var/76/images".<br><br>Wed Sep 30 08:58:00 2009 [TM][W]: Ignored: TRANSFER SUCCESS 76 -<br><br><b>And vm.log for 77:</b><br>
Wed Sep 30 08:57:50 2009 [VMM][D]: Monitor Information:<br> CPU : -1<br> Memory: 1048576<br> Net_TX: -1<br> Net_RX: -1<br>Wed Sep 30 08:58:50 2009 [VMM][I]: Command execution fail: virsh dominfo one-77<br>
Wed Sep 30 08:58:50 2009 [VMM][I]: STDERR follows.<br>Wed Sep 30 08:58:50 2009 [VMM][I]: Connecting to uri: qemu:///system<br>Wed Sep 30 08:58:50 2009 [VMM][I]: error: failed to get domain 'one-77'<br>Wed Sep 30 08:58:50 2009 [VMM][I]: error: Domain not found<br>
Wed Sep 30 08:58:50 2009 [VMM][I]: ExitCode: 1<br>Wed Sep 30 08:58:50 2009 [VMM][I]: VM running but it was not found. Restart and delete actions available or try to recover it manually<br>Wed Sep 30 08:58:50 2009 [LCM][I]: New VM state is UNKNOWN<br>
Wed Sep 30 08:59:50 2009 [VMM][I]: Command execution fail: virsh dominfo one-77<br>Wed Sep 30 08:59:50 2009 [VMM][I]: STDERR follows.<br>Wed Sep 30 08:59:50 2009 [VMM][I]: Connecting to uri: qemu:///system<br>Wed Sep 30 08:59:50 2009 [VMM][I]: error: failed to get domain 'one-77'<br>
Wed Sep 30 08:59:50 2009 [VMM][I]: error: Domain not found<br>Wed Sep 30 08:59:50 2009 [VMM][I]: ExitCode: 1<br>Wed Sep 30 08:59:50 2009 [VMM][I]: VM running but it was not found. Restart and delete actions available or try to recover it manually<br>
<br><div><b>And oned.log:</b></div><div><div>Wed Sep 30 08:57:55 2009 [ReM][D]: VirtualMachineAction invoked</div><div>Wed Sep 30 08:57:55 2009 [DiM][D]: Finalizing VM 76</div><div>Wed Sep 30 08:57:56 2009 [VMM][D]: Message received: LOG - 76 Driver command for 76 cancelled</div>
<div><br></div><div>Wed Sep 30 08:57:56 2009 [VMM][D]: Message received: CANCEL SUCCESS 76 -</div><div><br></div><div>Wed Sep 30 08:57:58 2009 [ReM][D]: VirtualMachinePoolInfo method invoked</div><div>Wed Sep 30 08:58:00 2009 [TM][D]: Message received: LOG - 76 tm_delete.sh: Deleting /srv/cloud/one/var/76/images</div>
<div><br></div><div>Wed Sep 30 08:58:00 2009 [TM][D]: Message received: LOG - 76 tm_delete.sh: Executed "rm -rf /srv/cloud/one/var/76/images".</div><div><br></div><div>Wed Sep 30 08:58:00 2009 [TM][D]: Message received: TRANSFER SUCCESS 76 -</div>
<div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][I]: Monitoring VM 72.</div><div>Wed Sep 30 08:58:50 2009 [VMM][I]: Monitoring VM 73.</div><div>Wed Sep 30 08:58:50 2009 [VMM][I]: Monitoring VM 77.</div><div>Wed Sep 30 08:58:50 2009 [VMM][I]: Monitoring VM 78.</div>
<div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: POLL SUCCESS 72 STATE=a USEDMEMORY=1048576</div><div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: LOG - 77 Command execution fail: virsh dominfo one-77</div>
<div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: LOG - 77 STDERR follows.</div><div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: LOG - 77 Connecting to uri: qemu:///system</div>
<div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: LOG - 77 error: failed to get domain 'one-77'</div><div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: LOG - 77 error: Domain not found</div>
<div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: LOG - 77 ExitCode: 1</div><div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: POLL SUCCESS 77 STATE=d</div><div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: POLL SUCCESS 73 STATE=a USEDMEMORY=1048576</div>
<div><br></div><div>Wed Sep 30 08:58:50 2009 [VMM][D]: Message received: POLL SUCCESS 78 STATE=a USEDMEMORY=1048576</div><div><br></div><div>Wed Sep 30 08:59:02 2009 [InM][I]: Monitoring host node1 (4)</div><div>Wed Sep 30 08:59:06 2009 [InM][D]: Host 4 successfully monitored.</div>
<div><br></div><div><b>In my opinion the above log files don't have much more information than VM 77 is no longer available.</b></div><div><b><br></b></div><div><b>Shi</b></div></div><br>On Wed, Sep 30, 2009 at 10:02 AM, Ruben S. Montero <<a href="mailto:rubensm@dacya.ucm.es">rubensm@dacya.ucm.es</a>> wrote:<br>
> Hi,<br>><br>> Does the 77 vm.log say something relevant? Can you send us also the<br>> vm.log file for 76? OpenNebula should never try to delete VM 77...<br>><br>> Thanks<br>><br>> Ruben<br>><br>
> On Wed, Sep 30, 2009 at 5:28 PM, Shi Jin <<a href="mailto:jinzishuai@gmail.com">jinzishuai@gmail.com</a>> wrote:<br>>> Hi there,<br>>><br>>> I am running one-1.4beta with KVM hypervisor.<br>>> Recently I found a very bizarre behavior of "onevm delete".<br>
>> For example, initially there are 5 VMs running:72,73,76,77,78.<br>>> Then I did "onevm delete 76" and found only 3 VMs are left: the VM 77<br>>> is also deleted.<br>>> "virsh list" on the host node shows VM 77 is gone and "onevm list"<br>
>> shows its status to be unknown.<br>>> I think this is a serious bug and requires a quick fix.<br>>> I tried to dig into the OpenNebula log files but found nothing other<br>>> than " Command execution fail: virsh dominfo one-77" after the "onevm<br>
>> delete 76" command.<br>>><br>>> On the host node, the syslog shows something interesting:<br>>> Sep 30 08:58:59 node1 kernel: [238101.304619] br1: port 5(vnet4)<br>>> entering disabled state<br>
>> Sep 30 08:58:59 node1 kernel: [238101.342851] device vnet4 left promiscuous mode<br>>> Sep 30 08:58:59 node1 kernel: [238101.342854] br1: port 5(vnet4)<br>>> entering disabled state<br>>> Sep 30 08:58:59 node1 kernel: [238101.424851] br1: port 4(vnet3)<br>
>> entering disabled state<br>>> Sep 30 08:59:00 node1 kernel: [238101.463031] device vnet3 left promiscuous mode<br>>> Sep 30 08:59:00 node1 kernel: [238101.463035] br1: port 4(vnet3)<br>>> entering disabled state<br>
>> Sep 30 08:59:53 node1 libvirtd: 08:59:53.480: error : Domain not found<br>>><br>>> Please note that the VM 76,77,78 all use the same vnet (bridged on<br>>> br1) and their corresponding ports are vnet3,vnet4 and vnet5. I am<br>
>> not exactly sure what the above message implies but I have a feeling<br>>> that something went wrong about the network. In this sense, this<br>>> problem may not be a OpenNebula bug, possibly a KVM bug too.<br>
>><br>>> Actually this kind of problem has happened many times for us. This is<br>>> my first time to carefully record what has happened.<br>>> I am wondering if anyone has encountered this kind of problem before<br>
>> and is there a fix?<br>>><br>>> Thanks a lot.<br>>><br>>> Shi<br>>> --<br>>> Shi Jin, Ph.D.<br>>> _______________________________________________<br>>> Users mailing list<br>
>> <a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>>> <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
>><br>><br>><br>><br>> --<br>> +---------------------------------------------------------------+<br>> Dr. Ruben Santiago Montero<br>> Associate Professor<br>> Distributed System Architecture Group (<a href="http://dsa-research.org">http://dsa-research.org</a>)<br>
><br>> URL: <a href="http://dsa-research.org/doku.php?id=people:ruben">http://dsa-research.org/doku.php?id=people:ruben</a><br>> Weblog: <a href="http://blog.dsa-research.org/?author=7">http://blog.dsa-research.org/?author=7</a><br>
><br>> GridWay, <a href="http://www.gridway.org">http://www.gridway.org</a><br>> OpenNebula, <a href="http://www.opennebula.org">http://www.opennebula.org</a><br>> +---------------------------------------------------------------+<br>
><br><br><br><br>-- <br>Shi Jin, Ph.D.<br><br><br></div>