The deletion code worked like this:<br><br><div>* Get the list of VMs</div><div>* Search the list of a VM with an specific UUID</div><div><br></div><div>When several VMs are being deleted it could happen that one VM marked for deletion is in the list of all VMs but by the time it checks its UUID it is already deleted. There is another patch for that problem [1] that you can apply to fix it.</div><div><br></div><div>Still, we have made some other changes to deal with VMs and templates in folders so I think it is better to use the version 4.10.1 that comes with all these fixes.</div><div><br></div><div>Cheers</div><div><br></div><div>[1] <a href="http://dev.opennebula.org/projects/opennebula/repository/revisions/e2886255464e666c6a62d91cc1d016da6ab1a643/diff/">http://dev.opennebula.org/projects/opennebula/repository/revisions/e2886255464e666c6a62d91cc1d016da6ab1a643/diff/</a></div><br><div class="gmail_quote">On Mon Jan 05 2015 at 12:56:49 PM Sebastiaan Smit <<a href="mailto:bas@echelon.nl">bas@echelon.nl</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="NL" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Javier,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The bug concerning the bulk creation of VM’s works as expected now. Do you have an idea of what the problem is while bulk deleting vm’s?<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Sebastiaan Smit<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">Van:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Javier Fontan [mailto:<a href="mailto:jfontan@opennebula.org" target="_blank">jfontan@opennebula.org</a>]
<br>
<b>Verzonden:</b> vrijdag 14 november 2014 15:44<br>
<b>Aan:</b> Sebastiaan Smit; <a href="mailto:users@lists.opennebula.org" target="_blank">users@lists.opennebula.org</a><br>
<b>Onderwerp:</b> Re: [one-users] Bulk delete of vCenter VM's leaves stray VM's<u></u><u></u></span></p></div></div><div lang="NL" link="blue" vlink="purple"><div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">There was a bug in the driver that caused error when deploying several VMs at the same time. To fix it change the file /var/lib/one/remotes/vmm/vcenter/vcenter_driver.rb at line 120 from this code:<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> def find_vm_template(uuid)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> vms = @dc.vmFolder.childEntity.grep(RbVmomi::VIM::VirtualMachine)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> return vms.find{ |v| v.config.uuid == uuid }<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> end<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">to this other one:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal"> def find_vm_template(uuid)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> vms = @dc.vmFolder.childEntity.grep(RbVmomi::VIM::VirtualMachine)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> return vms.find{ |v| v.config && v.config.uuid == uuid }<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> end<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We are still looking into the problem when deleting several VMs.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Thanks for telling us.<u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu Nov 13 2014 at 12:59:55 PM Javier Fontan <<a href="mailto:jfontan@opennebula.org" target="_blank">jfontan@opennebula.org</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal">Hi,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">We have opened an issue to track this problem:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><a href="http://dev.opennebula.org/issues/3334" target="_blank">http://dev.opennebula.org/issues/3334</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Meanwhile you can decrease the number of actions sent changing in /etc/one/oned.conf the parameter -t (number of threads) for VM driver. For example:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<div>
<p class="MsoNormal">VM_MAD = [<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> name = "vcenter",<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> executable = "one_vmm_sh",<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> arguments = "-p -t 2 -r 0 vcenter -s sh",<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> type = "xml" ]<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Cheers<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Wed Nov 12 2014 at 5:40:00 PM Sebastiaan Smit <<a href="mailto:bas@echelon.nl" target="_blank">bas@echelon.nl</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal">Hi list,<br>
<br>
We're testing the vCenter functionality in version 4.10 and see some strange behaviour while doing bulk actions.<br>
<br>
Deleting VM's sometimes leave stray VM's on our cluster. We see the following in de VM log:<br>
<br>
Sun Nov 9 15:51:34 2014 [Z0][LCM][I]: New VM state is RUNNING<br>
Wed Nov 12 17:30:36 2014 [Z0][LCM][I]: New VM state is CLEANUP.<br>
Wed Nov 12 17:30:36 2014 [Z0][VMM][I]: Driver command for 60 cancelled<br>
Wed Nov 12 17:30:36 2014 [Z0][DiM][I]: New VM state is DONE<br>
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Command execution fail: /var/lib/one/remotes/vmm/vcenter/cancel '423cdcae-b6b3-07c1-def6-96b9f3f4b7b3' 'demo-01' 60 demo-01<br>
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Cancel of VM 423cdcae-b6b3-07c1-def6-96b9f3f4b7b3 on host demo-01 failed due to "ManagedObjectNotFound: The object has already been deleted or has not been completely created"<br>
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 ExitCode: 255<br>
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Failed to execute virtualization driver operation: cancel.<br>
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: LOG I 60 Successfully execute network driver operation: clean.<br>
Wed Nov 12 17:30:41 2014 [Z0][VMM][W]: Ignored: CLEANUP SUCCESS 60<br>
<br>
We see it in a different manner while bulk creating VM's (20+ at a time):<br>
<br>
Sun Nov 9 16:01:34 2014 [Z0][DiM][I]: New VM state is ACTIVE.<br>
Sun Nov 9 16:01:34 2014 [Z0][LCM][I]: New VM state is PROLOG.<br>
Sun Nov 9 16:01:34 2014 [Z0][LCM][I]: New VM state is BOOT<br>
Sun Nov 9 16:01:34 2014 [Z0][VMM][I]: Generating deployment file: /var/lib/one/vms/81/deployment.0<br>
Sun Nov 9 16:01:34 2014 [Z0][VMM][I]: Successfully execute network driver operation: pre.<br>
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: Command execution fail: /var/lib/one/remotes/vmm/vcenter/deploy '/var/lib/one/vms/81/deployment.0' 'demo-01' 81 demo-01<br>
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: Deploy of VM 81 on host demo-01 with /var/lib/one/vms/81/deployment.0 failed due to "undefined method `uuid' for nil:NilClass"<br>
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: ExitCode: 255<br>
Sun Nov 9 16:01:36 2014 [Z0][VMM][I]: Failed to execute virtualization driver operation: deploy.<br>
Sun Nov 9 16:01:36 2014 [Z0][VMM][E]: Error deploying virtual machine<br>
Sun Nov 9 16:01:36 2014 [Z0][DiM][I]: New VM state is FAILED<br>
Wed Nov 12 17:30:19 2014 [Z0][DiM][I]: New VM state is DONE.<br>
Wed Nov 12 17:30:19 2014 [Z0][LCM][E]: epilog_success_action, VM in a wrong state<br>
<br>
<br>
I think these have two different root causes. The cluster is not under load.<br>
<br>
<br>
Has anyone else seen this behaviour?<br>
<br>
Best regards,<br>
--<br>
Sebastiaan Smit<br>
Echelon BV<br>
<br>
E: <a href="mailto:bas@echelon.nl" target="_blank">bas@echelon.nl</a><br>
W: <a href="http://www.echelon.nl" target="_blank">www.echelon.nl</a><br>
T: (088) 3243566 (gewijzigd nummer)<br>
T: (088) 3243505 (servicedesk)<br>
F: (053) 4336222<br>
<br>
KVK: 06055381<br>
<br>
<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><u></u><u></u></p>
</div>
</div>
</div>
</div></div></blockquote></div>