<div dir="ltr">Hi,<div><br></div><div style>Since 2.2 OpenNebula has to perform several new steps for each deployment. ACL rules, authrorization driver, usage quotas, and probably some other feature I'm missing.</div><div style>
<br></div><div style>To discard the scheduler, you can start the deployments with the onevm deploy command. This command takes a range of VM ids, but in that case the operations are sequential. I think this command should be enough to test concurrent deployments:</div>
<div style><br></div><div style>$ for i in `seq 1 4`; do (onevm deploy $i 0 &); done<br></div><div style><br></div><div style>You may also want to test different values for the TM and VMM -t parameter, to adjust the number of threads for each driver [1].</div>
<div style><br></div><div style>Regards</div><div style><br></div><div style>[1] <a href="http://opennebula.org/documentation:archives:rel3.8:oned_conf">http://opennebula.org/documentation:archives:rel3.8:oned_conf</a><br>
</div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">--<br>Join us at <a href="http://opennebulaconf.com" target="_blank">OpenNebulaConf2013</a> in Berlin, 24-26 September, 2013<br>--<div>Carlos Martín, MSc<br>
Project Engineer<br>OpenNebula - The Open-source Solution for Data Center Virtualization<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="mailto:cmartin@opennebula.org" style="color:rgb(42,93,176)" target="_blank"></a></span></div>
</div></div></div>
<br><br><div class="gmail_quote">On Tue, May 28, 2013 at 5:45 PM, Michael Berlin <span dir="ltr"><<a href="mailto:michael.berlin.xtreemfs@gmail.com" target="_blank">michael.berlin.xtreemfs@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<br>
I'm benchmarking the multi-deployment of VMs in OpenNebula to test the scalability of our distributed file system XtreemFS.<br>
<br>
Therefore, I do the following things:<br>
<br>
- stop the scheduler<br>
- "onevm create" multiple VMs<br>
- start the scheduler again<br>
<br>
- wait until the last VM has booted<br>
<br>
Recently, we upgraded our OpenNebula installation from 2.2 to 3.8 on our 32 node test cluster. With OpenNebula 2.2 the VMs were deployed almost simultaneously. But in 3.8 dispatching a single VM takes quite some time (1-2 seconds) for the scheduler.<br>
<br>
Here are the details:<br>
<br>
I benchmark the creation of the qcow2 snapshot in the "clone" transfer manager script and here's what it looked like for deploying 10 VMs with OpenNebula 2.2:<br>
<br>
1362253295.5779 clone_starting n03<br>
1362253295.5929 clone_starting n01<br>
1362253295.6138 clone_starting n00<br>
1362253295.6418 clone_starting n05<br>
1362253295.6428 clone_starting n04<br>
1362253295.6905 clone_starting n08<br>
1362253295.6960 clone_starting n09<br>
1362253295.7047 clone_starting n06<br>
1362253295.7113 clone_starting n02<br>
1362253295.7330 clone_starting n07<br>
1362253296.7047 clone_finished n05<br>
1362253296.7214 clone_finished n03<br>
1362253296.7353 clone_finished n01<br>
1362253296.7571 clone_finished n06<br>
1362253296.7677 clone_finished n09<br>
1362253296.7705 clone_finished n04<br>
1362253296.8035 clone_finished n08<br>
1362253296.8206 clone_finished n00<br>
1362253296.8214 clone_finished n02<br>
1362253296.8292 clone_finished n07<br>
<br>
The whole thing finished in under two seconds.<br>
<br>
With OpenNebula 3.8 it looks much different:<br>
<br>
1369752457.4118 clone_starting n13<br>
1369752457.4195 clone_finished n13<br>
1369752459.6483 clone_starting n17<br>
1369752459.6561 clone_finished n17<br>
1369752460.6465 clone_starting n08<br>
1369752460.6544 clone_finished n08<br>
1369752461.9516 clone_starting n12<br>
1369752461.9602 clone_finished n12<br>
1369752463.2860 clone_starting n15<br>
1369752463.2948 clone_finished n15<br>
1369752465.7036 clone_starting n14<br>
1369752465.7120 clone_finished n14<br>
1369752466.7329 clone_starting n11<br>
1369752466.7406 clone_finished n11<br>
1369752467.9151 clone_starting n10<br>
1369752467.9231 clone_finished n10<br>
1369752468.8460 clone_starting n16<br>
1369752468.8539 clone_finished n16<br>
1369752469.8849 clone_starting n09<br>
1369752469.8958 clone_finished n09<br>
<br>
Now, dispatching a single VM takes between 1-2 seconds. Here are the corresponding snippets from the sched.log:<br>
<br>
Tue May 28 16:47:35 2013 [VM][I]: Dispatching virtual machine 266 to host 98<br>
Tue May 28 16:47:36 2013 [VM][I]: Dispatching virtual machine 267 to host 102<br>
Tue May 28 16:47:36 2013 [VM][I]: Dispatching virtual machine 268 to host 93<br>
Tue May 28 16:47:39 2013 [VM][I]: Dispatching virtual machine 269 to host 97<br>
Tue May 28 16:47:41 2013 [VM][I]: Dispatching virtual machine 270 to host 100<br>
Tue May 28 16:47:41 2013 [VM][I]: Dispatching virtual machine 271 to host 99<br>
Tue May 28 16:47:43 2013 [VM][I]: Dispatching virtual machine 272 to host 96<br>
Tue May 28 16:47:44 2013 [VM][I]: Dispatching virtual machine 273 to host 95<br>
Tue May 28 16:47:44 2013 [VM][I]: Dispatching virtual machine 274 to host 101<br>
Tue May 28 16:47:45 2013 [VM][I]: Dispatching virtual machine 275 to host 94<br>
<br>
When I have a look at the sources, I suspect part of the problem is the blocking XML-RPC call to the one daemon (?):<br>
<br>
<a href="https://github.com/OpenNebula/one/blob/d732c5ae2fe774a2f0c0e24e6b60b3dc832a5f35/src/scheduler/src/pool/VirtualMachinePoolXML.cc#L133" target="_blank">https://github.com/OpenNebula/<u></u>one/blob/<u></u>d732c5ae2fe774a2f0c0e24e6b60b3<u></u>dc832a5f35/src/scheduler/src/<u></u>pool/VirtualMachinePoolXML.cc#<u></u>L133</a><br>
<br>
Nonetheless, it shouldn't take that long. Therefore, my questions are:<br>
<br>
- Is this normal? Can you please give advice how to further track down what takes so long?<br>
<br>
- With 2.2 you can clearly see the interleaving of multiple deployments while 3.8 processes them one at a time. Is there a way to get the old behavior back in a recent OpenNebula installation?<br>
<br>
Thank you very much for your help.<br>
<br>
Best regards,<br>
Michael<br>
______________________________<u></u>_________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/<u></u>listinfo.cgi/users-opennebula.<u></u>org</a><br>
</blockquote></div><br></div></div>