Hi Christoph,<div><br></div><div>I'm sorry but I ran out of ideas. The only thing I can try is to replicate your complete setup and give it a try with gdb when I find the time.</div><div><br></div><div>Could you please share your distribution, opennebula installation mode (from packages or source), and the versions of</div>
<div>- xmlrpc-c library</div><div>- ruby</div><div>- nokogiri ruby gem</div><div><br></div><div>Regards</div><div>--<br>Carlos Martín, MSc<br>Project Engineer<br>OpenNebula - The Open-source Solution for Data Center Virtualization<div>
<span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="mailto:cmartin@opennebula.org" style="color:rgb(42,93,176)" target="_blank"></a></span></div>
<br>
<br><br><div class="gmail_quote">On Wed, Sep 5, 2012 at 2:53 PM, Christoph Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hey, <br>
<br>
my oned.conf is very close to the default oned.conf. I only
changed the DB Backend. So the values around MANAGER_TIMER looks
like this:<br>
<br>
#MANAGER_TIMER = 30<br>
<br>
HOST_MONITORING_INTERVAL = 600<br>
#HOST_PER_INTERVAL = 15<br>
#HOST_MONITORING_EXPIRATION_TIME = 86400<br>
<br>
VM_POLLING_INTERVAL = 600<br>
#VM_PER_INTERVAL = 5<br>
#VM_MONITORING_EXPIRATION_TIME = 86400<br>
<br>
I assume, that the default values are used because of the
comments. <br>
<br>
I didn't activate any external authentication or authorization
drivers by hand and even could see any but the default in the
oned.conf. <br>
My user is the oneadmin.<br>
<br>
Before using MySQL I used SQLite as Database. My first step to
tackle the response time increasing was replacing SQLite with
MySQL.<br>
<br>
I append the output of the MySQL command to this email. I
executed after during a 400 seconds stuck. <br>
<br>
Thanks for your help.<br>
<br>
Regards, <br>
<br>
Christoph Robbert<br>
<br>
<br>
Am 05.09.2012 14:17, schrieb Carlos Martín Sánchez:<br>
</div><div><div>
<blockquote type="cite">Hi,
<div><br>
</div>
<div>Let's try to rule out one thing at a time.</div>
<div><br>
</div>
<div>Did you set any timer values in oned.conf that may overload
opennebula? If the values of MANAGER_TIMER, HOST and
VM MONITORING_INTERVAL are too low, opennebula could choke.</div>
<div><br>
</div>
<div>Do you have any external authentication or authorization
drivers enabled in oned.conf? Are you using oneadmin to do the
requests, or a regular user? Doing a call to external drivers
for each request may be a possible reason...</div>
<div><br>
</div>
<div>Is the communication with MySQL the problem? Next time you
see OpenNebula slowing down, you could try to execute, from the
front-end machine, the following:</div>
<div><br>
</div>
<div>$ mysql -u oneadmin -poneadmin -h localhost -P 0 opennebula
-e "SELECT body FROM vm_pool WHERE state<>6;"<br>
<br>
</div>
<div><br>
</div>
<div>Thanks for your feedback</div>
<div><br clear="all">
--<br>
Carlos Martín, MSc<br>
Project Engineer<br>
OpenNebula - The Open-source Solution for Data Center
Virtualization
<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a>
| <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"></span></div>
<br>
<br>
<br>
<div class="gmail_quote">On Tue, Sep 4, 2012 at 1:25 PM,
Christoph Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div>Hello, <br>
<br>
i use MySQL as Database. I query Opennebula from python
via pyoca[1]. But i registered the same effect using the
command "onevm list". <br>
<br>
The effect depend also on the number of running VMs, but
i run only at maximum 30 VMs. The effect starts at round
about 6 VMs. Usually the time increases to round about
one or two seconds. But suddenly it response very slow
>>>60 seconds or didn't answer. <br>
<br>
Also the creation of a new VM getting stucked (response
time also increases to over 60 seconds). <br>
<br>
Sometimes the time increases to round about 240 seconds
for one call. Then the next call takes about one or two
seconds. <br>
<br>
I couldn't see an xml-rpc request in the oned.log
because my gui wait until the last xml-rpc request is
finished.<br>
<br>
I profile every part of my code with time measurements
and traced it down to the xml-rpc requests to
opennebula.<br>
<br>
Hope this help.<br>
<br>
Regards, <br>
<br>
Christoph Robbert<br>
<br>
<br>
<br>
[1] <a href="https://github.com/lukaszo/python-oca" target="_blank"> https://github.com/lukaszo/python-oca</a><br>
<br>
<br>
<br>
Am 04.09.2012 12:59, schrieb Carlos Martín Sánchez:<br>
</div>
<div>
<div>
<blockquote type="cite">Hi,
<div><br>
</div>
<div>Can you share some more information about your
scenario? Are you using sqlite, or mysql? MySQL
can drastically improve the performance over
sqlite.</div>
<div><br>
</div>
<div>How are you querying OpenNebula, are you using
the CLI, our ruby/java OCA? The response time can
be affected by the xml processing that the OCA has
to do. If you are using Ruby, it is crucial that
you have the nokogiri gem installed</div>
<div><br>
</div>
<div>Does the response time increase always over
time, or is it related to the number of existing
VMs? If so, how many VMs does it take to make it
irresponsive?</div>
<div><br>
</div>
<div>Can you still see the xml-rpc requests in
oned.log each second?</div>
<div><br>
</div>
<div>I'm trying to reproduce the problem, having
over a 1000 running VMs. I'm doing a onevm create
& shutdown every 5 seconds while checking the
time it takes to do a onevm list each second, but
can't see any response taking more than one or two
seconds.</div>
<div><br>
</div>
<div>Regards<br clear="all">
--<br>
Carlos Martín, MSc<br>
Project Engineer<br>
OpenNebula - The Open-source Solution for Data
Center Virtualization
<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> |
<a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"></span></div>
<br>
<br>
<br>
<div class="gmail_quote">On Tue, Aug 28, 2012 at
1:40 PM, Christoph Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hello,<br>
<br>
I'm working on project with Opennebula 3.6 as
cloudcontroller. We start and stop VMs via
xml-rpc nearly every 15 seconds. To monitor
the actions in realtime, i implemented a gui,
which calls Opennebula every second via
xml-rpc. Now i notice a real big increase of
the response time after 10 minutes. The
response time increases from nearly 1 second
to 5 Minutes. Some time i have to restart
Opennebula because the response time increase
to infinity.<br>
Could you give me a hind where i should start
to trace the bottleneck in Opennebula?<br>
<br>
<br>
Best Regards,<br>
<br>
Christoph Robbert<br>
_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>