Hi there,<div><br></div><div>Thank you for the feedback.</div><div><br></div><div>I have a comment though. Nokogiri should be installed by the install_gems script [1], so maybe you skipped that step.</div><div>Nokogiri drastically improves the CLI speed, but since you use python it should be unrelated to your response time... </div>

<div><br></div><div>Regards</div><div><br></div><div>[1] <a href="http://opennebula.org/documentation:rel3.6:ignc#ruby_libraries_requirements_front-end">http://opennebula.org/documentation:rel3.6:ignc#ruby_libraries_requirements_front-end</a><br clear="all">

--<br>Carlos Martín, MSc<br>Project Engineer<br>OpenNebula - The Open-source Solution for Data Center Virtualization<div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="mailto:cmartin@opennebula.org" style="color:rgb(42,93,176)" target="_blank"></a></span></div>

<br>
<br><br><div class="gmail_quote">On Tue, Sep 25, 2012 at 11:21 AM, Christoph Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">


  
    
  
  <div bgcolor="#FFFFFF" text="#000000">
    <div>Hello Carlos, <br>
      <br>
      actually i'm running Ubuntu 11.04 on the x86_64 architecture.
      Opennebula 3.6 was installed from the sourcefiles.<br>
      <br>
      The xmlrpc-c is installed in version 1.16.32-0ubuntu3.1 .<br>
      Ruby is installed in Version 1.8.7 (2010-08-16 patchlevel 302).<br>
      <br>
      I didn't find any installation of the nokogiri ruby gem. So i
      can't provide you a version.<br>
      <br>
      Regards, <br>
      <br>
      Christoph<br>
      <br>
      <br>
      <br>
      <br>
      <br>
      Am <a href="tel:21.09.2012%2012" value="+12109201212" target="_blank">21.09.2012 12</a>:28, schrieb Carlos Martín Sánchez:<br>
    </div><div><div class="h5">
    <blockquote type="cite">Hi Christoph,
      <div><br>
      </div>
      <div>I'm sorry but I ran out of ideas. The only thing I can try is
        to replicate your complete setup and give it a try with gdb when
        I find the time.</div>
      <div><br>
      </div>
      <div>Could you please share your distribution, opennebula
        installation mode (from packages or source), and the versions of</div>
      <div>- xmlrpc-c library</div>
      <div>- ruby</div>
      <div>- nokogiri ruby gem</div>
      <div><br>
      </div>
      <div>Regards</div>
      <div>--<br>
        Carlos Martín, MSc<br>
        Project Engineer<br>
        OpenNebula - The Open-source Solution for Data Center
        Virtualization
        <div>
          <span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a>
            | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"></span></div>
        <br>
        <br>
        <br>
        <div class="gmail_quote">On Wed, Sep 5, 2012 at 2:53 PM,
          Christoph Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span>
          wrote:<br>
          <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
            <div bgcolor="#FFFFFF" text="#000000">
              <div>Hey, <br>
                <br>
                my oned.conf is very close to the default oned.conf. I
                only changed the DB Backend. So the values around
                MANAGER_TIMER looks like this:<br>
                <br>
                #MANAGER_TIMER = 30<br>
                <br>
                HOST_MONITORING_INTERVAL         = 600<br>
                #HOST_PER_INTERVAL               = 15<br>
                #HOST_MONITORING_EXPIRATION_TIME = 86400<br>
                <br>
                VM_POLLING_INTERVAL            = 600<br>
                #VM_PER_INTERVAL               = 5<br>
                #VM_MONITORING_EXPIRATION_TIME = 86400<br>
                <br>
                I assume, that the default values are used because of
                the comments. <br>
                <br>
                I didn't activate any  external authentication or
                authorization drivers by hand and even could see any but
                the default in the oned.conf. <br>
                My user is the oneadmin.<br>
                <br>
                Before using MySQL I used SQLite as Database. My first
                step to tackle the response time increasing was
                replacing SQLite with MySQL.<br>
                <br>
                I append the output of the MySQL command to this email. 
                I executed after during a 400 seconds stuck. <br>
                <br>
                Thanks for your help.<br>
                <br>
                Regards, <br>
                <br>
                Christoph Robbert<br>
                <br>
                <br>
                Am 05.09.2012 14:17, schrieb Carlos Martín Sánchez:<br>
              </div>
              <div>
                <div>
                  <blockquote type="cite">Hi,
                    <div><br>
                    </div>
                    <div>Let's try to rule out one thing at a time.</div>
                    <div><br>
                    </div>
                    <div>Did you set any timer values in oned.conf that
                      may overload opennebula? If the values
                      of MANAGER_TIMER, HOST and VM MONITORING_INTERVAL
                      are too low, opennebula could choke.</div>
                    <div><br>
                    </div>
                    <div>Do you have any external authentication or
                      authorization drivers enabled in oned.conf? Are
                      you using oneadmin to do the requests, or a
                      regular user? Doing a call to external drivers for
                      each request may be a possible reason...</div>
                    <div><br>
                    </div>
                    <div>Is the communication with MySQL the problem?
                      Next time you see OpenNebula slowing down, you
                      could try to execute, from the front-end machine,
                      the following:</div>
                    <div><br>
                    </div>
                    <div>$ mysql -u oneadmin -poneadmin -h localhost -P
                      0 opennebula -e "SELECT body FROM vm_pool WHERE
                      state<>6;"<br>
                      <br>
                    </div>
                    <div><br>
                    </div>
                    <div>Thanks for your feedback</div>
                    <div><br clear="all">
                      --<br>
                      Carlos Martín, MSc<br>
                      Project Engineer<br>
                      OpenNebula - The Open-source Solution for Data
                      Center Virtualization
                      <div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a> | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a> |
                          <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"></span></div>


                      <br>
                      <br>
                      <br>
                      <div class="gmail_quote">On Tue, Sep 4, 2012 at
                        1:25 PM, Christoph Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span>
                        wrote:<br>
                        <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
                          <div bgcolor="#FFFFFF" text="#000000">
                            <div>Hello, <br>
                              <br>
                              i use MySQL as Database. I query
                              Opennebula from python via pyoca[1]. But i
                              registered the same effect using the
                              command "onevm list". <br>
                              <br>
                              The effect depend also on the number of
                              running VMs, but i run only at maximum 30
                              VMs. The effect starts at round about 6
                              VMs. Usually the time increases to round
                              about one or two seconds. But suddenly it
                              response very slow >>>60 seconds
                              or didn't answer. <br>
                              <br>
                              Also the creation of a new VM getting
                              stucked (response time also increases to
                              over 60 seconds). <br>
                              <br>
                              Sometimes the time increases to round
                              about 240 seconds for one call. Then the
                              next call takes about one or two seconds.
                              <br>
                              <br>
                              I couldn't see an xml-rpc request in the
                              oned.log because my gui wait until the
                              last xml-rpc request is finished.<br>
                              <br>
                              I profile every part of my code with time
                              measurements and traced it down to the
                              xml-rpc requests to opennebula.<br>
                              <br>
                              Hope this help.<br>
                              <br>
                              Regards, <br>
                              <br>
                              Christoph Robbert<br>
                              <br>
                              <br>
                              <br>
                              [1] <a href="https://github.com/lukaszo/python-oca" target="_blank">
                                https://github.com/lukaszo/python-oca</a><br>
                              <br>
                              <br>
                              <br>
                              Am 04.09.2012 12:59, schrieb Carlos Martín
                              Sánchez:<br>
                            </div>
                            <div>
                              <div>
                                <blockquote type="cite">Hi,
                                  <div><br>
                                  </div>
                                  <div>Can you share some more
                                    information about your scenario? Are
                                    you using sqlite, or mysql? MySQL
                                    can drastically improve the
                                    performance over sqlite.</div>
                                  <div><br>
                                  </div>
                                  <div>How are you querying OpenNebula,
                                    are you using the CLI, our ruby/java
                                    OCA? The response time can be
                                    affected by the xml processing that
                                    the OCA has to do. If you are using
                                    Ruby, it is crucial that you have
                                    the nokogiri gem installed</div>
                                  <div><br>
                                  </div>
                                  <div>Does the response time increase
                                    always over time, or is it related
                                    to the number of existing VMs? If
                                    so, how many VMs does it take to
                                    make it irresponsive?</div>
                                  <div><br>
                                  </div>
                                  <div>Can you still see the xml-rpc
                                    requests in oned.log each second?</div>
                                  <div><br>
                                  </div>
                                  <div>I'm trying to reproduce the
                                    problem, having over a 1000 running
                                    VMs. I'm doing a onevm create &
                                    shutdown every 5 seconds while
                                    checking the time it takes to do a
                                    onevm list each second, but can't
                                    see any response taking more than
                                    one or two seconds.</div>
                                  <div><br>
                                  </div>
                                  <div>Regards<br clear="all">
                                    --<br>
                                    Carlos Martín, MSc<br>
                                    Project Engineer<br>
                                    OpenNebula - The Open-source
                                    Solution for Data Center
                                    Virtualization
                                    <div><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"><a href="http://www.OpenNebula.org" target="_blank">www.OpenNebula.org</a>
                                        | <a href="mailto:cmartin@opennebula.org" target="_blank">cmartin@opennebula.org</a>
                                        | <a href="http://twitter.com/opennebula" target="_blank">@OpenNebula</a></span><span style="border-collapse:collapse;color:rgb(136,136,136);font-family:arial,sans-serif;font-size:13px"></span></div>


                                    <br>
                                    <br>
                                    <br>
                                    <div class="gmail_quote">On Tue, Aug
                                      28, 2012 at 1:40 PM, Christoph
                                      Robbert <span dir="ltr"><<a href="mailto:chrobbert@googlemail.com" target="_blank">chrobbert@googlemail.com</a>></span>
                                      wrote:<br>
                                      <blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> Hello,<br>
                                        <br>
                                        I'm working on project with
                                        Opennebula 3.6 as
                                        cloudcontroller. We start and
                                        stop VMs via xml-rpc nearly
                                        every 15 seconds. To monitor the
                                        actions in realtime, i
                                        implemented a gui, which calls
                                        Opennebula every second via
                                        xml-rpc. Now i notice a real big
                                        increase of the response time
                                        after 10 minutes. The response
                                        time increases from nearly 1
                                        second to 5 Minutes. Some time i
                                        have to restart Opennebula
                                        because the response time
                                        increase to infinity.<br>
                                        Could you give me a hind where i
                                        should start to trace the
                                        bottleneck in Opennebula?<br>
                                        <br>
                                        <br>
                                        Best Regards,<br>
                                        <br>
                                        Christoph Robbert<br>
_______________________________________________<br>
                                        Users mailing list<br>
                                        <a href="mailto:Users@lists.opennebula.org" target="_blank">Users@lists.opennebula.org</a><br>
                                        <a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
                                      </blockquote>
                                    </div>
                                    <br>
                                  </div>
                                </blockquote>
                                <br>
                              </div>
                            </div>
                          </div>
                        </blockquote>
                      </div>
                      <br>
                    </div>
                  </blockquote>
                  <br>
                </div>
              </div>
            </div>
          </blockquote>
        </div>
        <br>
      </div>
    </blockquote>
    <br>
  </div></div></div>

<br>_______________________________________________<br>
Users mailing list<br>
<a href="mailto:Users@lists.opennebula.org">Users@lists.opennebula.org</a><br>
<a href="http://lists.opennebula.org/listinfo.cgi/users-opennebula.org" target="_blank">http://lists.opennebula.org/listinfo.cgi/users-opennebula.org</a><br>
<br></blockquote></div><br></div>