[one-users] econe-server problems with 3.2.1
Ulrich Schwickerath
ulrich.schwickerath at cern.ch
Tue Feb 7 09:39:22 PST 2012
Hi, Daniel,
> Would you mind to try the following patch?:
> https://gist.github.com/c7939c4148bb196ae725
This gives me an error when I try to run the query:
$ euca-describe-images
IMAGE ami-00000023 glExecWN_slc5_x86_64_kvm lsfadmin
available private i386 machine
IMAGE ami-00000024 glExecWN_slc6_x86_64_kvm lsfadmin
available private i386 machine
$ time euca-describe-instances
<!DOCTYPE html><html><head> <meta http-equiv="Content-Type"
content="text/html; charset=utf-8"/> <title>NoMethodError at /</title>
<script type="text/javascript"> //<!-- function toggle(id) { var
pre = document.getElementById("pre-" + id); var post =
document.getElementById("post-" + id); var context =
document.getElementById("context-" + id); if (pre.style.display ==
'block') { pre.style.display = 'none'; post.style.display =
'none'; context.style.background = "none"; } else {
pre.style.display = 'block'; post.style.display = 'block';
context.style.background = "#fffed9"; } } function
toggleBacktrace(){ var bt = document.getElementById("backtrace");
var toggler = document.getElementById("expando"); if (bt.className ==
'condensed') { bt.className = 'expanded'; toggler.innerHTML =
"(condense)"; } else { bt.className = 'condensed';
toggler.innerHTML = "(expand)"; } } //--> </script><style
type="text/css" media="screen"> * {margin: 0;
padding: 0; border: 0; outline: 0;} div.clear {clear: both;}
body {background: #EEEEEE; margin: 0; padding:
0; font-family: 'Lucida Grande', 'Lucida Sans
Unicode', 'Garuda';} code
{font-family: 'Lucida Console', monospace;
font-size: 12px;} li {height: 18px;}
ul {list-style: none; margin: 0; padding: 0;}
ol:hover {cursor: pointer;} ol li
{white-space: pre;} #explanation {font-size: 12px; color:
#666666; margin: 20px 0 0 100px;}/* WRAP */
#wrap {width: 1000px; background: #FFFFFF; margin: 0
auto; padding: 30px 50px 20px
50px; border-left: 1px solid
#DDDDDD; border-
(...)
<tr> <td>rack.request.query_string</td> <td
class="code"><div></div></td> </tr> <tr> <td>rack.run_once</td> <td
class="code"><div>false</div></td> </tr> <tr> <td>rack.url_scheme</td>
<td class="code"><div>http</div></td> </tr> <tr> <td>rack.version</td>
<td class="code"><div>[1, 0]</div></td> </tr> <tr>
<td>sinatra.error</td> <td class="code"><div>#<NoMethodError:
undefined method `[]' for nil:NilClass></div></td> </tr> </table>
<div class="clear"></div> </div> <!-- /RACK ENV --> <p
id="explanation">You're seeing this error because you haveenabled the
<code>show_exceptions</code> setting.</p> </div> <!-- /WRAP -->
</body></html>
real 1m9.148s
user 0m0.334s
sys 0m0.048s
>> One thing that might give a clue: I have a cron job which queries the system
>> and counts the number of running VMs. If not all leases are full,
>> some new batch VMs are started. This "sometimes" seems to work, however, if
>> it does not or if it takes too long to respond, there are concurrent queries
>> to the system (from up to 3 different client machines).
>>
>> From the output above I don't think the SSL proxy is causing the problem.
>> Could it be that one of my gems is too old or buggy ?
>>
>> Thanks for your help!
>> Ulrich
>>
>>
>>
>>
>> On 02/06/2012 04:33 PM, Ruben S. Montero wrote:
>>> Hi
>>>
>>> Could you try interacting directly with the server (i.e. without the
>>> proxy part?). We are not able to reproduce this.. The server makes a
>>> call equivalent to a onevm list and returns the output, in our
>>> installation with ~500 VMs we are seeing a 1s overhead because of the
>>> HTTP process. But the overall response time is< 2s for the EC2
>>> interface and<1s from the CLI.
>>>
>>> Cheers
>>>
>>> Ruben
>>>
>>> On Mon, Feb 6, 2012 at 11:42 AM, Ulrich Schwickerath
>>> <ulrich.schwickerath at cern.ch> wrote:
>>>> Hi, all,
>>>>
>>>> we are still experiencing problems after the upgrade to ONE3.2.1. The EC2
>>>> access is very very slow and unstable. This morning, I noticed a lot of
>>>> blocked requests to econe, and error message in the econe-server log
>>>> files:
>>>>
>>>>
>>>> NoMethodError - undefined method `[]' for nil:NilClass:
>>>> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:216:in `render_state'
>>>> (erb):20:in `describe_instances'
>>>> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `call'
>>>> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `each_element'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:239:in
>>>> `each'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
>>>> `upto'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
>>>> `each'
>>>> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:324:in `each_element'
>>>> /usr/lib/one/ruby/OpenNebula/Pool.rb:100:in `each'
>>>> (erb):14:in `describe_instances'
>>>> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:181:in
>>>> `describe_instances'
>>>> /usr/lib/one/ruby/cloud/econe/econe-server.rb:176:in `do_http_request'
>>>> /usr/lib/one/ruby/cloud/econe/econe-server.rb:158:in `POST /'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
>>>> `compile!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
>>>> `instance_eval'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
>>>> `route_eval'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in
>>>> `route!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in
>>>> `process_route'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
>>>> `catch'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
>>>> `process_route'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in
>>>> `route!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in
>>>> `each'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in
>>>> `route!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in
>>>> `dispatch!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
>>>> `call!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `instance_eval'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `invoke'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `catch'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `invoke'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
>>>> `call!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/head.rb:9:in `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in
>>>> `call'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/showexceptions.rb:21:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in
>>>> `synchronize'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in
>>>> `pre_process'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
>>>> `catch'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
>>>> `pre_process'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in
>>>> `process'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in
>>>> `receive_data'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
>>>> `run_machine'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
>>>> `run'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in
>>>> `start'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in
>>>> `run'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1234:in
>>>> `run!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/main.rb:25
>>>> /usr/lib/one/ruby/cloud/econe/econe-server.rb:165
>>>> NoMethodError - undefined method `[]' for nil:NilClass:
>>>> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:216:in `render_state'
>>>> (erb):20:in `describe_instances'
>>>> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `call'
>>>> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `each_element'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:239:in
>>>> `each'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
>>>> `upto'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
>>>> `each'
>>>> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:324:in `each_element'
>>>> /usr/lib/one/ruby/OpenNebula/Pool.rb:100:in `each'
>>>> (erb):14:in `describe_instances'
>>>> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:181:in
>>>> `describe_instances'
>>>> /usr/lib/one/ruby/cloud/econe/econe-server.rb:176:in `do_http_request'
>>>> /usr/lib/one/ruby/cloud/econe/econe-server.rb:158:in `POST /'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
>>>> `compile!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
>>>> `instance_eval'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
>>>> `route_eval'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in
>>>> `route!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in
>>>> `process_route'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
>>>> `catch'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
>>>> `process_route'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in
>>>> `route!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in
>>>> `each'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in
>>>> `route!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in
>>>> `dispatch!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
>>>> `call!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `instance_eval'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `invoke'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `catch'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
>>>> `invoke'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
>>>> `call!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/head.rb:9:in `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in
>>>> `call'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/showexceptions.rb:21:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in
>>>> `synchronize'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in
>>>> `call'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in
>>>> `pre_process'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
>>>> `catch'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
>>>> `pre_process'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in
>>>> `process'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in
>>>> `receive_data'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
>>>> `run_machine'
>>>>
>>>> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
>>>> `run'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in
>>>> `start'
>>>> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
>>>> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in
>>>> `run'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1234:in
>>>> `run!'
>>>> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/main.rb:25
>>>> /usr/lib/one/ruby/cloud/econe/econe-server.rb:165
>>>>
>>>> What I mean with slow is that to get a list of O(400) virtual machines
>>>> the
>>>> system needs O(5-10) minutes while local queries with onevm list take ~2
>>>> seconds.
>>>>
>>>> We are currently bitten badly by this problem because we use this to
>>>> refill
>>>> our virtual batch farm. Is there a downgrade path ?
>>>>
>>>> Any idea ?
>>>>
>>>> Thanks,
>>>> Ulrich
>>>>
>>>> --
>>>> --------------------------------------
>>>> Dr. Ulrich Schwickerath
>>>> CERN IT/PES-PS
>>>> 1211 Geneva 23
>>>> e-mail: ulrich.schwickerath at cern.ch
>>>> phone: +41 22 767 9576
>>>> mobile: +41 76 487 5602
>>>>
>>>> _______________________________________________
>>>> Users mailing list
>>>> Users at lists.opennebula.org
>>>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>>>
>>>
>>
>> --
>> --------------------------------------
>> Dr. Ulrich Schwickerath
>> CERN IT/PES-PS
>> 1211 Geneva 23
>> e-mail: ulrich.schwickerath at cern.ch
>> phone: +41 22 767 9576
>> mobile: +41 76 487 5602
>>
>> _______________________________________________
>> Users mailing list
>> Users at lists.opennebula.org
>> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
>
>
--
--------------------------------------
Dr. Ulrich Schwickerath
CERN IT/PES-PS
1211 Geneva 23
e-mail: ulrich.schwickerath at cern.ch
phone: +41 22 767 9576
mobile: +41 76 487 5602
More information about the Users
mailing list