[one-users] econe-server problems with 3.2.1
Ruben S. Montero
rsmontero at opennebula.org
Mon Feb 6 07:33:47 PST 2012
Hi
Could you try interacting directly with the server (i.e. without the
proxy part?). We are not able to reproduce this.. The server makes a
call equivalent to a onevm list and returns the output, in our
installation with ~500 VMs we are seeing a 1s overhead because of the
HTTP process. But the overall response time is < 2s for the EC2
interface and <1s from the CLI.
Cheers
Ruben
On Mon, Feb 6, 2012 at 11:42 AM, Ulrich Schwickerath
<ulrich.schwickerath at cern.ch> wrote:
> Hi, all,
>
> we are still experiencing problems after the upgrade to ONE3.2.1. The EC2
> access is very very slow and unstable. This morning, I noticed a lot of
> blocked requests to econe, and error message in the econe-server log files:
>
>
> NoMethodError - undefined method `[]' for nil:NilClass:
> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:216:in `render_state'
> (erb):20:in `describe_instances'
> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `call'
> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `each_element'
> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:239:in
> `each'
> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
> `upto'
> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
> `each'
> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:324:in `each_element'
> /usr/lib/one/ruby/OpenNebula/Pool.rb:100:in `each'
> (erb):14:in `describe_instances'
> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:181:in `describe_instances'
> /usr/lib/one/ruby/cloud/econe/econe-server.rb:176:in `do_http_request'
> /usr/lib/one/ruby/cloud/econe/econe-server.rb:158:in `POST /'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
> `compile!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
> `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
> `route_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in
> `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in
> `process_route'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
> `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
> `process_route'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in
> `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in
> `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in
> `dispatch!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
> `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
> `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/head.rb:9:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/showexceptions.rb:21:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in
> `synchronize'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in
> `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
> `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in
> `process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in
> `receive_data'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
> `run_machine'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
> `run'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in
> `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1234:in
> `run!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/main.rb:25
> /usr/lib/one/ruby/cloud/econe/econe-server.rb:165
> NoMethodError - undefined method `[]' for nil:NilClass:
> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:216:in `render_state'
> (erb):20:in `describe_instances'
> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `call'
> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:326:in `each_element'
> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:239:in
> `each'
> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
> `upto'
> /usr/lib/ruby/gems/1.8/gems/nokogiri-1.4.3.1/lib/nokogiri/xml/node_set.rb:238:in
> `each'
> /usr/lib/one/ruby/OpenNebula/XMLUtils.rb:324:in `each_element'
> /usr/lib/one/ruby/OpenNebula/Pool.rb:100:in `each'
> (erb):14:in `describe_instances'
> /usr/lib/one/ruby/cloud/econe/EC2QueryServer.rb:181:in `describe_instances'
> /usr/lib/one/ruby/cloud/econe/econe-server.rb:176:in `do_http_request'
> /usr/lib/one/ruby/cloud/econe/econe-server.rb:158:in `POST /'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1151:in
> `compile!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
> `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:724:in
> `route_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:708:in
> `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:758:in
> `process_route'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
> `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:755:in
> `process_route'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:707:in
> `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in `each'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:706:in
> `route!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:843:in
> `dispatch!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
> `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `instance_eval'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `catch'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:808:in
> `invoke'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:644:in
> `call!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:629:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/head.rb:9:in `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/showexceptions.rb:21:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1303:in
> `synchronize'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1272:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in
> `call'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in
> `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
> `pre_process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in
> `process'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in
> `receive_data'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
> `run_machine'
> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
> `run'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in
> `start'
> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/base.rb:1234:in
> `run!'
> /usr/lib/ruby/gems/1.8/gems/sinatra-1.2.6/lib/sinatra/main.rb:25
> /usr/lib/one/ruby/cloud/econe/econe-server.rb:165
>
> What I mean with slow is that to get a list of O(400) virtual machines the
> system needs O(5-10) minutes while local queries with onevm list take ~2
> seconds.
>
> We are currently bitten badly by this problem because we use this to refill
> our virtual batch farm. Is there a downgrade path ?
>
> Any idea ?
>
> Thanks,
> Ulrich
>
> --
> --------------------------------------
> Dr. Ulrich Schwickerath
> CERN IT/PES-PS
> 1211 Geneva 23
> e-mail: ulrich.schwickerath at cern.ch
> phone: +41 22 767 9576
> mobile: +41 76 487 5602
>
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org
--
Ruben S. Montero, PhD
Project co-Lead and Chief Architect
OpenNebula - The Open Source Toolkit for Data Center Virtualization
www.OpenNebula.org | rsmontero at opennebula.org | @OpenNebula
More information about the Users
mailing list