[one-users] Sunstone - Time out errors

Fri Mar 27 08:04:21 PDT 2015

Bart, 

Try to run 'onedb fsck' to check for any errors in DB [1].

[1] http://docs.opennebula.org/4.12/administration/references/onedb.html
--
Thank you,

Dmitri Chebotarov
VCL Sys Eng, Engineering & Architectural Support, TSD - Ent Servers & Messaging
223 Aquia Building, Ffx, MSN: 1B5
Phone: (703) 993-6175 | Fax: (703) 993-3404

> On Mar 12, 2015, at 6:50 , Bart <bart at pleh.info> wrote:
> 
> Hi Everyone,
> 
> We're currently experiencing a weird issue on our OpenNebula management node.
> 
> The node is setup with the following configuration:
> 
> 	• CentOS 6.6
> 	• Two interfaces in an balance-alb bond.
> 	• MySQL backend
> 	• /var/lib/one mounted on glusterfs and shared with all nodes.
> 	• OpenNebula 4.12
> 	• We're behind a proxy, so all proxy variables are set in the environment for all users.
> 
> 
> When working on the commandline everything works as a charm, you can list all nodes/datastores, view the details, etc. All working smooth and fast!
> 
> The problem we face however is working inside the sunstone interface. It seems that listing resources (nodes and datastores) works normally but when we click a datastore/node/whatever it takes ages (minutes) before it shows the contents.
> 
> The sunstone.error logs only give the following info (in debug mode):
> 
> Errno::ETIMEDOUT - Connection timed out - connect(2):
>  /usr/lib/ruby/1.8/net/http.rb:560:in `initialize'
>  /usr/lib/ruby/1.8/net/http.rb:560:in `open'
>  /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
>  /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
>  /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
>  /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
>  /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
>  /usr/lib/ruby/1.8/net/http.rb:542:in `start'
>  /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
>  /usr/lib/ruby/1.8/net/http.rb:772:in `get'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in `perform_request'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in `call'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/retry.rb:20:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/encode_json.rb:21:in `call'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/multipart.rb:14:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/upload.rb:16:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/etag_cache.rb:31:in `call'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/authorization.rb:38:in `call'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/parse_iso_dates.rb:11:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/logger.rb:20:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/callback.rb:14:in `call'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/raise_error.rb:9:in `call'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'
>  /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in `get'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:104:in `find!'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:119:in `find'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in `send'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in `find'
>  /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/client.rb:56:in `current_user'
>  /usr/lib/one/sunstone/routes/support.rb:66:in `zendesk_client'
>  /usr/lib/one/sunstone/routes/support.rb:121:in `GET /support/request'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/deflater.rb:13:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:63:in `context'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:58:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/showexceptions.rb:24:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in `call'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in `pre_process'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `pre_process'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in `process'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in `receive_data'
>  /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'
>  /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in `start'
>  /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
>  /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run'
>  /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:946:in `run!'
>  /usr/lib/one/sunstone/sunstone-server.rb:627
> 
> 
> 
> When I click a datastore in the interface, nothing will happen in the logs (only access logs give some info). Then, when this error appears, only then will we get information about the datastore.
> 
> So everything works eventually, but there's a massive timeout causing this problem.
> 
> I've also noticed that I'm getting these connection time out errors without using the interface, so it might be something in the background doing some action?! If so, this action seems to block all other activities untill its finally done.
> 
> At first I thought this was IPv6 related, so we've disabled that, then I doubted the bonding interface, but that wasnt it either. Then DNS, but even that didnt budge. We even did traces on the OS but didn't find anything suspicious.
> 
> Before we ran version 4.10, so after upgrading we hoped it would solve something, but that also didn't happen.
> 
> So right now we're kinda stuck... Does anyone have an idea on how to troubleshoot and solve this issue? What is causing this connection time out?
> 
> 
> 
> -- 
> Bart G.
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org