[one-users] Sunstone - Time out errors

Thu Mar 12 03:12:32 PDT 2015

Hi Everyone,

We're currently experiencing a weird issue on our OpenNebula management
node.

The node is setup with the following configuration:

   - CentOS 6.6
   - Two interfaces in an balance-alb bond.
   - MySQL backend
   - /var/lib/one mounted on glusterfs and shared with all nodes.
   - OpenNebula 4.12
   - We're behind a proxy, so all proxy variables are set in the
   environment for all users.

When working on the commandline everything works as a charm, you can list
all nodes/datastores, view the details, etc. All working smooth and fast!

The problem we face however is working inside the sunstone interface. It
seems that listing resources (nodes and datastores) works normally but when
we click a datastore/node/whatever it takes ages (minutes) before it shows
the contents.

The sunstone.error logs only give the following info (in debug mode):

Errno::ETIMEDOUT - Connection timed out - connect(2):
 /usr/lib/ruby/1.8/net/http.rb:560:in `initialize'
 /usr/lib/ruby/1.8/net/http.rb:560:in `open'
 /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'
 /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'
 /usr/lib/ruby/1.8/net/http.rb:560:in `connect'
 /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'
 /usr/lib/ruby/1.8/net/http.rb:542:in `start'
 /usr/lib/ruby/1.8/net/http.rb:1035:in `request'
 /usr/lib/ruby/1.8/net/http.rb:772:in `get'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in
`perform_request'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in
`call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in
`with_net_http_connection'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/retry.rb:20:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/encode_json.rb:21:in
`call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/multipart.rb:14:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/upload.rb:16:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/etag_cache.rb:31:in
`call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/authorization.rb:38:in
`call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/parse_iso_dates.rb:11:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/logger.rb:20:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/callback.rb:14:in
`call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in
`call'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/raise_error.rb:9:in
`call'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in
`build_response'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in
`run_request'
 /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in
`get'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:104:in
`find!'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:119:in
`find'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in
`send'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in
`find'
 /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/client.rb:56:in
`current_user'
 /usr/lib/one/sunstone/routes/support.rb:66:in `zendesk_client'
 /usr/lib/one/sunstone/routes/support.rb:121:in `GET /support/request'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in
`instance_eval'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in
`route_eval'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in
`dispatch!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in
`instance_eval'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in
`call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/deflater.rb:13:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:63:in
`context'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:58:in
`call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/showexceptions.rb:24:in
`call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in
`call'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in
`synchronize'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in
`call'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in
`pre_process'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in
`pre_process'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in
`process'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in
`receive_data'
 /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
`run_machine'
 /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in
`run'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in
`start'
 /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'
 /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run'
 /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:946:in `run!'
 /usr/lib/one/sunstone/sunstone-server.rb:627

When I click a datastore in the interface, nothing will happen in the logs
(only access logs give some info). Then, when this error appears, only then
will we get information about the datastore.

So everything works eventually, but there's a massive timeout causing this
problem.

I've also noticed that I'm getting these connection time out errors without
using the interface, so it might be something in the background doing some
action?! If so, this action seems to block all other activities untill its
finally done.

At first I thought this was IPv6 related, so we've disabled that, then I
doubted the bonding interface, but that wasnt it either. Then DNS, but even
that didnt budge. We even did traces on the OS but didn't find anything
suspicious.

Before we ran version 4.10, so after upgrading we hoped it would solve
something, but that also didn't happen.

So right now we're kinda stuck... Does anyone have an idea on how to
troubleshoot and solve this issue? What is causing this connection time out?

-- 
Bart G.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20150312/4fd07ed4/attachment.htm>