<div dir="ltr"><div><div>Hi Everyone,<br><br></div>We're currently experiencing a weird issue on our OpenNebula management node.<br><br></div>The node is setup with the following configuration:<br><br><ul><li>CentOS 6.6</li><li>Two interfaces in an balance-alb bond.</li><li>MySQL backend</li><li>/var/lib/one mounted on glusterfs and shared with all nodes.</li><li>OpenNebula 4.12</li><li>We're behind a proxy, so all proxy variables are set in the environment for all users.<br></li></ul><p><br></p><p>When
 working on the commandline everything works as a charm, you can list 
all nodes/datastores, view the details, etc. All working smooth and 
fast!</p><p>The problem we face however is working inside the sunstone 
interface. It seems that listing resources (nodes and datastores) works 
normally but when we click a datastore/node/whatever it takes ages 
(minutes) before it shows the contents.</p><p>The sunstone.error logs only give the following info (in debug mode):</p><p><span style="font-family:monospace,monospace">Errno::ETIMEDOUT - Connection timed out - connect(2):<br> /usr/lib/ruby/1.8/net/http.rb:560:in `initialize'<br> /usr/lib/ruby/1.8/net/http.rb:560:in `open'<br> /usr/lib/ruby/1.8/net/http.rb:560:in `connect'<br> /usr/lib/ruby/1.8/timeout.rb:53:in `timeout'<br> /usr/lib/ruby/1.8/timeout.rb:101:in `timeout'<br> /usr/lib/ruby/1.8/net/http.rb:560:in `connect'<br> /usr/lib/ruby/1.8/net/http.rb:553:in `do_start'<br> /usr/lib/ruby/1.8/net/http.rb:542:in `start'<br> /usr/lib/ruby/1.8/net/http.rb:1035:in `request'<br> /usr/lib/ruby/1.8/net/http.rb:772:in `get'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:80:in `perform_request'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:40:in `call'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:87:in `with_net_http_connection'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/adapter/net_http.rb:32:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/retry.rb:20:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/encode_json.rb:21:in `call'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/multipart.rb:14:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/upload.rb:16:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/request/etag_cache.rb:31:in `call'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/request/authorization.rb:38:in `call'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/parse_iso_dates.rb:11:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/logger.rb:20:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/callback.rb:14:in `call'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/response.rb:8:in `call'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/middleware/response/raise_error.rb:9:in `call'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/rack_builder.rb:139:in `build_response'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:377:in `run_request'<br> /usr/lib/ruby/gems/1.8/gems/faraday-0.9.1/lib/faraday/connection.rb:140:in `get'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:104:in `find!'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/actions.rb:119:in `find'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in `send'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/collection.rb:62:in `find'<br> /usr/lib/ruby/gems/1.8/gems/zendesk_api-1.4.6/lib/zendesk_api/client.rb:56:in `current_user'<br> /usr/lib/one/sunstone/routes/support.rb:66:in `zendesk_client'<br> /usr/lib/one/sunstone/routes/support.rb:121:in `GET /support/request'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `call'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:863:in `route'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `instance_eval'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:521:in `route_eval'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:500:in `route!'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `catch'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:497:in `route!'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `each'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:476:in `route!'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:601:in `dispatch!'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `instance_eval'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `catch'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:566:in `invoke'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:411:in `call!'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:399:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/commonlogger.rb:18:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/deflater.rb:13:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:63:in `context'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/session/abstract/id.rb:58:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/showexceptions.rb:24:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/methodoverride.rb:24:in `call'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:1005:in `synchronize'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:979:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/content_length.rb:13:in `call'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/chunked.rb:15:in `call'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:84:in `pre_process'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `catch'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:82:in `pre_process'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:57:in `process'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/connection.rb:42:in `receive_data'<br> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run_machine'<br> /usr/lib/ruby/gems/1.8/gems/eventmachine-0.12.10/lib/eventmachine.rb:256:in `run'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/backends/base.rb:61:in `start'<br> /usr/lib/ruby/gems/1.8/gems/thin-1.2.8/lib/thin/server.rb:159:in `start'<br> /usr/lib/ruby/gems/1.8/gems/rack-1.1.0/lib/rack/handler/thin.rb:14:in `run'<br> /usr/lib/ruby/gems/1.8/gems/sinatra-1.0/lib/sinatra/base.rb:946:in `run!'<br> /usr/lib/one/sunstone/sunstone-server.rb:627</span><br></p><p><br></p><p>When
 I click a datastore in the interface, nothing will happen in the logs 
(only access logs give some info). Then, when this error appears, only 
then will we get information about the datastore.</p><p>So everything works eventually, but there's a massive timeout causing this problem.</p><p>I've
 also noticed that I'm getting these connection time out errors without 
using the interface, so it might be something in the background doing 
some action?! If so, this action seems to block all other activities 
untill its finally done.<br></p><p>At first I thought this was IPv6 
related, so we've disabled that, then I doubted the bonding interface, 
but that wasnt it either. Then DNS, but even that didnt budge. We even 
did traces on the OS but didn't find anything suspicious.<br></p><p>Before we ran version 4.10, so after upgrading we hoped it would solve something, but that also didn't happen.</p><p>So
 right now we're kinda stuck... Does anyone have an idea on how to 
troubleshoot and solve this issue? What is causing this connection time 
out?<span class=""><font color="#888888"><br></font></span></p><br clear="all"><br>-- <br><div class="gmail_signature">Bart G.</div>
</div>