[one-users] Problem monitoring ESX host with the CentOS 6.2 appliance listed in marketplace

Fri Aug 24 04:43:17 PDT 2012

Dear Stephen,

Let's try to get to the bottom of this. I suggest removing the host
(onehost delete) and adding it again using the host's IP rather than the
hostname.

When OpenNebula tries to monitor the ESX host, it would be useful to have a
look to the log files in the ESX server to see if "virsh" is really getting
in touch with the ESX server of it is being cut off at some early stage
(ie, in the front-end or somewhere between the front-end and the ESX
server). Let us know if there is any new entry in any of the following log
files avaiable in the ESX server, matching the time shown in oned.log for
the monitor action upon the ESX server:

/var/log/auth.log
/var/log/vmkernel.log
/var/log/syslog.log

Best regards,

-Tino

--
Constantino Vázquez Blanco, MSc
Project Engineer
OpenNebula - The Open-Source Solution for Data Center Virtualization
www.OpenNebula.org | @tinova79 | @OpenNebula

 ****
>
>  ****
>
> *From:* Tino Vazquez [mailto:tinova at opennebula.org]
> *Sent:* 23 August 2012 15:22
> *To:* Howell, Stephen R
> *Cc:* users at lists.opennebula.org
> *Subject:* Re: [one-users] Problem monitoring ESX host with the CentOS
> 6.2 appliance listed in marketplace****
>
>  ****
>
> FYI
>
> On Thu, Aug 23, 2012 at 4:17 PM, Tino Vazquez <cvazquez at opennebula.org>
> wrote:****
>
>  Dear Stephan,****
>
>  ****
>
> Thanks a lot for the email, this way we can benefit from other people's
> insights. I couldn't see anything wrong in the env variables, so let's try
> the http_proxy env var possibility.****
>
>  ****
>
> Please replace the vmware.rb script with the one attached, and retry the
> monitorization. It basically does an unset of the http_proxy variable and
> then logs the complete command used. With the logged output we may be able
> to figure out what is going on.****
>
>  ****
>
> Best regards,****
>
>  ****
>
> -Tino
> --
> Constantino Vázquez Blanco, MSc
> Project Engineer
> OpenNebula - The Open-Source Solution for Data Center Virtualization
> www.OpenNebula.org | @tinova79 | @OpenNebula
>
>
>
> ****
>
> On Thu, Aug 23, 2012 at 3:42 PM, Howell, Stephen R <
> stephen.r.howell at jpmchase.com> wrote:****
>
>    I have opened a bug but it was closed.... I have no idea why this is
> not working.  Somehow the host info collection works when the script is run
> by oneadmin but not when invoked via the “onehost create” command.  Any
> help is welcome:****
>
>  ****
> Cannot add ESX host vai "onehost create" but
> /var/lib/one/remotes/im/run_probes vmware runs correctly under oneadmin
> account****
>
> Added by stephen howell <http://dev.opennebula.org/users/1197> about 23
> hours<http://dev.opennebula.org/projects/opennebula/activity?from=2012-08-22>ago. Updated about
> 2 hours<http://dev.opennebula.org/projects/opennebula/activity?from=2012-08-23>ago.
> ****
>
> *Status:*****
>
> Closed****
>
> *Start date:*****
>
> 08/22/2012****
>
> *Priority:*****
>
> Normal****
>
> *Due date:*****
>
> *Assignee:*****
>
> -****
>
> *% Done:*****
>
> 0%****
>
> *Category:*****
>
> -****
>
> *Target version:*****
>
> -****
>
> *Resolution:*****
>
> *Affected Version:*****
>
> OpenNebula 3.4****
>
> *Story points*****
>
> -****
>
> *Velocity based estimate*****
>
> -****
>     ------------------------------
>
> Quote <http://dev.opennebula.org/issues/1425> ****
>
> *Description*****
>
> When adding an ESX host to OpenNebula 3.4.1 using the CentOS appliance the
> host produces log similar to:****
>
> Monitoring host labhost (8)
> Tue Aug 21 14:42:48 2012 [InM][I]: Command execution fail: /var/lib/one/remotes/im/run_probes vmware 8 labhost
> Tue Aug 21 14:42:48 2012 [InM][E]: vmware.rb: Error executing: virsh -c 'esx://labhost/?no_verify=1&auto_answer=1' --readonly nodeinfo err: ExitCode: 1
>
> ****
>
> ** **
>
> ** **
>
>  ****
>
>  ****
>
> Tue Aug 21 14:42:48 2012 [InM][I]: out:
> Tue Aug 21 14:42:48 2012 [InM][I]: error: internal error curl_easy_perform() returned an error: Couldn't connect to server (7) : Failed to connect to X.X.X.X: Permission denied
>
> ****
>
> ** **
>
> ** **
>
>  ****
>
>  ****
>
> Tue Aug 21 14:42:48 2012 [InM][I]: error: failed to connect to the hypervisor
> Tue Aug 21 14:42:48 2012 [InM][E]: Error executing vmware.rb
> Tue Aug 21 14:42:48 2012 [InM][I]: ExitCode: 1
>
> ****
>
> ** **
>
> ** **
>
>  ****
>
>  ****
>
> Tue Aug 21 14:42:48 2012 [InM][E]: Error monitoring host 8 : MONITOR FAILURE 8 Error executing vmware.rb****
>
> HOWEVER, when running the command from the shell it completes without
> error:
> [oneadmin at ONE341 ~]$ /var/lib/one/remotes/im/run_probes vmware 8 labhost
> HYPERVISOR=vmware TOTALCPU=800 FREECPU=800 CPUSPEED=2612
> TOTALMEMORY=33550332 FREEMEMORY=26840265****
> History**** #1 <http://dev.opennebula.org/issues/1425#note-1>****
>  Updated by stephen howell <http://dev.opennebula.org/users/1197> about
> 23 hours<http://dev.opennebula.org/projects/opennebula/activity?from=2012-08-22>ago
> ****
>
> ESX host is ESX 4.1 update 1, SSH access for oneadmin is enabled, with
> passwords correctly added and SSH keys imported into the host.<http://dev.opennebula.org/issues/1425>
> ****
> #2 <http://dev.opennebula.org/issues/1425>****
> Updated by Tino Vázquez about 22 hours ago<http://dev.opennebula.org/issues/1425>
> ****
>
> Since the command executes correctly under the shell, I can only think of
> environmental variables. Could you compare the variables accesible from the
> vmware.rb script when ran directly on the shell compared with the ones
> available when ONE runs the script?****
>
> Also, are you running any sort of HTTP proxy?****
> *#3*****
>  Updated by *stephen howell* *about 22 hours* ago****
>
> Not familiar with ruby, could you provide a code snippet to show how I can
> show those environment variables?****
>
> No proxy here, direct connection although the system did have http_proxy
> set I have removed that and rebooted and symptoms remain unchanged.****
>
> Thanks****
> *#4*****
>  Updated by *Tino Vázquez* *about 22 hours* ago****
>
> Add the following in line 46 of vmware.rb****
>
> `env > /tmp/vmware_env_vars`****
>
> Run it once from the shell, backup the file, make ONE monitor the host,
> and compare both files.****
> *#5*****
>  Updated by *stephen howell* *about 21 hours* ago****
>
> Thanks for the assistance, this is the difference between the version
> called via "onehost create" and the one run from the shell:****
>
> [oneadmin at ONE341 ~]$ diff /tmp/vmware_env_vars /tmp/vmware_env_vars_shell
> 1a2,3****
>
> SELINUX_ROLE_REQUESTED=
> TERM=xterm****
>
> 3d4
> < TERM=linux
> 5c6,7
> < PRIORITY=19
> ---****
>
> SSH_CLIENT=169.66.210.159 58446 22
> SELINUX_USE_CURRENT_RANGE=****
>
> 7a10****
>
> SSH_TTY=/dev/pts/0****
>
> 8a12****
>
>
> LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=01;05;37;41:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lz=01;31:*.xz=01;31:*.bz2=01;31:*.tbz=01;31:*.tbz2=01;31:*.bz=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.rar=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.axv=01;35:*.anx=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=01;36:*.au=01;36:*.flac=01;36:*.mid=01;36:*.midi=01;36:*.mka=01;36:*.mp3=01;36:*.mpc=01;36:*.ogg=01;36:*.ra=01;36:*.wav=01;36:*.axa=01;36:*.oga=01;36:*.spx=01;36:*.xspf=01;36:
> ****
>
> 13a18****
>
> SELINUX_LEVEL_REQUESTED=****
>
> 15c20
> < SHLVL=5
> ---****
>
> SHLVL=3****
>
> 19a25****
>
> SSH_CONNECTION=169.66.210.159 58446 169.82.112.252 22****
>
> 22d27
> < ONE_MAD_DEBUG=****
> *#6*****
>  Updated by *stephen howell* *about 4 hours* ago****
>
> Please let me know if you need further information or if this bug is known
> to you and could be fixed by upgrading to a newer release. I have also
> noticed that I am unable to login to Sunstone, log error is " [E]: Error
> initializing authentication system" Perhaps the two are linked somehow.***
> *
> *#7*****
> Updated by *Tino Vázquez* *about 2 hours* ago****
>
>    - *Status* changed from *New* to *Closed*****
>
>  I would suggest an email to the support list describing the
> monitorization problem, and a second one for the sunstone problem. This way
> other members of the community can help. I am closing this bug as it is not
> clear wether this is a bug or a problem with configuration. We will reopen
> it if we find out in the mailing list that there is indeed a bug.****
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates. This transmission may contain
> information that is privileged, confidential, legally privileged, and/or
> exempt from disclosure under applicable law. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or use of the information contained herein (including any
> reliance thereon) is STRICTLY PROHIBITED. Although this transmission and
> any attachments are believed to be free of any virus or other defect that
> might affect any computer system into which it is received and opened, it
> is the responsibility of the recipient to ensure that it is virus free and
> no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and
> affiliates, as applicable, for any loss or damage arising in any way from
> its use. If you received this transmission in error, please immediately
> contact the sender and destroy the material in its entirety, whether in
> electronic or hard copy format. Thank you. Please refer to *
> http://www.jpmorgan.com/pages/disclosures* for disclosures relating to
> European legal entities. ****
>
>  ****
>
> _______________________________________________
> Users mailing list
> *Users at lists.opennebula.org*
> *http://lists.opennebula.org/listinfo.cgi/users-opennebula.org*****
>
>   <http://dev.opennebula.org/issues/1425>****
>
>   <http://dev.opennebula.org/issues/1425>****
>
>     ** **
>
> This communication is for informational purposes only. It is not intended
> as an offer or solicitation for the purchase or sale of any financial
> instrument or as an official confirmation of any transaction. All market
> prices, data and other information are not warranted as to completeness or
> accuracy and are subject to change without notice. Any comments or
> statements made herein do not necessarily reflect those of JPMorgan Chase &
> Co., its subsidiaries and affiliates. This transmission may contain
> information that is privileged, confidential, legally privileged, and/or
> exempt from disclosure under applicable law. If you are not the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution, or use of the information contained herein (including any
> reliance thereon) is STRICTLY PROHIBITED. Although this transmission and
> any attachments are believed to be free of any virus or other defect that
> might affect any computer system into which it is received and opened, it
> is the responsibility of the recipient to ensure that it is virus free and
> no responsibility is accepted by JPMorgan Chase & Co., its subsidiaries and
> affiliates, as applicable, for any loss or damage arising in any way from
> its use. If you received this transmission in error, please immediately
> contact the sender and destroy the material in its entirety, whether in
> electronic or hard copy format. Thank you. Please refer to
> http://www.jpmorgan.com/pages/disclosures for disclosures relating to
> European legal entities.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.opennebula.org/pipermail/users-opennebula.org/attachments/20120824/6a7b5ef5/attachment-0003.htm>