[one-users] Hung sshd processes on VM hosts

[Javier Fontan]

I've never had this problem myself but a thing worth trying is reusing
the connection. In ~oneadmin/.ssh/config we can add these lines:

--8<------
Host *
  ControlMaster auto
  ControlPath ~/.ssh/sockets/%r@%h-%p
  ControlPersist 600
------>8--

And create the directory ~oneadmin/.ssh/sockets. The first time it
connects to a host a new file will be created and will be reused for
the next connections to the same host.

I hope it fixes or alleviates the problem.

On Fri, Mar 14, 2014 at 6:55 PM, Steven Timm <timm at fnal.gov> wrote:
>
> We recently deployed several new and bigger hosts on our OpenNebula 3.2
> cloud and are seeing some issues.  At this point we are not sure if we are
> dealing with an OS problem with the sshd or something else.
> But the symptom is that we see a OpenNebula monitoring process come into the
> VM host as oneadmin, do its thing but then the sshd process
> (owned by root) that spawned the process starts using up to 100% of system
> cpu, and it is not killable at all.  strace of the sshd process simply hang.
> Eventually a lot of these build up on the VM host and it is almost
> impossible to do anything.  Only way to kill them we have found so far
> is to restart the parent sshd and then we can kill all the child sshd
> processes.
>
> The symptom tends to happen when there are more than 20 virtual machines on
> the same host.  These are new Ivy-Bridge based hosts that should be good for
> at least 40 VM's apiece.
>
> Has anyone seen anything like this before?  And yes, I know the 4.x series
> of opennebula is a lot more efficient in its monitoring and we are trying to
> get there as fast as we can.
>
> Steve Timm
> ------------------------------------------------------------------
> Steven C. Timm, Ph.D  (630) 840-8525
> timm at fnal.gov  http://home.fnal.gov/~timm/
> Fermilab Scientific Computing Division, Scientific Computing Services Quad.
> Grid and Cloud Services Dept., Associate Dept. Head for Cloud Computing
> _______________________________________________
> Users mailing list
> Users at lists.opennebula.org
> http://lists.opennebula.org/listinfo.cgi/users-opennebula.org



-- 
Javier Fontán Muiños
Developer
OpenNebula - The Open Source Toolkit for Data Center Virtualization
www.OpenNebula.org | @OpenNebula | github.com/jfontan