Many Processes in Down state?

[Schorschi]

Someone has to have seen this one? We had an ESX 3 host, running but currently had no active VMs, on Dell hardware, running Dell OMA 5.1.x agents, Netbackup IP client (on Service Console), memory dedicated to Service Console set to 800mb.

At some point early yesterday morning, multiple processes started changing state to 'down'. It appears that the host attempted to restart some of the processes these in turn failed, to the point that commands line ifconfig, would hang the SSH session or bind up the tty session at console.

Even vm-support would not run. We have never seen this before. We are hoping someone else has seen this?

We had all patches applied up to 3/29, less the OMA 5.0 and NFS patch, and the Susie specific patch. We deemed these patches not applicable since we are at OMA 5.1.1, don't have an Susie VMs, nor use NFS at all at this time.

[rreynol]

It is possible that the servic e console is bogged down by some runaway processes. What does the output of uptime look like, do you see a high load average? What about vmstat, does it show high cpu utilization? If you can run top, not esxtop, that might give you other clues.

We had an instance some time ago when some hp agents for storage caused a high load on the service console.

[Schorschi]

Unfortunately, by the time they engaged the appropriate team, they had already decided to rebuild the server, believing they had some type of filesystem corruption.

[Jae_Ellers]

you probably had a full /.

If you didn't customize the partition layout you will have a chance to troubleshoot this again.

My usual culprit is operators leaving .iso files connected that then get deleted or moved later. Outputs a log line every second until corrected or until the server blows.

Please mark questions as such and award points to helpful answers. Thanks!

Alternatively you could be out of service console memory if you have some extensive services running java apps (IBM director agents) or backup clients.