ESXi 5.5 SNMP Agent not responding (failed 11)

[bomartin]

Hello I am having issue with snmp in ESXi 5.

5.0 Build 1746018

SNMP test failing with the below error:

UpdateDaemon() call failed, reason: Agent not responding, connect uds socket(/var/run/snmp.ctl) failed 11, err= Resource temporarily unavailable

~ # esxcli system snmp get

   Authentication:

   Communities: COMMUNITY1234

   Enable: true

   Engineid: 00000063000000a100000000

   Hwsrc: indications

   Loglevel: info

   Notraps:

   Port: 161

   Privacy:

   Remoteusers:

   Syscontact:

   Syslocation:

   Targets: x.x.x.x@162 x.x.x.x@162 COMMUNITY1234COMMUNITY1234 

   Users:

   V3targets:

SNMP set to factory defaults could not resolve the problem.

Any suggestions?

[MKguy]

Make sure the snmpd service is actually running by executing these commands:

# /etc/init.d/snmpd status

# ps -c | grep -i snmp

# esxcli network ip connection list | grep -i snmp

Also check the log at /var/log/syslog.log after you run a test for any snmpd related messages.

The snmpd service can be enabled and started through the GUI here:

[bomartin]

Hi MKguy,

thanks for your fast response. I could not open logfile, I get an information "permission denied" .

~ # /etc/init.d/snmpd status

snmpd is running

~ # ps -c | grep -i snmp

52600303 52600303 sh                   /bin/sh /etc/init.d/snmpd restart

52996324 52996324 grep                 grep -i snmp

49654584 49654584 snmpd                /sbin/snmpd

52608243 52608243 sh                   /bin/sh /etc/init.d/snmpd restart

~ # esxcli network ip connection list | grep -i snmp

udp         0       0  :::0                             :::0                               49654584           snmpd

udp         0       0  0.0.0.0:39266                    0.0.0.0:0                          49654584           snmpd

udp     42032       0  0.0.0.0:161                      0.0.0.0:0                          49654584           snmpd

udp         0       0  :::161                           :::0                               49654584           snmpd

~ #

~ #

~ # /var/log/syslog.log

-sh: /var/log/syslog.log: Permission denied

~ # esxcli system snmp test

Agent not responding, connect uds socket(/var/run/snmp.ctl) failed 11, err= Resource temporarily unavailable

~ #

My other esxi host with the same version and setup with snmp works fine.

[brunofernandez1]

have a look at the ESXi firewall. are there the needed ports open?

[MKguy]

~ # /var/log/syslog.log

-sh: /var/log/syslog.log: Permission denied

Of course you have to open the logfile at /var/log/syslog.log with a "file viewer tool" like less, vi, more, cat etc. You are trying to execute the file it which is not permitted.

.

What happens when you manually restart the snmp agent via  /etc/init.d/snmpd restart?

[bomartin]

Okay, I can see at the syslogs that watchdog-snmp.PID does not exist. When I check my snmp config with esxcli system snmp get, I miss a "EngineID".

~ # esxcli system snmp get    (not working host)
   Authentication:
   Communities: COMMUNITY
   Enable: true
   Engineid:      (no PID!!!)

~ # esxcli system snmp get    (working host)
   Authentication:
   Communities: COMMUNITY
   Enable: true
   Engineid: 00000063000000a100000000   (PID!!!)

vmkernel.log:

2015-10-29T22:49:24.463Z cpu1:53692586)WARNING: MemSched: 15645: Group snmpd: Requested memory limit 0 KB insufficient to support effective reservation 2704 KB
2015-10-29T22:49:25.101Z cpu0:53692602)WARNING: MemSched: 15645: Group snmpd: Requested memory limit 0 KB insufficient to support effective reservation 2704 KB
2015-10-29T22:49:25.728Z cpu2:53692618)WARNING: MemSched: 15645: Group snmpd: Requested memory limit 0 KB insufficient to support effective reservation 2704 KB
2015-10-29T22:49:26.383Z cpu5:53692639)WARNING: MemSched: 15645: Group snmpd: Requested memory limit 0 KB insufficient to support effective reservation 2704 KB
2015-10-29T22:49:27.013Z cpu3:53692654)WARNING: MemSched: 15645: Group snmpd: Requested memory limit 0 KB insufficient to support effective reservation 2704 KB

syslog.log

2015-10-29T22:49:24Z watchdog-snmpd: PID file /var/run/vmware/watchdog-snmpd.PID does not exist
2015-10-29T22:49:24Z watchdog-snmpd: Unable to terminate watchdog: No running watchdog process for snmpd
2015-10-29T22:49:24Z root: snmpd Zero out resource pool request failed, try 5,
2015-10-29T22:49:25Z root: snmpd Zero out resource pool request failed, try 4,
2015-10-29T22:49:25Z root: snmpd Zero out resource pool request failed, try 3,
2015-10-29T22:49:26Z root: snmpd Zero out resource pool request failed, try 2,
2015-10-29T22:49:27Z root: snmpd Zero out resource pool request failed, try 1,

[MKguy]

1. Try to reset all configurations with esxcli system snmp set -r and reboot the host, this should assign a default EngineID (which is only relevant for SNMPv3 anyways)

2. After that configure SNMP again and restart the snmpd service.

3. Check and post the content of the file /etc/vmware/snmp.xml. It should reflect your SNMP settings.

From the logs it sounds like there is 0KB memory limit of the snmpd service. Check the snmpd process resource allocation here:

What does it look like in your case?

[acowles]

Has anyone else found a solution to this issue. I am currently experiencing the same thing, (ESXI 5.5 3116895) but noted a few more things.

I can disable SNMPD via the viclient, or using esxcfg-snmp tool, but in either case, when they show disabled, /etc/init.d/snmpd status shows the service as running.

~ # esxcli system snmp set --enable false

~ # /etc/init.d/snmpd status

snmpd is running

~ # ps -cs | grep -i snmp

35156 35156 snmpd                WAIT   LOCK   0-31 /sbin/snmpd

17437995 17437995 grep                 RUN    NONE   0-31 grep -i snmp

17424816 17424816 snmpd                WAIT   LOCK   0-31 snmpd

Trying to kill or pkill any of these processes runs the command with no results.

The memory allocation parameter is very similar to the one shown in the previous screenshot, except that snmpd doesn't expand down to a pid for mine, but has the same values as displayed for allocated resources.

The settings reflected in /etc/vmware/snmp.xml are correct, it just doesn't work, and we really aren't in a position to have this server rebooted right now.

Thanks in advance.

[amirsafayan]

Has ANYONE in the community actually resolved this issue ?  I'm up against the same issue.

Thanks,

Amir

[amirsafayan]

A host reboot resolved the issue.

Amir

[amirsafayan]

Can you mark this as answered so it helps other folks that are hitting the same issue?

Thanks,

Amir