Port 902 not listening on TCP

[berkshim]

We recently moved to VM 6.0 (vCenter on 3018524) and I am currently having issues with backing up all of my vm servers.  We use CommVault (with whom I opened a support ticket) and they identified that the software could not connect on port 902.  After much troubleshooting, thinking that the firewalls were the issue, but were not as we killed off all firewalls on the affected devices with no change....we noticed that the VC was not listening on port TCP 902....it is listening on UDP 902 though.  I've spent a few hours combing through the internet trying to find a decent solution....but unable to find one.  Does anyone out here have any ideas on why this might be happening?

[MKguy]

I don't think this is the cause of your issues. We also use CommVault and I checked my 5.5 vCenters, they are only listening on 902/UDP as well. It's well known that port 902/TCP is needed on the ESX(i) hosts, but it seems that's not the case for vCenter, at least since 5.x versions.

VMware KB:    TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts,...

The Firewall KB article is a bit ambiguous. In the list they mention TCP/UDP in the protocol column, but the purpose description implies it only uses UDP:

Product     Port     Protocol          Source      Target                  Purpose

ESXi 5.x     902     TCP/UDP     ESXi 5.x     vCenter Server     (UDP) Status update (heartbeat) connection from ESXi to vCenter Server

There is also this statement at another section that refers to the well known connection from vCenter to hosts on port 902, it also mentions only a UDP connection to vCenter the other way around:

Product          Port      Protocol       Source                Target                  Purpose

vCenter 6.0     902     TCP/UDP     vCenter Server     ESXi 5.x   

The default port that the vCenter Server system uses to send data to managed hosts. Managed hosts also send a regular heartbeat over UDP port 902 to the vCenter Server system.

This port must not be blocked by firewalls between the server and the hosts or between hosts.

Port 902 must not be blocked between the vSphere Client and the hosts. The vSphere Client uses this port to display virtual machine consoles.

Because of this I am fairly sure you need to look elsewhere for your issue, perhaps you could describe it in more detail?

[berkshim]

Interesting.  According to CommVault Tech Support as of yesterday TCP 902 is a manditory / must have port open.  Even says it in the logs....

Well....the error that CommVault sends in the email is: Failure Reason: Failed to backup all the virtual machines. Please check event viewer for individual virtual machine failure message. 

The Job, when you go look at it in the event details it gives:  Unable to open the disk(s) for virtual machine [xxxxxx].  Please ensure the following: 1) the proxy is able to communicate with the ESX host and resolve the ESX host address  2) the correct transport mode has been selected  3) the disk types configured to the virtual machine are supported.

Contacting CommVault support and looking in the detailed logs, they show that our VC is Actively Refusing connections over TCP 902:

From CommVault Support Case Notes:

-Reviewed VSBKP and VIXDISKLIB Logs. We were seeing Failed to open disk error messages for the operation.
-Noting in VIXDISKLIB, there was NBD_ERR_CONNECT error messages.
*Via CVPING, checked out to VCenter connection over port 902, connection noted was Actively Refused.

[MKguy]

they show that our VC is Actively Refusing connections over TCP 902

- Reviewed VSBKP and VIXDISKLIB Logs. We were seeing Failed to open disk error messages for the operation.

- Noting in VIXDISKLIB, there was NBD_ERR_CONNECT error messages.
* Via CVPING, checked out to VCenter connection over port 902, connection noted was Actively Refused.

I don't think that last point is an actual log message during the backup process. It looks more like the guy arbitrarily tried that cvping utility (see Client Connectivity) against vCenter, when it should be run against hosts.  The real error statement before does not mention the destination host.

Use wireshark/tcpdump or some other packet sniffing tool on your vCenter or backup server when a backup runs and filter for traffic on port 902. This will tell you where the backup server actually tries to connect, or if such a packet actually arrives at the vCenter.

Have you tried to connect to your ESXi hosts on port 902 from your backup server? You can just use the telnet utility on Windows for example (or try that cvping tool but I don't know how trustworthy it is):

telnet myesxiserver.local 902

If you get a blank prompt session and/or the ESXi banner message like "220 VMware Authentication Daemon [...]" then the connection between your backup server and ESXi hosts on port 902 is fine.

[berkshim · Accepted solution]

OK....well...finally got a solution.  This is actually a multi-part problem.  First off, the CommVault folks sent me on a merry chase down a wrong path.  What they said was that I HAD to have TCP 902 open on the Virtual Center.....but instead I needed to have TCP 902 open on the hosts.  Sure enough....once that was identified, we saw that 902 was in fact not open on the hosts for that cluster.  Which led us down the path of realizing that there was a mis-configuration on the Distributed Virtual Switches on that cluster.  Once that was corrected, everything started working properly. 

[winadminSZ]

Hi berkshim,

We have the same problem, since moved to vCenter 6.0: can you explain, how you fixed that Problem in the vswitch.?

What was the mis-configuration on the distrivuted Virtual Switches ?

Tahnk you very much for a Feedback.

Kind regards from switzerland....

winadminSZ

[berkshim]

Sure....the root issue is that had to reconfigure our VMotion settings to get the ability to migrate VMs from one datacenter to another datacenter (new feature in version 6).  When we reconfigured the vmotion IPs, we used the same IP scheme in our 1st Virtual switch that was being used in the other datacenter.  That way, as they are both in the same IP range, the VMs could vmotion between datacenters.  Well....our issue was that the vlan we changed the vmotion to in the first Distributed Virtual Switch (DvS), was already in use in the second DvS on the same cluster.  So...we created a loop inside the one datacenter between our two DvS's.....yes...our vmotions were also failing between datacenters...imagine that.  We disabled the vmotion in the 1st DvS and just configured vmotion to work on the 2nd DvS on the proper vlan and everything just started working!  Hopefully this makes sense...if you need further clarification, be glad to help out! 

Good Luck from the Hoosier Heartland of Indiana!

Mike