Mounting NFS Datastore - Mounting with random port numbers?

[zstar69]

I am mounting an NFS datastore to our QNAP. We opened 111/tcp, and 2049/tcp in the firewalls. When trying to mount, the firewall logs show:

Block from $ESXI to $QNAP on port 47110

I can understand the qnap sending back to the source at 47110, but it's sending to the NFS server on that port. On another hypervisor it's sending 42445.

So we've been adding the NFS datastore to all of our hypervisors, watching the firewall logs, waiting for it to get blocked to reveal the randomly generated port number above 40,000 and then adding that port to the firewall, then adding the datastore for the 2nd time.

Anyone else dealing with this? What is this port, and where is it coming from? Our firewall is becoming messy with all these ephemeral ports set as destinations..

Thanks.

[Rubeck]

Hi zstar69...

I can't really answer your question, but what you see would definitely explain the ESXi firewall ruleset:

~ #   esxcli network firewall ruleset rule list

Ruleset             Direction  Protocol  Port Type  Port Begin  Port End

------------------  ---------  --------  ---------  ----------  --------

sshServer           Inbound    TCP       Dst                22        22

sshClient           Outbound   TCP       Dst                22        22

nfsClient           Outbound   TCP       Dst                 0     65535

The reason why only ports 2049 and 111 are documented in the VMware KB: TCP and UDP Ports required to access vCenter Server, ESXi/ESX hosts, and other network co... article is then another question... 

And what's up the ports also using UDP..? Can't seem to find them in the ESXi rule sets.. hmmm..

/Rubeck

[zstar69]

Yes the firewall is allowing any port outbound on TCP. But why is the system establishing the connection with the QNAP on some random port? Why is it not contacting it initially on 2049? or 111?

If ESXI uses any port it pleases to make the initial connection, how will we ever figure out what the rules should be on the firewall until we try once and watch the packet get blocked? Right now it's kind of a guessing game.

Our procedure right now is to tail the log files for blocked packets, then add the QNAP to the datastore. OH LOOK, the ESXI server was blocked going to the QNAP on 53888 this time, shucks. Add rule to firewall for ESXI -> Qnap on Port 53888 along with 2049 and 111. All works now.

Does this sound right to anybody? It seems wrong to me. I should only need to open 2049 and 111. End of story. Why is ESXI the only system that acts this way? All other systems only need 2049 and 111 open for NFS mounting. ESXI starts off by connecting to the server on some random port....?

This is an example of our firewall and how it's becoming messy:

ESXI 1 rule

Allow from ESXI1 to QNAP on port 47110,111,2049

ESXI 2 rule

Allow from ESXI2 to QNAP on port 45991,111,2049

ESXI3 rule

Allow from ESXI3 to QNAP on port 53881,111,2049

ESXI4 rule

Allow from ESXI4 to QNAP on port 58283,111,2049

Funny, if I were to add a 2ND mount to the same QNAP on ESXI4, it would use again ANOTHER port. The rule would need to look like this

Allow from ESXI4 to QNAP on port 49223,58283,111,2049

This issue is actually quite a joke. I just don't get it.

Also, when I say firewall, I am talking about the firewalls that lie in between the ESXI servers and QNAPS. Not the buuilt in firewall. The middle firewalls are blocking since they have specific ports set (2049,111/tcp)

[dariusd · Accepted solution]

The portmapper (on tcp/111 and udp/111) does exactly that: It allows the server to allocate somewhat-arbitrary TCP and UDP ports (i.e. not well-known port numbers) to provide certain RPC (Remote Procedure Call) services, and provides a lookup service so that RPC clients can determine how to connect to each service – see man 8 portmap for more information.  It's up to the RPC server (in this case, your QNAP) to determine which ports should be used for each service.

In this case, ESXi is contacting the portmapper service on your QNAP's udp/111 or tcp/111 and requesting some RPC service.  The QNAP is responding to each client with a message saying "hey, sure, you can get to that service by connecting back to me on tcp/42445" (or whatever), so that is what your ESXi box does.  You may be able to use tools like rpcinfo to find out which proto/port on the QNAP is servicing each RPC program/version.  That might help with understanding the nature and origin of these connections...

Cheers,

--

Darius

[zstar69]

Aha, so the initial connection is not the random one, but after the connection on port 111 is established, is the random. Okay this makes a bit more sense I suppose, still not sure why ESXI is the only system thats dealing with this. But I will check out the things you've mentioned and I appreciate your help!

Edit:

That was it. Very weird. I wonder if my other servers are using NFSv4 where ESXI is still using V3. Either way I don't really care since now I have statically set it using "rpc.mountd -p 44444" and now I will just need to open 44444,111 and 2049 from now on.

Thanks for helping me get to the bottom of this.

+1