VMware Cloud Community
AnonAdmin
Enthusiast
Enthusiast
‎02-24-2021 06:54 AM

NFC connections

While attempting to upgrade some hosts from ESXi 6.7 to 7.0 recently I noticed that several of them got hung up while entering maintenance mode. While stuck, the status was "Waiting for active nfc sessions to terminate" and in some cases, this 'entering maintenance mode' Task eventually timed out around 60 minutes later and the upgrade job would fail.

I've seen references to NFC connections  in my google search from the Veeam backup product however we use Commvault in our environment. Nevertheless, I wonder if something changed in our backup environment that's now causing this issue.

I also attempted to restart the management services on these affected hosts with mixed results. 

Is there any particular log(s) that I can review to validate our backups are causing this issue? And is there a setting that I can configure to tune these NFC connections? We're running version 7.0 of the vCenter VCSA appliance. 

0 Kudos
9 Replies
Alpha_RUN
VMware Employee
VMware Employee
‎03-13-2021 09:07 PM

Restart the VPXA ,  This is resolved in subsequent releases 

Lalegre
Virtuoso
Virtuoso
‎03-14-2021 04:50 AM

Try what @Alpha_RUN mention here but just for clarification, NFC is used as protocol for copying data from the ESXi to the Backup solution when they work in Network mode because they leverage the connectivity to the Management VMkernel over IP.

PS: In Veeam is called network mode, I am not sure how is it called in Commvault.

0 Kudos
Optum
Contributor
Contributor
‎05-08-2021 01:08 AM

Tho restarting the vpxa service does clear the NFC session issue, it also causes the upgrade job to fail in SDDC. Is there an alternate way to kill the sessions?

0 Kudos
Divy2478
Contributor
Contributor
‎09-21-2022 02:07 AM

I faced the same issue while trying to put esxi hosts v6.7  in my prod environment into maintenance mode,it was stick at 100 % with waiting to "Waiting for active nfc sessions to terminate" .

 

I tried restarting the managements agents :

/etc/init.d/hostd restart

/etc/init.d/vpxa restart 

 

which with one host disconnected the host from vCenter and I had to maually connect it to the vCenter and with other host was able to put it into maintenance mode using the vim-cmd command.

 

vim-cmd /hostsvc/maintennace_mode_enter

 

Emaevans345
Contributor
Contributor
‎05-08-2023 03:14 AM

Hello there,

It sounds like you are experiencing some issues with upgrading your hosts from ESXi 6.7 to 7.0, and encountering "Waiting for active nfc sessions to terminate" messages when trying to enter maintenance mode. This can certainly be frustrating, especially when you're trying to complete an upgrade.

NFC (Network File Copy) is used by VMware to transfer virtual disks and other files between hosts and storage devices. It's possible that your backup environment is using NFC to transfer backup data, which could be causing the issue you're seeing. It's worth investigating whether there have been any changes to your backup environment that could be causing this.

To troubleshoot this issue, you can review the ESXi logs to see if there are any errors or warnings related to NFC or your backup environment. You can access the logs by logging in to the ESXi host's console or SSH session and navigating to /var/log. You can also check the vCenter Server logs for any related errors or warnings.

As for tuning NFC connections, there are a few settings that you can adjust to optimize performance. One is the "Net.TcpipHeapSize" parameter, which controls the amount of memory allocated to TCP/IP networking. Increasing this value can improve performance, but be careful not to set it too high as it can cause memory issues. You can also adjust the "Net.VmkLinuxSwapActivitionThreshold" parameter, which controls when ESXi will start swapping memory to disk. Setting this value too low can cause performance issues.

 

Also, checkout these resources.

https://communities.vmware.com/t5/Converter-Standalone-Discussions/NFC-connection-fault-during-a-blo... download

https://communities.vmware.com/t5/VMware-vSphere-Discussions/MAX-NFC-connection-buffers-for-network-...

https://communities.vmware.com/t5/vSphere-Storage-Discussions/ESXI-4-1-Datastore-browser-NFC-error/m...

 

I hope this helps you troubleshoot your issue and get your hosts upgraded successfully. Good luck!

0 Kudos
degvm
Enthusiast
Enthusiast
‎05-08-2023 08:09 AM

During the upgrade process, go to the vmkernel adapters of the hosts, search for NFC in the "Enabled Services", might be similar in 6.7 and then uncheck  "vSphere Replication NFC". I think vSphere Backup NFC is used in esxi 7.x, not available in 6.x

degvm_0-1683558322702.png

degvm_1-1683558329318.png

 

0 Kudos
virtualColin
Contributor
Contributor
‎06-23-2023 04:46 PM

THANK YOU

0 Kudos
chapri
Contributor
Contributor
‎08-27-2023 11:55 PM

 

Greetings,

It appears that you are encountering difficulties while upgrading your hosts from ESXi 6.7 to 7.0. Specifically, you are receiving "Waiting for active nfc sessions to terminate" messages when attempting to enter maintenance mode. Undoubtedly, this can be quite frustrating, especially when striving to complete an upgrade successfully.

VMware employs NFC (Network File Copy) to facilitate the transfer of virtual disks and other files between hosts and storage devices. It is plausible that your backup environment utilizes NFC for transferring backup data, which could be the root cause of the issue you are currently facing. It would be prudent to investigate whether any modifications have been made to your backup environment that could be attributing to this problem.

To address this matter, you can delve into the ESXi logs to identify any errors or warnings associated with NFC or your backup environment. These logs can be accessed by logging in to the ESXi host's console or SSH session and navigating to the /var/log directory. Additionally, it would be beneficial to inspect the vCenter Server logs for any relevant errors or warnings.

Regarding optimizing NFC connections, there are a few settings that can be adjusted to enhance performance. One such parameter is "Net.TcpipHeapSize," which governs the allocation of memory to TCP/IP networking. Increasing this value can potentially improve performance, but it is crucial to exercise caution and avoid setting it too high, as it may result in memory issues. Another parameter that can be adjusted is "Net.VmkLinuxSwapActivitionThreshold," which determines when ESXi will commence swapping memory to disk. Setting this value too low can lead to performance problems.

Pleae Check out these useful resurces: 

https://communities.vmware.com/t5/vSphere-Storage-Discussions/ESXI-4-1-Datastore-browser-NFC-error/m...

https://communities.vmware.com/t5/VMware-vSphere-Discussions/MAX-NFC-connection-buffers-for-network-...

https://communities.vmware.com/t5/Converter-Standalone-Discussions/NFC-connection-fault-during-a-blo... Download

This post will help you to solving your problem.

 

0 Kudos