ESXi host too many open files (using as backup proxy)
Our ESXi server version is:
VMware ESXi 6.5.0 build-5969303
VMware ESXi 6.5.0 Update 1
We have an ESXi host dedicated to handling backup requests for VMs from our Commvault environment. Recently our backup admin changed some backup schedules. Now the ESXi host is losing connection to vCenter a couple times a night and during the time that this happens many log entries in vmware.log similar to this.
<163>2019-02-14T04:18:00.804963-08:00 esxihost1.company.com Vpxa: error vpxa[A1D4B70] [Originator@6876 sub=vpxNfcClient opID=37aa37a2-a8] [VpxNfcClient] Unable to get info for local file ds:///vmfs/volumes/f06fbf9f-0ca4b79f/VM1/VM1.vmx
<164>2019-02-14T04:18:00.804879-08:00 esxihost1.company.com Vpxa: warning vpxa[A1D4B70] [Originator@6876 sub=Libs opID=37aa37a2-a8] [NFC ERROR] NfcFile_GetInfo: Failed to open file
From looking online it seems like we are running out of file descriptors. Does that sound right? Using "ulimit -n" we are at the default of 256.
Can we raise this limit safely? If so how would we do this? Any input and recommendations would be wonderful.
I have asked the backup admin to lower the number of backups running concurrently which they have done. However we are hoping to increase this again since the ESXi host has plenty of CPU/RAM/Network throughput.