Hi,
We have 60 nfs datastore and couple of vms have over 20TB,60TB vmdk on the nfs datastore in our platform, sometimes some other vms cannot boot or restart on some esxi host, Vmware and vmkernel logs are clearly state that, the virtual machine cannot access to its nfs harddisk, because of the esxi host hit te max heap size.
Aalso virtual machine nfs harddisk shows 0 byte under vcenter summary tab, and from the esx cli you see 0 files listed inside any nfs datastore. Remounting nfs datastore return "vm file system cannot open volume"
Here is what is dumping in the vmware.log while push the start button and got the error message on vcenter.
2020-05-04T16:15:28.910Z| vmx| I125+ Power on failure messages: File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ File system specific implementation of GetObject[fs] failed
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ Heap nfsclient already at its maximum size of 295973752. Cannot expand.
2020-05-04T16:15:28.910Z| vmx| I125+ Cannot allocate memory
2020-05-04T16:15:28.910Z| vmx| I125+ Cannot open the disk '/vmfs/volumes/6c10c2fe-964d5016/vmname-abc/vmname-abc.vmdk' or one of the snapshot disks it depends on.
2020-05-04T16:15:28.910Z| vmx| I125+ Module 'Disk' power on failed.
2020-05-04T16:15:28.910Z| vmx| I125+ Failed to start the virtual machine.
We already made sure it is not something relate one of these;
Nothing related to this 'one of the snapshot disks it depends on...'
Changing some advanced values need reboot, like tcpheap, sunrpc etc.
Restarting nfs service, or full service restart doesnt help.
Seperate vmkernel adapter? not sure more likely it depends to nfs service.
We have not noticed performance issue in our nfs storage device, at least now.
Thanks a lot for any advice.
