Today I have fixed failed process of NSX 6.4 Host Preparation on Nested vSAN ESXi hosts in my Lab. Software versions used NSX 6.4.0 Build 7564187 / vSphere 6.5.0.13000 Build 7515524.
I have originally stripped ESXi hosts to minimum possible configuration. For each nested ESXi installation I have used 5GB Disk. For vSAN datastore I have used one 7 GB Disk as Cache Tier and two 10GB disks for Capacity Tier per each nested ESXi host in 3 node vSAN Cluster. As far as vSAN was tested it worked perfectly fine.
Problem appeared when I tried to prepare same nested hosts for NSX. First I have successfully deployed NSX Manager and Controller. Then I initiated Host Preparation for NSX on each nested ESXi. It scanned nested ESXi hosts then proceeded with Install task but didnt proceed with Install agent Task on ESXi hosts (as you can see on picture).
NSX Host Preparation Install agent Task didnt proceed on nested ESXi hosts
In /var/log/esxupdate.log there was this message:
2018-07-25T07:25:35Z esxupdate: 79906: esxupdate: ERROR: vmware.esximage.Errors.LiveInstallationError: ([], "Error in running ['/etc/init.d/netcpad', 'start', 'install']:\nReturn code: 1\nOutput: ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored.\nFailed to set memory reservation for netcpa, retrying...
After some troubleshooting + googling for similar error messages I have created fresh new nested ESXi without vSAN. I have created bigger 40GB local disk to be sure there is enough space. And voilà suddenly magically NSX Host Preparation finished successfully.
Successful NSX Host Preparation on fresh new nested ESXi with proper size of local disk
Or maybe not enough memory of host, I have fixed the similar problem with memory addition.
[root@sa-esxi-02:~] esxcli software vib install -d /tmp/vxlan.zip --force
[LiveInstallationError]
Error in running ['/etc/init.d/vShield-Stateful-Firewall', 'start', 'install']:
Return code: 1
Output: vShield-Stateful-Firewall is not running
watchdog-dfwpktlogs: PID file /var/run/vmware/watchdog-dfwpktlogs.PID does not exist
watchdog-dfwpktlogs: Unable to terminate watchdog: No running watchdog process for dfwpktlogs
ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored.
Failed to release memory reservation for vsfwd
Resource pool 'host/vim/vmvisor/vsfwd' release failed. retrying..
Resource pool 'host/vim/vmvisor/vsfwd' release failed. retrying..
Resource pool 'host/vim/vmvisor/vsfwd' release failed. retrying..
Resource pool 'host/vim/vmvisor/vsfwd' release failed. retrying..
Resource pool 'host/vim/vmvisor/vsfwd' release failed. retrying..
Set memory minlimit for vsfwd to 512MB
ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored.
Failed to set memory reservation for vsfwd to 512MB
ERROR: ld.so: object '/lib/libMallocArenaFix.so' from LD_PRELOAD cannot be preloaded: ignored.
Failed to release memory reservation for vsfwd
Resource pool 'host/vim/vmvisor/vsfwd' released.
Resource pool creation failed. Not starting vShield-Stateful-Firewall
It is not safe to continue. Please reboot the host immediately to discard the unfinished update.
Please refer to the log file for more details.
I was in the same place when building POC in someones homelab.
As latelanzh confirmed its just due to low memory, I was using nested esxi with 6GB RAM, up'd first host to 8GB and it worked and tried another nested esxi with 7GB and that worked too.
Yep, adding more memory to my nested ESXi hosts did the trick as well!
I was also using nested environment and faced the same issue. It has rectified after increasing VM host memory from 4 GB to 8GB.
The error i have found during NSX-t 2.5.1 host preparation for ESXi 6.7 update3.
[root@esx2:~] esxcli software vib install -d /tmp/nsx-lcp-2.5.1.0.0.15314289-esx67.zip
[LiveInstallationError]
Error in running ['/etc/init.d/nsx-mpa', 'start', 'install']:
Return code: 1
Output: Unable to setmemconfig for mpa
host/vim/vmvisor/mpa
Resource pool 'host/vim/vmvisor/mpa' released.
Waiting for /usr/lib64/vmware/nsx-mpa/mpa to start...
Waiting for /usr/lib64/vmware/nsx-mpa/mpa to start...
Waiting for /usr/lib64/vmware/nsx-mpa/mpa to start...
Waiting for /usr/lib64/vmware/nsx-mpa/mpa to start...
Waiting for /usr/lib64/vmware/nsx-mpa/mpa to start...
Unable to start /usr/lib64/vmware/nsx-mpa/mpa
It is not safe to continue. Please reboot the host immediately to discard the unfinished update.
Post increasing memory from 4 GB to 8GB it has worked perfectly.
[root@esx2:~] esxcli software vib install -d /tmp/nsx-lcp-2.5.1.0.0.15314289-esx67.zip
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: VMware_bootbank_nsx-adf_2.5.1.0.0-6.7.15314402, VMware_bootbank_nsx-aggservice_2.5.1.0.0-6.7.15314423, VMware_bootbank_nsx-cli-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-common-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-context-mux_2.5.1.0.0esx67-15314456, VMware_bootbank_nsx-esx-datapath_2.5.1.0.0-6.7.15314311, VMware_bootbank_nsx-exporter_2.5.1.0.0-6.7.15314423, VMware_bootbank_nsx-host_2.5.1.0.0-6.7.15314289, VMware_bootbank_nsx-metrics-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-mpa_2.5.1.0.0-6.7.15314423, VMware_bootbank_nsx-nestdb-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-nestdb_2.5.1.0.0-6.7.15314393, VMware_bootbank_nsx-netcpa_2.5.1.0.0-6.7.15314440, VMware_bootbank_nsx-netopa_2.5.1.0.0-6.7.15314363, VMware_bootbank_nsx-opsagent_2.5.1.0.0-6.7.15314423, VMware_bootbank_nsx-platform-client_2.5.1.0.0-6.7.15314423, VMware_bootbank_nsx-profiling-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-proxy_2.5.1.0.0-6.7.15314435, VMware_bootbank_nsx-python-gevent_1.1.0-9273114, VMware_bootbank_nsx-python-greenlet_0.4.9-12819723, VMware_bootbank_nsx-python-logging_2.5.1.0.0-6.7.15314402, VMware_bootbank_nsx-python-protobuf_2.6.1-12818951, VMware_bootbank_nsx-rpc-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-sfhc_2.5.1.0.0-6.7.15314423, VMware_bootbank_nsx-shared-libs_2.5.1.0.0-6.7.15036308, VMware_bootbank_nsx-upm-libs_2.5.1.0.0-6.7.15314375, VMware_bootbank_nsx-vdpi_2.5.1.0.0-6.7.15314422, VMware_bootbank_nsxcli_2.5.1.0.0-6.7.15314296, VMware_bootbank_vsipfwlib_2.5.1.0.0-6.7.15314343
VIBs Removed:
VIBs Skipped:
[root@esx2:~]
Thanks
Deepak