I tried to test this out in my lab and I didnt see any latency or delay in retrieving the list of the available host / portgroup.
Can you please upload the vmkernel log ,vobd.log, hostd.log of the host where the VC sits
also upload the vpxd logs with the timestamp