I have searched some topics about how to rename a ESXi display name (which it is "IP" by default) to a custom name! but all guides were pointing to remove the host from inventory, then add a FQDN on esxi, make A record, and finally add ESXi using FQDN to vCenter, then it still will show as FQDN.
So, is there any way to put the display name as a custom name, not FQDN/IP?
No there is not. If you add the host to vCenter by IP address then it will show the IP address in vCenter. If you add the host by FQDN and that name can resolve via DNS then it will show the FQDN.
The only way to have 'custom' names would be to create an A record in DNS (such as FOO) that resolves to an IP address then you can add the host as FOO. But there is no way just to change the display name in vCenter.