You need a licensed version of ESXi (even Essentials will do) in order to be able to take image-based backups of the VMs riding on it. That coupled with ESXi configuration backups give you the ability to restore the config of a host and its VMs. Secondly, you need some halfway decent backup software that has scheduling abilities. Those backups should be stored external to this host.
If you want replication (which is not the same as backup), you'd need a second ESXi host to serve as the replica host. VMs from host A would be replicated to host B.
There is obviously more, but this is some starter information.
There are several option for available.
The currently most popular one might be the Veeam Backup & Replication Community Edition, which you can use to either do a VM based backup (that however requires at least an Essentials license for the ESXi host), or an agent based backup (treating the VM like a physical server). It may even be possible to just install the Veeam Agent on the server to do the backup, which however provides less options.
As you mentioned, backing up the ESXi host itself with a simple configuration may result in more work than to reinstall it, if it ever breaks.