Afaik there are no cmdlets nor APIs that allow you to change the complexity rules.
The bypass would be to logon to the "support-only" console of the ESXi and edit the /etc/pam.d/common-password file.
A valid, in my opinion, alternative would be to go for AD integration.
That way you set the password complexity in AD.
____________
Blog: LucD notes
Twitter: lucd22