Couple of things to check:
Have you enabled Forged Transmits on the port-group connecting the VyOS VM?
Have you increased the MTU settings to 1600 or jumbo frames in the underlay?
Have you tried to ping between the VM's on same logical switch within the same ESXi host?
Move the VM to where the vyata VM exist and ping and see if it works.
Thanks,
Dev