Good day all,
I am trying to max out the performance of the NAS but at the same time I have to be very cautious as I have to deploy ESXI in 3 datacenters and the links between them are just 1Gb. All that while keeping HA happy too! Ideally I would like to use the directly connected NASes on 10Gb SFP+ ports for the VMs on their datacenter unless something happens and I need to migrate the vms to another host temporarily. Then the NASes would be also connected to the other esxi hosts using the 1Gb link that they'll use while the catastrophe is being sorted.
The 3 esxi (essentials plus 6.5) hosts have 4x 1Gb RJ45 NIC and 1 SFP+ 10Gb card, 2 NICs are for management and the other 2 for the VMs while the SFP+ card is directly connected to their respective NAS. The NASes have 2x 1Gb RJ45 NIC and 1 SFP+ 10Gb, the 2 NICs are for management (barely used for that really) and NFS for share storage and the SFP+ are directly connected to the esxi hosts.
The connection is as follows, NAS1 10Gb SFP+ port ----> ESXI1 10Gb port (on dedicated subnet no switching no nothing just P2P)
NAS2 10Gb SFP+ port ----> ESXI2 10Gb port
NAS3 10Gb SFP+ port ----> ESXI3 10Gb port
Also the NASes are connected to all the other ESXI hosts using the server subnet NICs (2x 1Gb) for HA.
Taking this IP Address table example:
Server Network Peer 2 Peer Network
ESXI1 192.168.40.20 10.0.0.1 ---> 10.0.0.4
ESXI2 192.168.40.21 10.0.0.2 ---> 10.0.0.5
ESXI3 192.168.40.22 10.0.0.3 ---> 10.0.0.6
NAS1 192.168.40.23 10.0.0.4 ---> 10.0.0.1
NAS2 192.168.40.24 10.0.0.5 ---> 10.0.0.2
NAS3 192.168.40.25 10.0.0.6 ---> 10.0.0.3
So, with the connections between ESXI1 and ESXI2 would be as follows
ESXI1 to NAS1 dest. 10.0.0.4
ESXI1 to NAS2 dest. 192.168.40.24
ESXI1 to NAS3 dest. 192.168.40.25
ESXI2 to NAS1 dest. 192.168.40.23
ESXI2 to NAS2 dest. 10.0.0.5
ESXI2 to NAS3 dest 192.168.40.25
You get the picture for esxi 3 I assume.
The problem, when adding the storages, esxi seems to get the UID based on IP address. If I try to mount datastore to additional hosts it fails like a pro as it is trying to connect it to the other host using the unroutable subnet. Then If I try to add the datastore using their "public ip" using the same name, it complains that the datastore with different backing up system is already added.
So I end up with loads of datastores and the "on-the-fly" migration to another server doesn't work. Migrating them works but of course it takes time. As I populate the hosts with VMs it will just take more and more time.
I am stuck here, don't know what to do and budget is extremely tight. vSan is out of the question (hosts have 2 cpus each). Isn't there a command to specify the UID or tell esxi to bridge the internal 10.0.0.x network or some sort of static forwarding or natting? I can't think of anything but there are many ways to fool the system but then again... That can break it too.
Open to suggestions,
Thanks
PS: quick network topology for you amusement:
