kmcd03's Posts

Thank you for documenting this. I was leaning towards creating a new vlan TZ.  And applying to the nodes using the Transport Node Profile that VCF created.  Is  attaching, or detaching, the TNP to a Baseline WLD disruptive?  I assume in this case only adding additional TZ to the nodes, so it won't take VMs or nodes offline.

For the isolation address, I referenced Duncan Epping's blog (vSphere HA heartbeat datastores, the isolation address and vSAN | Yellow Bricks)  We created a Switch Virtual Interface (SVI) on the physical switches.  With the IP in same subnet as vSAN and one for each site.  Then configured the advanced option setting das.isolationaddress0.

If only the Witness host loses connectivity to both sites, VMs will stay online. Last week our 14+1 stretched cluster (between two data centers) lost connectivity to third data center where the witness was located. (redundant network to all sites, but outage was caused by firewall misconfig)  There was no affect to the VMs at either data center (preferred and secondary fault domain). vSAN health checks alerted on multiple errors, like connection to the Witness host and network partition has occurred. But no affect to guest VMs. However when connectivity was restored several hours later, the witness would not rejoin the cluster.  We also have two 2-node clusters with witness at third site that wouldn't re-connect to their Witnesses either.  I confirmed that could ping between vSAN hosts and witness across the appropriate interface (vmk) for all clusters. I opened ticket with GSS and only solution was to disable the stretch configuration.  Then creating the stretch configuration again, putting the hosts in the correct fault domain and choosing the Witness host.  Once the Witness was back online and participating in the cluster could see in the health check that objects were rebuilding on the Witness.

We're experiencing this same problem, but with fc640 blades using and FD332 storage sleds with FD332-PERC (dual NOC) controller.  In last two weeks I've had vSAN mark SSDs permanently disabled (PDL) on two different hosts.  I had same problem eight months ago (May). I had three different hosts have same problem and I opened tickets with GSS and Dell support. Recommendation was to update the controller firmware from version 25.5.5.0004 to 25.5.5.0005.  And the lsi_mr3 driver from version 7.703.18.00 to 7.703.20.  Both version combinations of firmware and driver are on the VCG for vSAN. I have new tickets opened both with GSS and Dell Support for over ten days now.  I was asked to upgrade firmware to 25.5.6.0009 that was released in Sept-2019.  I think there continues to be a problem with the H730 family of controllers and vSAN 6.x.  And replacing the current controllers with HBA330 cards is the fix. How much effort is needed to replace/change controllers with vSAN (configured for encryption and de-dupe)?  Is it as simple as putting host in maintenance mode with option ensure accessibility then replace controller?  Will vSAN see the SSDs and diskgroups as unchanged?  The controller is in pass-through mode so wondering as long as the drivers load, there's no change to the SSDs, disk IDs/signatures, and diskgroups. (14+1 stretched cluster so capacity for applied storage policies) Or do I have to evacuate all stored components from the host and delete the diskgroups, and then swap controller and recreate the diskgroups?

Follow up if anyone else encounters similar problem of Affinity rule for Preferred/Secondary in a storage policy not following the Preferred fault domain designation in a stretched cluster.  With help of GSS was able to identify a duplicate entry for preferred fault domain in CMMDS.  Object was removed and Affinity now aligns with the cluster setting for Preferred/Secondary.  In my case the duplicate entry referenced a non-existent witness host that was used for initial, but failed configuration of stretched cluster.  The witness was successful in communicating to ESXi hosts at layer-3, but ESXi hosts failed to communicate to witness because of asymmetric route.  The routing problem was corrected and stretch configured with new witness. I suspect removing the witness used in first attempt did not successfully remove object from CMMDS so Affinity in Storage Policy was detached from the cluster designation for Preferred.

Sorry, I'm probably over describing the problem. The Affinity setting in the Storage Policy for Preferred/Secondary fault domain is diametric from the designated Preferred fault domain setting configured in the clusters Configure | "Fault Domain & Stretched Cluster" section of vCenter Server Web Client. I could leave it as is and accept that the site in the Affinity rule is just the opposite.  But my concern is the risk that a future change, like a patch or upgrade, corrects this problem and the VMs objects are moved unintentionally. I'm guessing I will have to disable the stretched cluster setting and re-configure to designate the fault domain I want to be preferred. The goal was to have the ability to pin VMs to specific sites using Storage Policies and DRS VM-to-Host rules.  There would be a storage policy with PFTT=0 and Affinity=Preferred for Site-A.  And a second storage policy with PFTT=0 and Affinity=Secondary for Site-B.  The storage policies would allow me to keep VM (all objects) local to a site.  And the DRS rules would pin the VMs to hosts at a site. I will be migrating existing production VMs into this new vSAN cluster.  These VMs do not have a stretched layer-3 networking, so must be pinned to specific site.  And there are Test/Dev VMs that do not require replication between sites.  Next phase will be to deploy NSX to stretch the network for VMs.  Can then fully take advantage of a storage policy with PFTT=1 (and Affinity not relevant). Eventually there is a use case in our environment for wanting to change the Preferred fault domain designation.  We are required by vendor to demonstrate DR every six months.  Having the workloads move between sites will satisfy that requirement.  Being able to change the Preferred designation would help us balance the workload across sites and still maintain continuity in a network outage. Thanks.

Thanks for reply.  Unfortunately workaround didn't fix the problem.  I changed storage policy by adding IOPS limit = 0 and did an update.  But the VM objects location does not move/change. If I change the Affinity Preferred/Secondary back and forth, the objects move between hosts in the fault domain.  But the Preferred/Secondary designation in the Storage Policy Affinity is opposite than the setting observed in vCenter. I've also ran the command  "esxcli vsan cluster preferredfaultdomain get" on the hosts and witness, and all reporting preferred fault domain is same as vCenter Server Web Client. This is a new environment with just few test and non-prod VMs running, so do have an opportunity to try things.  I've been doing some failure testing of this new cluster prior to moving legacy production VMs.  We simulated a cut on the 10Gb connection between data centers used by vSAN.  This did work previously, but last week changed the Preferred Fault Domain designation and discovered this problem. Thanks!

We have a pair of switches for data node/ESXi host management and VM traffic.  A second pair of switches were added for vSAN traffic to the ESXi hosts.  A 10 GB circuit between the data centers is also on this second stack.  The data nodes are using L2 for vSAN.  We're using a /23 for vSAN vmk interfaces with the bottom /24 at one DC and top /24 at other.  An route was configured for a switch at primary data center.  Hosts at one site could connect to the witness, but witness couldn't connect to hosts.  We had to create a route for the other data center and also add static host routes (/32) for each host for traffic to work. I opened ticket with GSS to confirm if on the Witness host could consolidate Management and Witness traffic on to vmk0.  The Witness added with no errors and also passed health checks.  However discovered unintended side effect is access to KMS (coincidentally same site and vlan/IP subnet as the Witness) broke.  It looks like the traffic for the encryption KMS is traversing the vSAN vmk of the data hosts.