vSAN1

Hi!

Upgrade VC and all ESXi. vSAN Network Mode no change to unicast. New clusters are also created in multicast mode.

Please tell me how to fix it?

Addition.

Use RVC

> vsan.ondisk_upgrade ~vsan --allow-reduced-redundancy

+------------------------+-----------+-------------+----------------+----------------+------------------+----------------+----------------+

| Host                   | State     | ESX version | v1 Disk groups | v2 Disk groups | v2.5 Disk groups | v3 Disk groups | v5 Disk groups |

+------------------------+-----------+-------------+----------------+----------------+------------------+----------------+----------------+

| r1.local | connected | 6.5.0       | 0              | 0              | 0                | 0              | 1              |

| r2.local  | connected | 6.5.0       | 0              | 0              | 0                | 0              | 1              |

| r3.local  | connected | 6.5.0       | 0              | 0              | 0                | 0              | 1              |

| r4.local   | connected | 6.5.0       | 0              | 0              | 0                | 0              | 1              |

| r5.local   | connected | 6.5.0       | 0              | 0              | 0                | 0              | 1              |

| r6.local   | connected | 6.5.0       | 0              | 0              | 0                | 0              | 1              |

+------------------------+-----------+-------------+----------------+----------------+------------------+----------------+----------------+

2017-05-26 08:59:25 +0000: Running precondition checks ...

2017-05-26 08:59:29 +0000: Passed precondition checks

2017-05-26 08:59:29 +0000:

2017-05-26 08:59:29 +0000: Target file system version: v5

2017-05-26 08:59:29 +0000: Disk mapping decommission mode: ensureObjectAccessibility

2017-05-26 08:59:51 +0000: Check cluster status for disk format conversion.

2017-05-26 08:59:51 +0000: Upgrade tool stopped due to error, please address reported issue and re-run the tool again to finish upgrade.

Hello,

On-disk format has nothing to do with network connectivity mode (also, your disks are all v5 which is the highest version at present).

Do you see Unicast Mode enabled when you run this via SSH on all hosts?:

#esxcli vsan cluster get

Please provide the build number of your vCenter and your hosts.

You should be able to set Unicast mode manually by following the steps I have outlined in my comments here:

https://communities.vmware.com/thread/563705

(though no guarantees this will be persistent across vCenter reboots so test this)

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

All host

esxcli vsan cluster get

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-26T10:51:42Z

   Local Node UUID: 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Local Node Type: NORMAL

   Local Node State: AGENT

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

Current version details  vCSA

From your previous snapshot, it does appear that the mode was changed to unicast

Unicast Mode Enabled: true

You can run the following to make sure each host sees its neighbors.

esxcli vsan cluster unicastagent list

Hello Dave,

Yes I PM'ed CaptainEmilyJones​ with the steps for checking unicast agent list and adding nodes if necessary.

My last comment also links to a more expansive explanation of these steps.

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

Yes, hosts see each other. However, in the General settings of the cluster, the network mode is multicast, whereas in the documentation it is written that it should be replaced by a unicast.

(All hosts) esxcli vsan cluster unicastagent list

NodeUuid                              IsWitness  Supports Unicast  IP Address       Port  Iface Name

------------------------------------  ---------  ----------------  --------------  -----  ----------

4312e1d8-9b60-8fa8-cb63-70e2840d9e88          0              true  192.168.99.213  12321

5745b6b9-d5d2-73d7-2bfd-70e2840da18a          0              true  192.168.99.214  12321

574da762-529d-0413-2107-70e2840d9e04          0              true  192.168.99.216  12321

57cfd45c-794d-9b42-6528-70e2840d9e90          0              true  192.168.99.217  12321

57d04aff-5c84-aa0e-6548-70e2840e0f20          0              true  192.168.99.218  12321

57a76c06-0a38-c90c-85bf-70e2840d9cc6          0              true  192.168.99.219  12321

Hello,

Can you check on ALL nodes that 'esxcli vsan cluster get' returns 'Unicast Mode Enabled: true' ?

You said 'after reboot' the agent lists were blank - Is this after rebooting vCenter or the hosts?

I ask this as it is possible that the vCenter may be unaware that Unicast should be used and thus pushes a blank list.

Which did you upgrade first - the hosts or the vCenter?

Bob

-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-

All hosts.

r1

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-30T05:15:00Z

   Local Node UUID: 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Local Node Type: NORMAL

   Local Node State: AGENT

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

r2

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-30T05:17:36Z

   Local Node UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Local Node Type: NORMAL

   Local Node State: BACKUP

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

r3

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-30T05:18:17Z

   Local Node UUID: 57cfd45c-794d-9b42-6528-70e2840d9e90

   Local Node Type: NORMAL

   Local Node State: AGENT

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

r4

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-30T05:19:01Z

   Local Node UUID: 574da762-529d-0413-2107-70e2840d9e04

   Local Node Type: NORMAL

   Local Node State: AGENT

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

r5

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-30T05:19:46Z

   Local Node UUID: 5745b6b9-d5d2-73d7-2bfd-70e2840da18a

   Local Node Type: NORMAL

   Local Node State: AGENT

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

r6

Cluster Information

   Enabled: true

   Current Local Time: 2017-05-30T05:20:19Z

   Local Node UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Local Node Type: NORMAL

   Local Node State: MASTER

   Local Node Health State: HEALTHY

   Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88

   Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20

   Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182

   Sub-Cluster Membership Entry Revision: 1

   Sub-Cluster Member Count: 6

   Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6

   Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88

   Unicast Mode Enabled: true

   Maintenance Mode State: OFF

After restarting the ESXi, agents from it disappear.

First I updated the vCSA and then the all ESXis.

I checked everything. There are agents everywhere. But after a reboot, they disappear. Also in the settings, the network mode remained multicast.

Hi

Have you solved this problem? I have the same problem

Hello skyliuliu​,

What update has been applied to the vCenter and the hosts and which was upgraded first?

A minimum of version 6.5d for both is required.

If build-versions are okay then check that all disks have been upgraded to on-disk format v5.

Check that all nodes have unicastagent lists populated with the correct details for all other member hosts of the cluster:

#esxcli vsan cluster unicastagent list

If not then check that this is set to 0:

# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates

If still no Unicast then manually populate the unicastagent lists from the CLI (needs to be done for all other members on each host):

# esxcli vsan cluster unicastagent add -t node -U 1 -u <uuid> -i <vmk#> -a <IP_of_vSAN_vmk>

https://kb.vmware.com/s/article/2150303

More info:

https://storagehub.vmware.com/export_to_pdf/vsan-multicast-removal

Hope this helps.

Bob