Hi!
Upgrade VC and all ESXi. vSAN Network Mode no change to unicast. New clusters are also created in multicast mode.
Please tell me how to fix it?
Hello skyliuliu,
What update has been applied to the vCenter and the hosts and which was upgraded first?
A minimum of version 6.5d for both is required.
If build-versions are okay then check that all disks have been upgraded to on-disk format v5.
Check that all nodes have unicastagent lists populated with the correct details for all other member hosts of the cluster:
#esxcli vsan cluster unicastagent list
If not then check that this is set to 0:
# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates
If still no Unicast then manually populate the unicastagent lists from the CLI (needs to be done for all other members on each host):
# esxcli vsan cluster unicastagent add -t node -U 1 -u <uuid> -i <vmk#> -a <IP_of_vSAN_vmk>
https://kb.vmware.com/s/article/2150303
More info:
https://storagehub.vmware.com/export_to_pdf/vsan-multicast-removal
Hope this helps.
Bob
Addition.
Use RVC
> vsan.ondisk_upgrade ~vsan --allow-reduced-redundancy
+------------------------+-----------+-------------+----------------+----------------+------------------+----------------+----------------+
| Host | State | ESX version | v1 Disk groups | v2 Disk groups | v2.5 Disk groups | v3 Disk groups | v5 Disk groups |
+------------------------+-----------+-------------+----------------+----------------+------------------+----------------+----------------+
| r1.local | connected | 6.5.0 | 0 | 0 | 0 | 0 | 1 |
| r2.local | connected | 6.5.0 | 0 | 0 | 0 | 0 | 1 |
| r3.local | connected | 6.5.0 | 0 | 0 | 0 | 0 | 1 |
| r4.local | connected | 6.5.0 | 0 | 0 | 0 | 0 | 1 |
| r5.local | connected | 6.5.0 | 0 | 0 | 0 | 0 | 1 |
| r6.local | connected | 6.5.0 | 0 | 0 | 0 | 0 | 1 |
+------------------------+-----------+-------------+----------------+----------------+------------------+----------------+----------------+
2017-05-26 08:59:25 +0000: Running precondition checks ...
2017-05-26 08:59:29 +0000: Passed precondition checks
2017-05-26 08:59:29 +0000:
2017-05-26 08:59:29 +0000: Target file system version: v5
2017-05-26 08:59:29 +0000: Disk mapping decommission mode: ensureObjectAccessibility
2017-05-26 08:59:51 +0000: Check cluster status for disk format conversion.
2017-05-26 08:59:51 +0000: Upgrade tool stopped due to error, please address reported issue and re-run the tool again to finish upgrade.
Hello,
On-disk format has nothing to do with network connectivity mode (also, your disks are all v5 which is the highest version at present).
Do you see Unicast Mode enabled when you run this via SSH on all hosts?:
#esxcli vsan cluster get
Please provide the build number of your vCenter and your hosts.
You should be able to set Unicast mode manually by following the steps I have outlined in my comments here:
https://communities.vmware.com/thread/563705
(though no guarantees this will be persistent across vCenter reboots so test this)
Bob
-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-
All host
esxcli vsan cluster get
Cluster Information
Enabled: true
Current Local Time: 2017-05-26T10:51:42Z
Local Node UUID: 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
Current version details vCSA
From your previous snapshot, it does appear that the mode was changed to unicast
Unicast Mode Enabled: true
You can run the following to make sure each host sees its neighbors.
esxcli vsan cluster unicastagent list
Hello Dave,
Yes I PM'ed CaptainEmilyJones with the steps for checking unicast agent list and adding nodes if necessary.
My last comment also links to a more expansive explanation of these steps.
Bob
-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-
Yes, hosts see each other. However, in the General settings of the cluster, the network mode is multicast, whereas in the documentation it is written that it should be replaced by a unicast.
I checked everything. There are agents everywhere. But after a reboot, they disappear. Also in the settings, the network mode remained multicast.
(All hosts) esxcli vsan cluster unicastagent list
NodeUuid IsWitness Supports Unicast IP Address Port Iface Name
------------------------------------ --------- ---------------- -------------- ----- ----------
4312e1d8-9b60-8fa8-cb63-70e2840d9e88 0 true 192.168.99.213 12321
5745b6b9-d5d2-73d7-2bfd-70e2840da18a 0 true 192.168.99.214 12321
574da762-529d-0413-2107-70e2840d9e04 0 true 192.168.99.216 12321
57cfd45c-794d-9b42-6528-70e2840d9e90 0 true 192.168.99.217 12321
57d04aff-5c84-aa0e-6548-70e2840e0f20 0 true 192.168.99.218 12321
57a76c06-0a38-c90c-85bf-70e2840d9cc6 0 true 192.168.99.219 12321
Hello,
Can you check on ALL nodes that 'esxcli vsan cluster get' returns 'Unicast Mode Enabled: true' ?
You said 'after reboot' the agent lists were blank - Is this after rebooting vCenter or the hosts?
I ask this as it is possible that the vCenter may be unaware that Unicast should be used and thus pushes a blank list.
Which did you upgrade first - the hosts or the vCenter?
Bob
-o- If you found this comment useful please click the 'Helpful' button and/or select as 'Answer' if you consider it so, please ask follow-up questions if you have any -o-
All hosts.
r1
Cluster Information
Enabled: true
Current Local Time: 2017-05-30T05:15:00Z
Local Node UUID: 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
r2
Cluster Information
Enabled: true
Current Local Time: 2017-05-30T05:17:36Z
Local Node UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Local Node Type: NORMAL
Local Node State: BACKUP
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
r3
Cluster Information
Enabled: true
Current Local Time: 2017-05-30T05:18:17Z
Local Node UUID: 57cfd45c-794d-9b42-6528-70e2840d9e90
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
r4
Cluster Information
Enabled: true
Current Local Time: 2017-05-30T05:19:01Z
Local Node UUID: 574da762-529d-0413-2107-70e2840d9e04
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
r5
Cluster Information
Enabled: true
Current Local Time: 2017-05-30T05:19:46Z
Local Node UUID: 5745b6b9-d5d2-73d7-2bfd-70e2840da18a
Local Node Type: NORMAL
Local Node State: AGENT
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
r6
Cluster Information
Enabled: true
Current Local Time: 2017-05-30T05:20:19Z
Local Node UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Local Node Type: NORMAL
Local Node State: MASTER
Local Node Health State: HEALTHY
Sub-Cluster Master UUID: 4312e1d8-9b60-8fa8-cb63-70e2840d9e88
Sub-Cluster Backup UUID: 57d04aff-5c84-aa0e-6548-70e2840e0f20
Sub-Cluster UUID: 524b8a36-3d57-3082-8d49-e9fef1ad3182
Sub-Cluster Membership Entry Revision: 1
Sub-Cluster Member Count: 6
Sub-Cluster Member UUIDs: 57d04aff-5c84-aa0e-6548-70e2840e0f20, 574da762-529d-0413-2107-70e2840d9e04, 57cfd45c-794d-9b42-6528-70e2840d9e90, 4312e1d8-9b60-8fa8-cb63-70e2840d9e88, 5745b6b9-d5d2-73d7-2bfd-70e2840da18a, 57a76c06-0a38-c90c-85bf-70e2840d9cc6
Sub-Cluster Membership UUID: 03b22759-81c6-c446-8262-70e2840d9e88
Unicast Mode Enabled: true
Maintenance Mode State: OFF
After restarting the ESXi, agents from it disappear.
First I updated the vCSA and then the all ESXis.
Hi
Have you solved this problem? I have the same problem
Hello skyliuliu,
What update has been applied to the vCenter and the hosts and which was upgraded first?
A minimum of version 6.5d for both is required.
If build-versions are okay then check that all disks have been upgraded to on-disk format v5.
Check that all nodes have unicastagent lists populated with the correct details for all other member hosts of the cluster:
#esxcli vsan cluster unicastagent list
If not then check that this is set to 0:
# esxcfg-advcfg -g /VSAN/IgnoreClusterMemberListUpdates
If still no Unicast then manually populate the unicastagent lists from the CLI (needs to be done for all other members on each host):
# esxcli vsan cluster unicastagent add -t node -U 1 -u <uuid> -i <vmk#> -a <IP_of_vSAN_vmk>
https://kb.vmware.com/s/article/2150303
More info:
https://storagehub.vmware.com/export_to_pdf/vsan-multicast-removal
Hope this helps.
Bob