SAN 6.2 on disk upgrade fails at 5%

vikkx · ‎05-02-2016

At 5% the upgrade pauses for a bit then I get a simple "General Virtual SAN error. Disk format conversion failed due to unexpected error." (see attached image). All other health issues are green (see attached image). Any ideas?

zdickinson · ‎05-02-2016

Good morning, this has been reported by another vSAN user. Ondisk Upgrade stalled 5% I would open a support ticket or monitor that thread. Thank you, Zach.

vikkx · ‎05-02-2016

OK thanks!

alainrussell · ‎05-04-2016

Hi vikkx‌ did you get this resolved, did you open a support case?

NickBowie · ‎05-30-2016

Hi vikkx‌

I've encountered this same error for a customers VSAN implementation. Empty vsanDatastore. Only thing I can think of is when we ran performance stress tests, the IOBlazer run may not have cleaned itself up correctly. Even ran this as a check: -----

[root@aklesx19:/tmp] python VsanRealign.py precheck

VSAN Disk Format Conversion helper script revision 2

Starting namespace scan

Finished scanning, compiling results No issues were found during scanning.

----- Raising a support ticket now anyway. Will also follow this thread. Have you got any updates? Thanks.

NickBowie · ‎05-31-2016

This issue, where the upgrade fails at 5% with "General VSAN error" is different to the issue experienced in the other thread that's referenced (that one stalls at 5%, but continues to run). I have an open SR about this - #16126628305 - currently unresolved.

vikkx‌ - if you have worked with VMw Support on this and have a resolution, please advise.

Thank you.

NickBowie · ‎06-02-2016

Hi Vikkx,

I have completed the upgrade for the VSAN I was having an issue with - this information may help you. Key to this, is observing that each attempt at the "vsan.ondisk_upgrade" task incremented a new disk group to v2.5 before failing at 5% with the unhelpful error.

e.g.:

-->

vsan.resync_dashboard .

2016-05-31 05:58:41 +0000: Querying all VMs on VSAN ...

2016-05-31 05:58:41 +0000: Querying all objects in the system from aklesx11.localdom.co.nz ...

2016-05-31 05:58:42 +0000: Got all the info, computing table ...

+-----------+-----------------+---------------+

| VM/Object | Syncing objects | Bytes to sync |

+-----------+-----------------+---------------+

| Total | 0 | 0.00 GB |

+-----------+-----------------+---------------+

/aklvvc32.localdom.co.nz/AKL/computers/AKL-CL03> vsan.ondisk_upgrade .

+--------------------+-----------+-------------+----------------+----------------+------------------+----------------+

+--------------------+-----------+-------------+----------------+----------------+------------------+----------------+

| aklesx11.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx13.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx14.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx15.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx12.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx16.localdom.co.nz | connected | 6.0.0 | 0 | 1 | 3 | 0 |

| aklesx18.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx17.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

+--------------------+-----------+-------------+----------------+----------------+------------------+----------------+

2016-05-31 06:20:33 +0000: Running precondition checks ...

2016-05-31 06:20:35 +0000: Passed precondition checks

2016-05-31 06:20:35 +0000:

2016-05-31 06:20:35 +0000: Target file system version: v3

2016-05-31 06:20:35 +0000: Disk mapping decommission mode: evacuateAllData

2016-05-31 06:20:41 +0000: Upgrade tool stopped due to error, please address reported issue and re-run the tool again to finish upgrade.

/aklvvc32.localdom.co.nz/AKL/computers/AKL-CL03> vsan.ondisk_upgrade .

+--------------------+-----------+-------------+----------------+----------------+------------------+----------------+

+--------------------+-----------+-------------+----------------+----------------+------------------+----------------+

| aklesx11.localdom.co.nz | connected | 6.0.0 | 0 | 3 | 1 | 0 | < increment

| aklesx13.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx14.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx15.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx12.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx16.localdom.co.nz | connected | 6.0.0 | 0 | 0 | 4 | 0 | < and here

| aklesx19.localdom.co.nz | connected | 6.0.0 | 0 | 0 | 0 | 0 |

| aklesx18.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

| aklesx17.localdom.co.nz | connected | 6.0.0 | 0 | 4 | 0 | 0 |

+--------------------+-----------+-------------+----------------+----------------+------------------+----------------+

2016-05-31 06:36:57 +0000: Running precondition checks ...

2016-05-31 06:36:59 +0000: Passed precondition checks

2016-05-31 06:36:59 +0000:

2016-05-31 06:36:59 +0000: Target file system version: v3

2016-05-31 06:36:59 +0000: Disk mapping decommission mode: evacuateAllData

2016-05-31 06:37:05 +0000: Upgrade tool stopped due to error, please address reported issue and re-run the tool again to finish upgrade.

<-----

All I had to do then was to continue this process until all disk groups were at v2.5. After that, I executed the upgrade and it proceeded as expected - moving data off, removing and re-creating the disk group in a rolling fashion.

Hope this helps.