VMware Cloud Community
CollinChaffin
Enthusiast
Enthusiast

**WARNING - iSCSI volumes WILL FAIL using build 3073146 other NETWORK FAILURES!!!!**

Dear VMware, thanks for wasting 3 solid days of my life to find out it was your server product patch that caused the entire thing to fail.  You owe me those days back.

Only after rebuilding our esxi 6 hosts from 3073146 to 3247720 is this fixed.

Note the only urgent "fix" of build 3247720 is related to "Changed Block Tracking (CBT) in ESXi 6.0, the CBT API call QueryDiskChangedAreas() ". Um, yea - it apparently impacts more than that with the API - and the network because all iSCSI vm's became UNUSABLE at <5mB and logging errors (funny the vms aren't flagged with CBT in their configs, yet we see CBT errors in vmkernel.log prior to updating.  In over 27yrs this is the #1 hardest to troubleshoot since this issue is 100% buried inside this product and was almost impossible to track down if I didn't have the new build just to prove it I would still be testing.

Anyway, hope google indexes this and it saves anyone else from utterly wasting days away from their babies troubleshooting mass server failures all due to a bad patch.

0 Kudos
2 Replies
CollinChaffin
Enthusiast
Enthusiast

I should clarify that the prior build (3073146) is the one that is BROKEN.  You need to update right away to 3247720 if you have iSCSI mounted datastores or you will see these issues.  If you have huge teamed NIC bandwidth it may take longer but on the lower end - it will be obvious but not obvious WHAT is causing it since the iSCSI volume doesn't drop, continues to be accessible but is so utterly slow with degrading speeds and random slows to almost zero that you will literally tear apart your infrastructure from switches to cabling to server hw until you patch vmware and it's fixed.

0 Kudos
Ruddster
Contributor
Contributor

Just wanted to say thanks for this post - hopefully it will resolve an issue we have been seeing with a brand new environment.  Had changed a lot of other settings (changing multi-pathing policy etc.) which helped to a point but issue was still occuring although less frequently.

0 Kudos