julianhainesasl
Contributor
Contributor

ESXi 7.0.2 update 3 release date

Good Day,

Does anyone know when the update for 7.0.2 to fix the SDCard is going to be released?

Thanks

41 Replies
pkvmw
VMware Employee
VMware Employee

Hi,

7.0 Update 3 will take longer time, as it's a bigger release than just a patch. However the SD-card fix you're looking for will be in the next upcoming patch release, which comes before Update 3. As official release dates are not publicly communicated, please be a bit patient.

Thanks.

Regards,
Patrik

0 Kudos
vmrulz
Hot Shot
Hot Shot

This issue and vmware's response has not been good. Many customers have been dealing with it for months. Support told me U3 will be out by mid August at the latest. I'm very happy we only have a development cluster on v7 but for those with busy production clusters this has to be a major PITA.

drtv
Contributor
Contributor

Hello there! This is my first time hearing this. 

I was just thinking about setting the recent ESXi verison up on an MicroSD Card device. 

Right now the "old" version runs on an USB Stick. 

So am I am getting this right: I should wait till U3 or is it better to get rid of the MicroSD Card at all. 

My problem so far was that in the recent years one USB Stick broke and I thougt it would be a good Idea to install ESXi fresh on a new MIcroSD Card. This Card I bought for over 100 Dollar is from HP exactly for this purpose. So I thought this is the more professional and safe solution. 

 

Thank you so much! Thomas

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

The current issues with 7.0U2a are NOT specific to micro-SD cards per se.  The issue affects any 7.02a boot device connected to a USB-controller, like your USB stick you are currently using as your boot device as well as many major system manufacturers' motherboards that include a SD/micro-SD slot (which is actually connected to the USB controller) on the motherboard.  The impact of the issues can be minimized (but not entirely eliminated) by relocating Scratch location, Coredump location and VMtools bits location to <not-the-boot-device> , which you should be doing anyway if booting from USB stick, but even with these mitigations I still encounter the problem periodically.  The workaround to regain control of a host in the throes of this failure is well-documented by @JailBreak  here:  https://www.provirtualzone.com/vsphere-7-update-2-loses-connection-with-sd-cards-workaround/

and in many other posts on this forum and others.

Booting from USB-based devices is still a supported method that VMware is calling "Legacy", but boot....  I have hundreds of servers in the field using microSD boot devices and will continue to operate them that way as long as it is supported by VMware, and by then those servers will probably fall off the compatibility list for current versions anyway - but I am impatiently waiting for U3 or the patch which contains the fix for this issue.  New systems I order will be spec'd with local disks, mostly because I don't have the time or energy too play whack-a-mole again fixing host issues for months while waiting for a fix the next time some update or patch completely changes I/O patterns and does not get thoroughly QA'd against a major portion of the installed hardware base.

drtv
Contributor
Contributor

Thank you for the answer. I am devastated. 

My internal USB Stick with ESXi 6.7 U2 broke. So I installed a new one fresh with 7U2. 

After that  I was concerned about the USB Sticks and bought an HPE 32GB MicroSD card because I thought it would run longer then my 10 Dollar USB Stick.....

And right now I am reading that either on USB or Micro SD I am screwed because my ESXi will freeze. 

Will that happen any ways, or could I be lucky that it won't happen? 

is it possible to downgrade to 6.7U2?

For your understanding, I am running a very small enviroment: 

Server HP DL 380 Gen 9

ESXi runs on this server and on the ESXi are two VMs. That's it. But in order to do business these two VMs have to WORK!

 

OMG what shall I do. I am so thankful for your help so far!!!!

 

0 Kudos
PatrickDLong
Enthusiast
Enthusiast

The issues that trigger the problem are myriad so I cannot say for sure that you will definitely encounter it or will definitely NOT encounter it.  In my experience what I see is that vm's on an affected host continue to run but they cannot be vmotioned off to other hosts (not an problem for you since you have just a single standalone host.).  Solarwinds has trouble checking on the vm status and other activities that partially rely on the host (like backups) can throw errors or fail.  ESXi 7.02a is the problem version - since you installed that version onto a new USB stick holding the HPE 32GB MicroSD card,  there is no prior version to easily roll back , and it would require a reinstall to get back to either 7.01 or to 6.7.U3.<latest>. 

From what I have read, EITHER a post-U2a patch containing a fix for this issue OR the full U3 patch will be released possibly in August - so you could roll the dice and just keep your eye on it.  I want to stress that info is hearsay and I have no personal insights to share.  If you DO encounter the issues noted above or other guest wierdness, the first thing I would do is ssh to the host and run ls -n to list the root directory contents.  If the command does not complete (or completes but shows bootbank and/or altbootbank in red text) you will need to follow the steps in the article I linked above to regain control of your host.  Once you have done those steps, the host should return to normal operation.  The process recommends rebooting your host once you regain control, but I have had success even WITHOUT rebooting the host (i.e. no down time for the vm's.)  Of course it is up to you to decide whether reloading a prior release for stability is worth the extra effort or if you can tolerate the risk of possible - but easily recoverable - host instability until a patch is released.

drtv
Contributor
Contributor

May I ask a Question at this point an get your professional expertise?

As said above I have a small environment. I have one physical host (HP ProLiant DL380 Gen9). It has 8 Bays. 6 Bays with Raid5 as Datastorage. 2 Bays with SSDs configured as smart cache.

I have a small dental office. Mainly I access SQL-Databases with my clients. Either for the case management or X-ray files. That’s it.

 

When I got this right, the best thing to do for the future would be to set up the ESXi not onto a USB Stick or Micro SD Card.

 

The best hardware option would be the OS Boot Device HPE NS204i-p. Unfortunately this is just available for Gen10 servers.

 

So I could get rid of the smart cache and use the two SSDs as Raid 1 and install ESXi on them.

Or I get a a 2xSFF upgrade Kit for my Gen9 server and ad two SSDs as Raid 1.

Do you think that the performance of the smart cache solution is better not missing and rather get two additional SSDs?

 

 

And the last question (hopefully you don’t hang me for it):

Does the new version of ESXi change anything on the VMs? Could I just shut down the host, pull the USB Stick and install 6.7 U2 fresh and mount my VMs?

 

And what is the issue with “relocating Scratch location, Coredump location and VMtools bits location to <not-the-boot-device> , which you should be doing anyway if booting from USB stick”.

What problem do I get if I don’t do so? I haven’t done it before. Two times already I installed the newest version of ESXi on a fresh usb-drive and “just mounted my VMs”. Was that wrong?

 

I see that my questions are going far beyond the purpose of this forum. But my problem is that here in in my city in Germany nobody can answer me this questions. My IT-office just installed my the newest version of ESXi on a USB-Stick, not knowing that there is this issue – so that says everything.

 

THANKS!!!

0 Kudos
JailBreak
Hot Shot
Hot Shot


@drtv wrote:

May I ask a Question at this point an get your professional expertise?

As said above I have a small environment. I have one physical host (HP ProLiant DL380 Gen9). It has 8 Bays. 6 Bays with Raid5 as Datastorage. 2 Bays with SSDs configured as smart cache.

I have a small dental office. Mainly I access SQL-Databases with my clients. Either for the case management or X-ray files. That’s it.

When I got this right, the best thing to do for the future would be to set up the ESXi not onto a USB Stick or Micro SD Card

The best hardware option would be the OS Boot Device HPE NS204i-p. Unfortunately this is just available for Gen10 servers.

So I could get rid of the smart cache and use the two SSDs as Raid 1 and install ESXi on them.

Or I get a a 2xSFF upgrade Kit for my Gen9 server and ad two SSDs as Raid 1.

Do you think that the performance of the smart cache solution is better not missing and rather get two additional SSDs?

And the last question (hopefully you don’t hang me for it):

Does the new version of ESXi change anything on the VMs? Could I just shut down the host, pull the USB Stick and install 6.7 U2 fresh and mount my VMs?

And what is the issue with “relocating Scratch location, Coredump location and VMtools bits location to <not-the-boot-device> , which you should be doing anyway if booting from USB stick”.

What problem do I get if I don’t do so? I haven’t done it before. Two times already I installed the newest version of ESXi on a fresh usb-drive and “just mounted my VMs”. Was that wrong?

I see that my questions are going far beyond the purpose of this forum. But my problem is that here in in my city in Germany nobody can answer me this questions. My IT-office just installed my the newest version of ESXi on a USB-Stick, not knowing that there is this issue – so that says everything.

THANKS!!!


If you did not apply updates after the 7.0 U2, yes you can rollback to the previous version.

You can add 2 SDD disks, or SSA disks, use Raid 1 and install ESXi on it. That is what we have in most of our environments (using SSDs, or SSA disks).

Assuming that you have your VMs in a SAN.

Regarding adding VMs back to ESXi hosts. It depends on if you have a vCenter or not. If you have, you can do one by one(ESXi), and move VMs hosts.

If you don't have vCenter, then you need to do it manually. Install the new ESXi on local disks, then you need to browse your SAN find vmx VMs, and add them manually to the ESXi.

Note: But you should create a new topic for this so that people can help and you award them the points.


Luciano Patrao
ICT Senior Infrastructure Engineer
Tech Lead for VMware / Virtual Backups
________________________________
If helpful Please award points
Thank You
Blog: https://www.provirtualzone.com | Twitter: @Luciano_PT
e_espinel
Expert
Expert

Hello.
I recommend you to go for the option of installing a 2xSFF upgrade Kit and 2 SAS disks  of 160 or 300GB 15K or 10k in RAID 1. These disks are more secure and reliable, they are only for the ESXI boot.


You will have to install VMware vSphere on these new disks, then run an HBA and Storage scan to access the datastores of the disk group in RAID 5 and finally register the VMs on the ESXi host.

 

 

Enrique Espinel
Senior Technical Consultant IBM, Lenovo and VMware.
VMware VSP-SV 2018, VTSP-SV 2018 VMware Technical Solutions Professional Hyper-Converged Infrastructure (VTSP-HCI 2018)
VMware Technical Solutions Professional (VTSP) 4 / 5.
Please mark my comment as the Correct Answer/Kudos if this solution resolved your problem Thank you.
Пожалуйста, отметьте мой комментарий как "Правильный ответ/Кудос", если это решение решило вашу проблему. Спасибо.
drtv
Contributor
Contributor

Thank you so much for this answer. 

 I already orderd the kit. Tomorrow I wanted to order the harddrives. 

Why take a SAS harddrive and no SSD? Is there a reasion? I thought SSD would be the most secuere and reliable thing to do! But good to know. 

Thank you so much for the help!

Most appriciated! 

0 Kudos
LarryBlanco2
Expert
Expert

I don't understand this rationale.  You have so many postings with people having issues with the U2 SD card botch/debacle. It seems there is a fix/patch for it today but want to make everyone wait? So the patch could be released  as part of a larger update?   

 

I ended up having to roll back to U1 with no updates and redirect tools to RAM.

 

In July we call VMware and get bad advise from support tech and then figure out the problem on our own. its the U2 SD Card issue. Get apology from support tech and lets me know fix will likely be released end of July.  End of July comes and go's.  Now I hear the fix will likely be release August.  Week and a half into August.. Then I hear the fix is likely to be released in September and is to be included in U3. 

Might as well just said we don't know anything and if you want to solve the problem today, roll back to a version that doesn't blow up.

 

vmrulz
Hot Shot
Hot Shot

Considering the hotfix is supposedly imminent I'd just hang tight and limp your hosts along depending on the size of your infra. For us we only have a 6 host cluster on 7u2. I setup a cluster reboot script which keeps us out of trouble usually. Also when you have a host getting wonky (vm backups failing is usually our canary in the coal mine so to speak), running esxcfg-rescan -d vmhba32 from the CLI several times until it comes back quickly and clean is a great way to get vcenter comm back in shape.  I just brought host back online this morning with that. It is also helpful to minimize the number of 3rd party processes hitting your hosts. We had Turbonomic doing some host/vm management and disabled that for the time being with good results.

0 Kudos
WuGeDe
Enthusiast
Enthusiast

Everytime I asked that question to VMware support the support gave another date.

Starting with mid June 21, then mid Juli 21 and now the latest answer was mid August 21. 🤣

0 Kudos
pkvmw
VMware Employee
VMware Employee

That's most likely one of the reasons we in GS are not allowed to provide dates when specific releases come out. The patch was re-scheduled from July to a later date, which is not far away anymore. Also there's a Patch Release (P03) coming out - NOT an update, which also leads to some confusion from what I've heard.

0 Kudos
WuGeDe
Enthusiast
Enthusiast

@pkvmw let us hope the release date will not be too far in the future and let us hope the patch will at last fix that defect.

0 Kudos
pkvmw
VMware Employee
VMware Employee

Srijithk
Enthusiast
Enthusiast

Documentation and Links for the patch fixing the bootbank issue

Download Link:

 

WuGeDe
Enthusiast
Enthusiast

@pkvmw 
Nice. Thanks.
Now let us see if the fix does really fix it.

@Srijithk 
Thanks for sharing the download links.

0 Kudos
WuGeDe
Enthusiast
Enthusiast

Update 2b was a total failure it seems.
It even was deleted in the list of Build numbers and versions of VMware ESXi.

https://kb.vmware.com/s/article/2143832?lang=en_US

0 Kudos