FreddyFredFred
Hot Shot
Hot Shot

windows 10 1809 slow

I downloaded the Windows 10 1809 and Server 2019 ISOs the day they became available so I can start working on my templates.

I built the templates with EFI, paravirtual for the C drive and vmxnet3 adapter. I've been using this combo for other versions of windows 10/8/7and windows server 2008r2/2012r2/1016 without issue.

So far Windows 2019 (with desktop experience) seems to be ok at least for a basic vm and guest customization. Haven't tried anything else yet.

Windows 10 1809 on the other hand is very, very slow to reboot after the initial install or even just rebooting after making some changes post install. After installing the OS it took 10-15 minutes for the initial windows setup stuff (user, security settings, etc) to appear . I tried a VM set to BIOS and it seemed faster but was still quite slow. Server 2019 and other versions of windows 10 have no issue.

The hosts are esxi 6.5 and 6.7.

I haven't had a chance to try every combo of BIOS/EFI/vmxnet3/e1000e/paravirtual/lsi sas to see if one is the cause of the issue but was wondering if anyone else had noticed any issues or if it was just me?

Thanks

126 Replies
batuhandemirdal
Enthusiast
Enthusiast

Hi,

Windows 10 is a problem in this version, microsoft recently released a patch update package in 1809 said the problem was solved.

0 Kudos
FreddyFredFred
Hot Shot
Hot Shot

I know about the issues that forced Microsoft to pull the ISOs originally but I don't think that's the problem here. I patched the vm in question but it's still very, very slow to reboot. (I don't have access to the republished ISOs at the moment)

When I reboot, see the windows logo, the spinning dots for a second and then just the logo for several minutes. When I check esxtop I see this:

cpu (c):

NAME             NWLD   %USED %RUN    %SYS   %WAIT %VMWAIT    %RDY %IDLE  %OVRLP   %CSTP %MLMTD  %SWPWT

win1809slow        10 211.90  171.48    6.02 966.99   55.94    0.39 0.00    3.30    0.00 0.00    0.00

disk device (u):

WQLEN ACTV QUED %USD  LOAD CMDS/s  READS/s WRITES/s MBREAD/s MBWRTN/s DAVG/cmd KAVG/cmd GAVG/cmd QAVG/cmd?

64     - 0    0    0 0.00  3649.39  3647.17 2.23     1.78     0.01     0.11     0.00     0.11      0.00

vm disk (v)

all zeros!

For some reason the disk seems to stop all io, the cpu goes very high and the datastore it's on gets a ton of reads but the VM is almost stuck. This is only affecting win 10 1809. Any other OS built the same way on the same server/storage is fine.

0 Kudos
FreddyFredFred
Hot Shot
Hot Shot

I think I found the problem: snapshots! I was building my new templates and had snapshots so I could undo a mistake without having to start over. For some reason Windows 10 1809 really doesn't like snapshots. As long as there's no snaps everytihng is fine. If I take a snap and reboot, the time it takes is noticebly longer.

ESXi 6.5 U1 seems to be the worst.

ESXi 6.7 seems better by comparison

ESXi 6.0 seems like it might not be affected (but this is on another environment)

I did a test by installing without having a snap and it was fast.

Rebooted right after and it was fast

Shutdown and took a snapshot (the only one) then powered on and noticeably slower.

Maybe there's an advanced setting I can play with but again, this problem didn't happen on Server 2019. ANy ideas?

My users take a bunch of snapshots as part of their tests so yes, this is going to be an issue for me

FreddyFredFred
Hot Shot
Hot Shot

Still fighting with this one but it might be snapshot and VMFS6 related. Same host, two different datastores on the same SAN. One formatted with VMFS5, the other VMFS6.

Create a new VM, take a snapshot and then install the OS (I know this isn't "right" but for the test, it's the fastest/easiest way to see the issue).

Only VMFS6 has the problem and only with windows 10 1809. Still not sure if normal operation is ok or if it's just reboot/startup which is super slow.

Can anyone else reproduce?

DanielRG
Contributor
Contributor

Hi FreddyFredFred, I have the same issue.

- ESXi, 6.7.0, 10302608

- Windows 10 1809

- Storage version VMFS6

When the VM has snapshots, the boot was very slow.

I tried all kind of configuration and only found the solution when moved the VM to VMFS5 storage, like you say. Thanks for post the solution! Smiley Happy

peetz
Leadership
Leadership

When looking at what's new in VMFS6 the "automatic space reclamation / unmap" feature stands out.

Maybe that causes the issue ... You can disable it on a per datastore basis. Can you try if that makes a difference?

Andreas

Twitter: @VFrontDe, @ESXiPatches | https://esxi-patches.v-front.de | https://vibsdepot.v-front.de
0 Kudos
FreddyFredFred
Hot Shot
Hot Shot

I tried a datastore with both auto unmap enabled and disabled and it made no difference.

FreddyFredFred
Hot Shot
Hot Shot

I have a case open with VMware. I'm waiting on the guy to get a lab setup with a VMFS6 datastore to try and repro the issue so they can maybe fix it.

0 Kudos
StevenPa
Contributor
Contributor

FreddyFredFred,

I have the exact same issue, this only started happening after I upgraded from 1803  to 1809, same issue with a fresh install. I am also using VMFS6.

Also upgraded from 6.5 U1 to 6.7, same issue. Has support responded yet ?

My work-around is to remove all snapshots... VM loads just fine.

0 Kudos
FreddyFredFred
Hot Shot
Hot Shot

Still waiting on VMware to get back to me (the guy assigned to my case had to get a new lab created with VMFS6 datastores because the first one he got was VMFS5 and we couldn't repro the issue). Hopefully I'll hear back this week.

I know the reboots are super slow but does it affect using the VM as well once it's loaded? I'm just making templates and not really doing much inside the VM but it didn't notice anything once the vm was booted.

0 Kudos
PXCSW
Contributor
Contributor

Same issue here running ESX 6.7 Update 1 and VMFS6 Datastores.

It does affect usage after bootup. And it seems it gets exponentially worse with more snapshots on the machine.

On one of my test machines had 1 Snapshot (turned off) and 1 snapshot (turned on). Tried a silent installation of firefox, which usually takes about 15 seconds.

It took 3-4 Minutes on 1809 (reproduceable).

RShumway
Contributor
Contributor

Exact same issue here.  Boots fast without snapshots.  Runs fine once up.  Problem for me is that I was going to test it with View and snapshots are needed for images.  I can't have a user waiting two minutes for their machine to startup.  They already like to complain about being virtual.

Aaron206
Contributor
Contributor

Same exact issue.

Win 10 1809 on VMFS 6 with 6 snaps... SLOW boot... (Horizon IC Image)

Moved the Win 10 and snapshots to a VMFS 5 datastore.. 8-second boot.

FreddyFredFred
Hot Shot
Hot Shot

Just an update for anyone interested...

I got an email from VMware saying they have an ongoing bug which matches my issue and they are waiting for an update from engineering.

Not the solution we all want but at least things are happening (not as fast as we would all hope but at least they seem to admit there is an issue that needs fixing)

LukaszDziwisz
Hot Shot
Hot Shot

Are you using EFI boot or BIOS mode for your master image? I do see the same problem with EFI mode. Moving to VMFS5 datastore does not resolve it for me with EFI mode.

Windows 10 1809 LTSC, vSphere 6.7, Horizon 7.6, AppVolumes 2.15

Here is what I'm seeing

EFI mode. 2 snapshots

VMFS6 - Boot time 3 minutes

VMFS5 - Boot time 3 minutes

BIOS Mode. 2 snapshots

VMFS6 - Boot time 1 minute 35 seconds

VMFS5 - Boot time 8 seconds

FreddyFredFred
Hot Shot
Hot Shot

Going from memory at this point but I think I saw it with both BIOS and EFI. Maybe BIOS was a bit better but still not what it should be.

0 Kudos
nschlip
Contributor
Contributor

Having the same issue here - Win10 1809 x64 Gold VM (master) reboots just fine, typically less than 30 seconds (around 26 seconds). However, all linked clone VM's (we have VMWare Advanced licensing, so no instant cloning, UEM, etc.) are taking upwards of 20-30 minutes.

We're running:

x5 ESXi 6.7. U1 hosts

Horizon 7.5.1

Tested with both EFI and BIOS Win10 1809; I have a support ticket in with VMWare, but it's been difficulting connecting at the same time. It sounds like someone else on here has a support ticket open, and they've escalated it to engineering and are waiting a response? If anyone has any input it'd be greatly appreciated. As it stands now, reboot a Win10 VM is taking far too long to fully boot to the login screen.

0 Kudos
FreddyFredFred
Hot Shot
Hot Shot

I have a ticket open for a while now and the last communication I got from VMware was on Tuesday saying: "VMWare Engineering team is currently working with Microsoft to root cause the issue and provide a fix."

Previous to that they I got an email saying they have a bug matching my issue and are working on it.

0 Kudos
nschlip
Contributor
Contributor

Thanks for the info, good to know - we've built a new Win10 gold VM in View, and are going to test with it. I'll let you know how it turns out.

0 Kudos