E105: PANIC: NOT_REACHED bora\lib\disklib\sparse.c:7082 on Workstation 15 Pro 15.5.2 build-15785246

[MichaelHaider]

Hi everyone,

I am running Workstation 15 Pro on Windows 10, have a Ubuntu 18.04 VM running on which I am operating a jFrog XRay 3.

All for testing and evaluation purposes.

During heavy disk io operations I keep encountering E105: PANIC: NOT_REACHED bora\lib\disklib\sparse.c:7082 errors, completely stopping my VM.

So far Google research has not led to anything helpful.

Can anybody here give me a hint on what to do to prevent this problem from happening or work around it?

Kind regards,

Michael Haider

[scott28tt]

What is the underlying storage for the files of your VM? And what levels of disk utilisation does Windows report?

[Mikero]

Could be file system or hardware corruption.

When you say 'heavy io', what sort of numbers are we looking at?

[MichaelHaider]

The underlying storage is a magnetic disc Toshiba MQ01ACF050 with a Windows NTFS filesystem.

There are no entries in the Windows eventlog about any irregularities with this drive or filesystem.

A check of the filesystem has shown no errors.

Michael

[MichaelHaider]

I can' really say exactly as those errors always seem to happen while I am not watching, but here is a part of the vmware.log shortly before and including the PANIC event:

2020-07-22T18:32:41.950+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.822 seconds (ok)

2020-07-22T18:32:42.209+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.083 seconds (ok)

2020-07-22T18:33:02.896+02:00| vmx| I125: scsi0:0: Command READ(10) took 1.078 seconds (ok)

2020-07-22T18:33:08.824+02:00| vmx| I125: DISKLIB-LIB   : numIOs = 3000000 numMergedIOs = 132532 numSplitIOs = 11470

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: caller 1 numvm 2 locked pages: num 3605792 max 7264512

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: locked Page Limit: host 7694704 config 7280896 dynam 8320769

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: minmempct 50  timestamp 250827

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: VM 0 min 1067725 max 2116301 shares 2097152 paged 1022125 nonpaged 8436 anonymous 10713 locked 1067731 touchedPct 12 dirtiedPct 4 timestamp 250826 vmResponsive is 1

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: VM 1 min 1336262 max 2646982 shares 2621440 paged 1024185 nonpaged 10246 anonymous 15296 locked 2538061 touchedPct 15 dirtiedPct 4 timestamp 250827 vmResponsive is 1

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: locked 2538061 target 2646982 balloon 0 0 1658964 swapped 80081 0 allocd 43 512 state 0 100

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: states: 0 14401 : 1 0 : 2 0 : 3 0

2020-07-22T18:33:10.290+02:00| vmx| I125: MemSched: Balloon enabled 1 guestType 1 maxSize 2552253

2020-07-22T18:33:26.414+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.606 seconds (ok)

2020-07-22T18:33:26.494+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.680 seconds (ok)

2020-07-22T18:33:26.605+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.783 seconds (ok)

2020-07-22T18:33:26.652+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.820 seconds (ok)

2020-07-22T18:33:26.701+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.118 seconds (ok)

2020-07-22T18:33:26.809+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.891 seconds (ok)

2020-07-22T18:33:26.862+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.938 seconds (ok)

2020-07-22T18:33:26.954+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.025 seconds (ok)

2020-07-22T18:33:27.022+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.089 seconds (ok)

2020-07-22T18:33:27.124+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.183 seconds (ok)

2020-07-22T18:33:27.192+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.608 seconds (ok)

2020-07-22T18:33:27.237+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.652 seconds (ok)

2020-07-22T18:33:27.298+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.714 seconds (ok)

2020-07-22T18:33:27.379+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.793 seconds (ok)

2020-07-22T18:33:27.517+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.931 seconds (ok)

2020-07-22T18:33:27.593+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.005 seconds (ok)

2020-07-22T18:33:27.695+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.107 seconds (ok)

2020-07-22T18:33:28.150+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.561 seconds (ok)

2020-07-22T18:33:28.171+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.581 seconds (ok)

2020-07-22T18:33:28.200+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.609 seconds (ok)

2020-07-22T18:33:28.238+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.646 seconds (ok)

2020-07-22T18:33:33.348+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.008 seconds (ok)

2020-07-22T18:33:33.840+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.499 seconds (ok)

2020-07-22T18:33:33.841+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.499 seconds (ok)

2020-07-22T18:33:33.880+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.537 seconds (ok)

2020-07-22T18:33:34.305+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.962 seconds (ok)

2020-07-22T18:33:34.353+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.010 seconds (ok)

2020-07-22T18:33:34.524+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.843 seconds (ok)

2020-07-22T18:33:34.525+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.181 seconds (ok)

2020-07-22T18:33:34.919+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.075 seconds (ok)

2020-07-22T18:33:35.263+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.417 seconds (ok)

2020-07-22T18:33:35.513+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.670 seconds (ok)

2020-07-22T18:33:35.567+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.723 seconds (ok)

2020-07-22T18:33:35.702+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.858 seconds (ok)

2020-07-22T18:33:35.768+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.923 seconds (ok)

2020-07-22T18:33:35.891+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.046 seconds (ok)

2020-07-22T18:33:35.943+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.098 seconds (ok)

2020-07-22T18:33:36.074+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.228 seconds (ok)

2020-07-22T18:33:36.277+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.430 seconds (ok)

2020-07-22T18:33:36.395+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.549 seconds (ok)

2020-07-22T18:33:41.639+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.081 seconds (ok)

2020-07-22T18:33:47.676+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 2.708 seconds (ok)

2020-07-22T18:33:53.440+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 1.825 seconds (ok)

2020-07-22T18:33:56.727+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 5.499 seconds (ok)

2020-07-22T18:34:10.865+02:00| vmx| I125: DISKLIB-LIB   : numIOs = 3050000 numMergedIOs = 133576 numSplitIOs = 11567

2020-07-22T18:34:16.240+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 39.509 seconds (ok)

2020-07-22T18:34:16.999+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 39.851 seconds (ok)

2020-07-22T18:34:17.000+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 36.736 seconds (ok)

2020-07-22T18:34:17.003+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 37.222 seconds (ok)

2020-07-22T18:34:17.005+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 37.227 seconds (ok)

2020-07-22T18:34:17.006+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 40.084 seconds (ok)

2020-07-22T18:34:17.006+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 39.931 seconds (ok)

2020-07-22T18:34:17.007+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 39.932 seconds (ok)

2020-07-22T18:34:17.007+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 37.226 seconds (ok)

2020-07-22T18:34:17.276+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 16.845 seconds (ok)

2020-07-22T18:34:17.276+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 32.307 seconds (ok)

2020-07-22T18:34:17.278+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 26.457 seconds (ok)

2020-07-22T18:34:17.279+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 26.466 seconds (ok)

2020-07-22T18:34:17.282+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 26.461 seconds (ok)

2020-07-22T18:34:17.383+02:00| vmx| I125: scsi0:0: Command WRITE(10) took 26.570 seconds (ok)

2020-07-22T18:34:17.889+02:00| vmx| E105: PANIC: NOT_REACHED bora\lib\disklib\sparse.c:7082

2020-07-22T18:34:17.890+02:00| vmx| I125: CoreDump_CoreDump: faking exception to get context

2020-07-22T18:34:17.914+02:00| vmx| I125: CoreDump: Minidump file D:\New Virtual Machines\XRay 3 RPM on Ubuntu 18.04 64-bit\vmware-vmx.dmp exists. Rotating ...

2020-07-22T18:34:17.948+02:00| vmx| W115: CoreDump: Writing minidump to D:\New Virtual Machines\XRay 3 RPM on Ubuntu 18.04 64-bit\vmware-vmx.dmp

2020-07-22T18:34:19.415+02:00| vmx| I125: CoreDump: including module base 0x0x7ff7322b0000 size 0x0x01493000

2020-07-22T18:34:19.415+02:00| vmx| I125:   checksum 0x00ffeaf9 timestamp 0x5e64013a

2020-07-22T18:34:19.415+02:00| vmx| I125:   image file C:\Program Files (x86)\VMware\VMware Workstation\x64\vmware-vmx.exe

2020-07-22T18:34:19.415+02:00| vmx| I125:   file version 15.5.2.54704

Somebody has already suggested that this might be related to a virus scanner blocking access to the VMDK file at certain points.

Problem is - I have no rights to stop or disable the virus scanner.

Michael

[MichaelHaider]

I have done some measuring today:

While the XRay initial database sync is running on the machine there is an average disk io on the VMs VMDK file with an overall thoughput of between 2.5 and 4 million B/s shown by resource monitor in Windows, split into roughly 1/3 read and 2/3 write operations.

At the same time I see heavy disk io coming from the Trend Micro Virus Scanner in its trac.log, which sort of suggests that my problem may really be connected to the virus scan engine operating.

Michael

[wila]

Hi,

Try and see if there's an option to exclude the vmdk files in Trend Micro.

If you can't add exceptions yourself, then ask your administrator to exclude the VM's vmdk files from the virus scanner. It's not like Trend Micro will find viruses in that vmdk file, in order to protect the VM you would have to run an antivirus product in the VM itself.

Hope this helps,

--

Wil