3D Application Stops Rendering/Freezes

[SleeplessInKans]

System Information:

Host OS: CentOS 7.4

Guest OS Windows 7 Pro 64-bit

VMWare Workstation 14.1.1

Resources allocated to GuestOS:

4 Cores, 8 GB RAM, 3D Acceleration enabled, 2 GB Video RAM

The 3D application starts and runs fine for 5-10 minutes and then it just stops rendering/freezes (there is still some CPU activity on the process and small changes to memory). The application is using significantly more memory running in a VM vs native.

I see the following entries in the vmware.log around the time this occurs:

2018-02-09T10:10:43.566-06:00| vcpu-0| I125: Guest: vm3d: VM3DDevAdapterGrowOTableItem: Grow OTable 1 from 1024 entries to 2048 entries, use 32 pages

2018-02-09T10:10:45.967-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:11:30.891-06:00| vcpu-0| I125: Guest: vm3d: ResizeDXCOTable: Resizing COTable 2 from 512 entries to 1024 entries

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: caller 0 numvm 1 locked pages: num 2031271 max 3539968

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: locked Page Limit: host 3977587 config 3548160 dynam -1

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: minmempct 50  timestamp 254205

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: VM 0 min 1066668 max 2115244 shares 2097152 paged 899476 nonpaged 7183 anonymous 10909 locked 2031271 touchedPct 3 dirtiedPct 0 timestamp 254205 vmResponsive is 1

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: locked 2031271 target 2115244 balloon 0 0 1363148 swapped 21978 0 allocd 0 512 state 0 100

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: states: 0 1801 : 1 0 : 2 0 : 3 0

2018-02-09T10:30:05.322-06:00| vmx| I125: MemSched: Balloon enabled 1 guestType 4 maxSize 0

I've attached the full log as well.

I've looked at the virtual memory in Windows and it looks fine and none of the swap in Cent OS is being used. There seems to be plenty of physical and virtual memory free. Also if I kill the application and restart it, it exhibits the same behavior.

Any thoughts on what is happening? Ae them some settings I can tweak to get it working? Thanks!

[bluefirestorm]

There are also several lines of

svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

If we take these log entries at face value, one (hopefully intelligent) guess/interpretation is that the svga driver has failed to allocate 4096 bytes to the paging file inside the VM. Or those log entries might as well been saying "We're not in Kansas anymore" or "You're not in Kansas anymore" (pop culture references to "Wizard of Oz" and "Avatar")

This might seem counterintuitive. You could try to reduce the amount of video memory from 2048MB to 1024MB. The way the VMware virtual video adapter memory works looks to be shared with the virtual RAM. You can see this in dxdiag. So it does not work the way a discrete graphic adapter does like that of the GTX 760 that you have on the host; that the video memory is dedicated to itself and separate from the RAM of the (virtual) PC. This might also explain why it appears to use more memory inside a VM.

Alternative to reducing virtual video adapter memory is to increase the paging file size of the Windows 7 VM. If it is system managed, Windows 7 can make it grow as much as 3 times of the RAM (in this case 8GB). So you may want to make sure if you have at least 24GB available/free disk space for inside the VM for the paging file to grow.

2018-02-09T10:02:46.360-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:02:46.360-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:10:45.967-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:10:45.967-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:11:16.600-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:11:16.601-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:11:26.643-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:11:26.643-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:12:16.732-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:12:16.733-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:13:21.707-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:13:21.707-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:15:24.865-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T10:15:24.865-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T11:12:14.236-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T11:12:14.237-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T11:12:30.514-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T11:12:30.514-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T11:13:17.358-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-09T11:13:17.359-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

[SleeplessInKans]

Thanks for your suggestions.

I tried to set the Windows VM size to 24 GB and had the same result. There is just no indication from the OS that memory (physical or virtual) are low. Lowering the video memory causes the issue to appear more quickly. I enabled debug logging and this is what was seen immediately before the "cannot allocate memory" messages.

2018-02-14T15:26:59.501-06:00| vcpu-1| I125: Guest: vm3d: ResizeDXCOTable: Resizing COTable 2 from 128 entries to 256 entries

2018-02-14T15:27:10.249-06:00| vcpu-1| I125: Guest: vm3d: ResizeDXCOTable: Resizing COTable 9 from 4096 entries to 8192 entries

2018-02-14T15:27:12.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck limits: ovhdText=22528(88.0MB) ovhdStatic=50176(196.0MB) ovhdMalloc=589716(2303.5MB)

2018-02-14T15:27:12.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck usage: ovhdText=15146(59.1MB) ovhdStatic=32635(127.4MB) ovhdMalloc=166270(649.4MB)

2018-02-14T15:27:12.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck: total=198905(776.9MB) arena=99194(387.4MB) maps=67076(262.0MB) stacks=18944(74.0MB) data=13691(53.4MB)

2018-02-14T15:27:12.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck: malloc used=376814528 (91996 pgs) free=29484096 (7199 pgs) tot=406298624 (99194 pgs)

2018-02-14T15:27:20.713-06:00| vmx| I125: LOADAVG: 1.03 0.81 0.58

2018-02-14T15:27:32.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck limits: ovhdText=22528(88.0MB) ovhdStatic=50176(196.0MB) ovhdMalloc=589716(2303.5MB)

2018-02-14T15:27:32.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck usage: ovhdText=15146(59.1MB) ovhdStatic=32635(127.4MB) ovhdMalloc=173431(677.4MB)

2018-02-14T15:27:32.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck: total=206066(804.9MB) arena=106355(415.4MB) maps=67076(262.0MB) stacks=18944(74.0MB) data=13691(53.4MB)

2018-02-14T15:27:32.038-06:00| vmx| I125: OvhdMemVmx_SizeCheck: malloc used=407067456 (99382 pgs) free=28562624 (6974 pgs) tot=435630080 (106355 pgs)

2018-02-14T15:27:45.966-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

2018-02-14T15:27:45.966-06:00| svga| I125: MainMem: Failed to map 0x1000 bytes to paging file, 'Cannot allocate memory' (12).

One side effect of lowering the video memory was seeing that while the application was "not responsive" it did eventually start rendering for a bit and stop again. Much later in the log it had the following.

2018-02-14T17:35:51.081-06:00| svga| I125: SVGA Driver Error: svgaDX.c:1935 Unknown

2018-02-14T17:35:51.081-06:00| svga| I125: SVGA Driver Error: svgaDXCmd.c:2015 DXDestroyQuery

2018-02-14T17:35:51.081-06:00| svga| I125: SVGA Driver Error: svgaFifo.c:725 DXDestroyQuery

2018-02-14T17:35:51.081-06:00| svga| I125: SVGA Driver Error: svgaFifo.c:728 DXDestroyQuery

2018-02-14T17:35:51.081-06:00| svga| I125: SVGA Driver Error: svgaFifo.c:2431 Unknown

This repeated over and over for 7-8 minutes. Then happens again every hour or so for 7-8 minutes.