vmware-vmx process hogs cpu after resuming guest

[timeOday]

I am running vmware fusion 4.1.2 (683185) on a 4 core macbook pro with ample ram and disk.

Every time I resume the guest (Windows 7, tools 8.8.3), it resumes and is responsive after just a couple seconds, but the vmware-vmx process on the host uses 140% CPU for 2 or 3 minutes.  It is very consistent, even if I suspend the guest and then resume just a couple seconds later, it will burn a couple minutes of CPU time immediately upon resume.  No other process on either the host or guest is using over 2% cpu at the time.  After this time period, it settles back down.

Here are things I have tried that did not help:

* Verified guest cpu is completely idle

* Disabled shared folders

* Disabled guest networking

* Time machine disabled on host

* Removed VMWare folder (where the guest is stored) from being indexed by spotlight

* Killed mds and mdworker processes on the host

* Disabled antivirus protection on host

* Disabled antivirus protection on the guest

* Watched vmware-vmfusion.log (last messages before the cpu burn are "Failed to get guest app info hash: Internal error")

dtruss shows the same repetitive set of calls both during and after the burn:

kevent(0x24, 0x0, 0x0)           = 1 0

poll(0x1646EAB40, 0x1, 0xFFFFFFFF)           = 1 0

read(0x42, "\273\0", 0x40)           = 1 0

ioctl(0x12, 0xC0045608, 0x1646EADE0)           = 0 0

ioctl(0x12, 0xC0045608, 0x1646EADE0)           = 0 0

Any ideas?  Right now just firing up Fusion cooks off about 7% of my battery charge without accomplishing anything that I can see.

Thanks.

[timeOday]

As a partial workaround, reducing the guest from 8 cores to 1 does reduce the amount of wasted battery and excess heat whenever Fusion is resumed.  This way vmware-vmx then consumes only 1 core. 

[dariusd]

Hi timeOday, and welcome to the VMware Communities!

Can you post (as an attachment) the vmware.log which can be found inside the VM's bundle right after one of these CPU cooking episodes has ended?  (Right-click on the VM in Finder, choose "Show Package Contents")

vmware-vmfusion.log is for the user-interface, so it is unlikely to contain anything relevant for a problem that causes the virtual machine itself (vmware-vmx) to burn CPU.  The most relevant stuff will be in the abovementioned vmware.log.

It might also help if you could try using Activity Monitor to take a sample of the vmware-vmx process while it's chewing CPU.  Posting that file as another attachment could give us some hints as to what's going on.

Also,  what's the Model ID of your host (i.e. MacBookPro6,1, visible in System Profiler) and which Mac OS version are you running on it?

Cheers,

--

Darius

[timeOday]

Thanks Darius, here is the info.  vmware.log and sample of vmware-vmx are attached.

Hardware Overview:

  Model Name:     MacBook Pro

  Model Identifier:     MacBookPro8,2

  Processor Name:     Intel Core i7

  Processor Speed:     2.3 GHz

  Number of Processors:     1

  Total Number of Cores:     4

  L2 Cache (per Core):     256 KB

  L3 Cache:     8 MB

  Memory:     8 GB

  Boot ROM Version:     MBP81.0047.B04

  SMC Version (system):     1.69f1

System Software Overview:

  System Version:     Mac OS X 10.6.8 (10K549)

  Kernel Version:     Darwin 10.8.0

  Boot Volume:     SnowLeopardOSX

  Boot Mode:     Normal

  Secure Virtual Memory:     Enabled

  64-bit Kernel and Extensions:     Yes

  Time since boot:     14 days 16:56

[dariusd]

Hi timeOday,

Thanks for posting that report and those files.  Unfortunately the smoking gun I was seeking is nowhere to be found...

And I should have also said earlier: that was an epic first post... our average customer generally doesn't dtrace/dtruss our application for us without being talked through it.  Mad props for that.  :smileycool:

Can you measure or estimate how many ioctl(..., 0xC0045608, ...) calls are being made per second?  It's quite normal for that ioctl to be "busy" (maybe hundreds per second), but not devastatingly busy (tens of thousands per second).  I suspect it'll be devastatingly busy when your system is in CPU-thrashing mode, but it'd be great to confirm or deny that theory.

Thanks in advance,

--

Darius

[timeOday]

Darius,

Here are the results if I run this command in one shell:

while test 1; do sudo dtruss -n vmware-vmx -t ioctl 2>&1 | grep 'ioctl.*0xC0045608' | wc -l; done

and in another:

while test 1; do sleep 1; sudo killall dtrace; done

When Fusion is running normally (has been up for a while with nothing much happening on the guest).

I continued watching for several minutes and never saw over 399, until it spiked (up to 5434) for just a couple seconds.

88

88

96

92

87

94

293

74

95

93

Below is the data for an entire burn (about 3.5 minutes of 100% vmware-vmx cpu usage) plus another 10 seconds.

There is definitely something going on, though not for the entire duration.

Thanks again.

520

3225

619

162

5306

849

9922

518

8662

961

12299

9667

10723

10818

366

6628

12979

10298

10096

7369

6

381

1008

44

41

60

743

197

241

201

228

27

345

41

170

34

40

31

76

24

9

92

180

26

41

18

19

197

17

340

56

1205

213

2913

637

41

49

8979

11905

7379

8348

621

12896

7396

9741

11439

8471

362

186

324

61

66

51

94

326

92

233

36

20

28

97

36

32

28

20

52

28

31

20

228

41

15

30

226

44

103

278

19

30

334

38

19

18

36

54

65

237

35

27

104

263

25

41

25

29

61

17

11

20

239

46

359

273

230

343

43

13

26

23

30

50

18

220

35

36

49

226

21

34

38

63

28

13

21

35

297

4

329

52

222

38

21

29

262

26

17

20

28

225

17

8

20

325

40

19

16

245

34

837

<-------- burn is over

83

341

82

90

99

70

76

73

82

85

[dariusd]

Hi timeOday,

There are all sorts of reasons for the ioctl count to vary and to spike -- and sometimes, as you've seen, it'll vary wildly -- but as long as it isn't pegged constantly in the tens of thousands, it's not going to point us in the direction of the root cause.  As far as I can tell, the counts in the log you posted seem normal enough for a "normal" VM.  The search for a smoking gun continues.

Interestingly, one possible locus of this issue is gravitating towards that of another issue which has been vexing me lately.  On the surface, the parameters don't seem to match up, but I feel compelled ask: Are you using an MCE OptiBay, an OWC Data Doubler, or equivalent piece of hardware to add another internal storage device to your MacBook Pro?

Thanks for your patience,

--

Darius

[DJHead]

Hi DariusD,

Just to let you know i am suffering the same issue as timeOday, also on a Macbook Pro (2011) but using a datadoubler as you mentioned. SSD for OS and Virtual Machine and SATA in the DVD slot for other data.

I also find even if i eventually get the Windows 7 VM to shutdown (takes a while) after i restart it (without shutting fusion) it instantly consumes 100%+ CPU. The only way to stop this is to quit fusion completely and then relaunch and start the Windows 7 VM again.

Details a follows:
VMware Tools: 8.8.3.12575

Hardware Overview:

  Model Name:     MacBook Pro

  Model Identifier:     MacBookPro8,2

  Processor Name:     Intel Core i7

  Processor Speed:     2.2 GHz

  Number of Processors:     1

  Total Number of Cores:     4

  L2 Cache (per Core):     256 KB

  L3 Cache:     6 MB

  Memory:     16 GB

  Boot ROM Version:     MBP81.0047.B27

  SMC Version (system):     1.69f3

As a side note, the Fusion 2012 Tech Preview does not suffer this same problem on this macbook with a copy of the Windows 7 VM and updated Tech Preview (HW v9) tools.

Regards,

David.