VMware Communities
angelsfan1
Contributor
Contributor
Jump to solution

USB air card causing kernel panics only when running Fusion 4.1.1

I have had numerous kernal panics when plugging in and unplugging a Sierra Wireless U250 3G/4G Sprint USB aircard when I have Fusion 4.1.1 also running on Lion 10.7.2. The aircard works fine and plugs in and out with no problems in Lion until I open Fusion. When I do I immediately get kernel panics. This has caused me to lose numerous files on the virtual machine because the last snapshot I had was at the end of November 2011. The Genius Bar told me it was being caused by Fusion and to contact VMware or Sierra Wireless for help. I would like to use the aircard on both Lion and Fusion, but if I have to I will disconnect it from Fusion if I have to and if possible. I see the connection show up in the settings on Fusion as a USB connection right before it crashes. I have attached the latest kernel panic log I have. Any help or ideas would be appreciated. Thanks!

Reply
0 Kudos
1 Solution

Accepted Solutions
jkozlow3
Enthusiast
Enthusiast
Jump to solution

I have fixed the problem on my Macbook thanks to Haihong's tip!!  I'm writing this post now using my Sprint air card which I plugged in while Fusion was running - no kernel panic!

Angelsfan1, here's what you need to do:

  1. Install SmartView 2.6
  2. Open Finder, and go to System>Library>Extensions
  3. Copy the Beceem kext to your desktop and then trash the installed one in the Extensions directory
  4. Right-click on the kext you copied and "show package contents"
  5. Open the info.plist file with TextEdit or another editor
  6. Carefully delete all 28 entries of the following 2 lines:
  7. <key>IOMatchCategory</key>
    <string>com_beceem_BeceemAppleWiMAXAdapter</string>
  8. Save the file.
  9. Install the modified kext.  For this you'll need a free kext installer.  I used Kext Drop, but there are tons of free ones out there.
  10. The newly installed kext should now appear in the System>Library>Extensions directory.
  11. Reboot
  12. Open up SmartView and connect!

Thanks again for the tip and all the research done by your team Haihong!  And thank-you for starting this thread Angelsfan1!

View solution in original post

Reply
0 Kudos
17 Replies
dariusd
VMware Employee
VMware Employee
Jump to solution

Hi angelsfan1,

Thanks for your detailed post and for providing the panic log.  The log indicates that the fault is almost certainly within the drivers for your USB wireless card.  I can give you more gory details if you want (even really gory details), but suffice it to say the USB wireless card driver is performing an operation that it should never ever perform (regardless of whether or not Fusion is running), and it's that operation which is very directly causing the host to panic.

Can you check for updated drivers for that USB wireless card?  My quick research suggests you might be out of luck -- I can't even find any supported Lion 64-bit drivers...

Cheers.

--

Darius

Reply
0 Kudos
angelsfan1
Contributor
Contributor
Jump to solution

Hi dariusd,

Thanks for you response. That is the newest driver given to me by Sierra Wireless. Sprint does not show the SmartView driver for that USB card on their website, only the older 32bit driver. I contacted Sierra Wireless and they sent me the link for the newer 64bit Lion driver. No matter how "gory" the panic logs may say the USB aircard driver may be, the fact is that the kernel panics only happen with Fusion open. I can plug in and unplug the aircard at will on 64bit Lion with Fusion closed with no panics whatsoever. The second I try to plug it in or unplug it with Fusion open, or start up Fusion while it is plugged in connected to 3G or 4G or not, it will immediately go into a kernel panic. The Genius Bar said that was the only program they were getting the kenel panic from. They said it may have to do with the sharing of network connections or USB connections. The old 32bit 3G only driver I had when using Fusion 1 through 3 worked flawlessly. The 3G-4G 64bit driver is when Fusion started doing this. I know the problem everytime is with the- beceem.BeceemAppleWiMAXAdapter. I know that is the 64bit Lion 4G WiMAX driver. What is the log stating about the problem that would not point to the Fusion program causing this? Thanks!

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee
Jump to solution

Hi angelsfan1,

Here's my analysis and commentary for the panic log you posted.  The most important part is the Backtrace, which is interpreted starting at the endmost line and reading upwards.

panic(cpu 1 caller 0xffffff80002c266d): Kernel trap at 0xffffff80002b3777, type 14=page fault, registers:
[...]
RAX: 0xffffff80ccfb1000, RBX: 0xffffff80ccfb1000, RCX: 0xffffff80fda23db4, RDX: 0x0000000000000000
[...]
CR2: 0x0000000000000008, Error code: 0x0000000000000000, Faulting CPU: 0x1

Backtrace (CPU 1), Frame : Return Address
0xffffff80fda239a0 : 0xffffff8000220702
0xffffff80fda23a20 : 0xffffff80002c266d
0xffffff80fda23bc0 : 0xffffff80002d7a1d trap_from_kernel + 38 (kernel is handling the page fault)
0xffffff80fda23be0 : 0xffffff80002b3777 lck_mtx_lock + 7 (page fault, causing panic, occurs here)
0xffffff80fda23ce0 : 0xffffff7f81190401  /\
0xffffff80fda23d20 : 0xffffff7f811904d9 /||\ (No Fusion code though.)
0xffffff80fda23d50 : 0xffffff7f8119051c  ||
0xffffff80fda23d80 : 0xffffff7f81190d2f  ||  (This is all a mystery.)
0xffffff80fda23dc0 : 0xffffff7f81199c8e  ||
0xffffff80fda23df0 : 0xffffff7f81197133  ||  (Into com.beceem.BeceemAppleWiMAXAdapter's start function...)
0xffffff80fda23e60 : 0xffffff80006267dd IOService::startCandidate(IOService*) + 105 (The kernel found a driver for a device/service and is connecting them)
0xffffff80fda23eb0 : 0xffffff800062700e IOService::probeCandidates(OSOrderedSet*) + 1932
0xffffff80fda23f30 : 0xffffff8000627389 IOService::doServiceMatch(unsigned int) + 387 (The kernel is matching up drivers to new devices/services)
0xffffff80fda23f70 : 0xffffff8000625f6f _IOConfigThread::main(void*, int) + 281 (Something changed in kernel devices/services)
0xffffff80fda23fb0 : 0xffffff8000820057 call_continuation + 23 (start here and read upwards)


Dump of assembler code for function lck_mtx_lock:
0xffffff80002b3770 <lck_mtx_lock+0>:    push   %rbp
0xffffff80002b3771 <lck_mtx_lock+1>:    mov    %rsp,%rbp
0xffffff80002b3774 <lck_mtx_lock+4>:    mov    %rdi,%rdx
0xffffff80002b3777 <lck_mtx_lock+7>:    mov    0x8(%rdx),%ecx <=== page fault occurs here

The faulting address is in CR2 in the panic log.  The page fault occurred while reading address 0x0000000000000008, so the kernel was asked to acquire a mutex at address zero (%rdx is 0).  Nothing should ever do this -- It indicates a coding error and/or memory corruption.  It'd be extremely unlikely that Fusion could be corrupting kernel memory in a way that only leads to the reproducible crash that you describe occurring with this wireless card, and there is no Fusion code anywhere in the backtrace.  The most logical conclusion is that the Beceem driver is incorrectly handling some aspect of its environment (maybe it is confused by something it sees on the host due to Fusion, or maybe something in Fusion is perturbing the timing of USB or Ethernet connect/disconnect events in a way that doesn't otherwise occur) leading it to perform this illegal operation.  Even if Fusion is necessary for the condition to occur, it's the responsibility of the Beceem folks to produce a driver that's robust against all possible conditions on the host, just as we're responsible for making our own kernel extensions robust against all possible conditions.

There's a superficially similar incompatibility with certain versions of Fusion and of Cisco's VPN Client for Mac OS.  Sometimes, a host would panic immediately after installing, uninstalling or launching Fusion while the VPN kext (com.cisco.nke.ipsec) was loaded on the host, but it is in fact a coding error in that Cisco kext which causes the panic.  Unfortunately, there is absolutely nothing we can do about that one other than to advise users to not install that piece of software and Fusion onto the same host, until/unless Cisco fixes their software... but it is painful for us because the end-user was always doing something with Fusion when the system crashed, even if the VPN was not being used at all.

Can you PM me with the link to the newer driver, or post the link here if it's not in some way confidential?  If I can get my hands on the same version of the kext, I might be able to demystify some of the mystery parts in the backtrace.

Cheers,
--
Darius

angelsfan1
Contributor
Contributor
Jump to solution

Hi dariusd,

Thanks for all of the research so far! I find it odd as well that this only happens with Fusion starting or running. I see nothing in the log that would point to Fusion, but that really is the only program that is causing this everytime. The Genius Bar and myself went through program after program to duplicate it, and checked many logs, but Fusion was the only one with problems. Below is the link to the download with the drivers that Sierra Wireless directed me to use as a fix for the 250U USB modem that uses Sprint SmartView 2.6 3G-4G. The previous 2.5 Smartview was only 3G and only tested and used on 32bit. They said that Sprint had for some reason not updated the downloads on their website, but that this was the build for SmartView 2.6 for all USB modems and aircards from Sierra Wireless and Novatel Wireless using Sprint mobile broadband.

http://www6.sprint.com/downloads/sprint_smartview/executables/SSV_Mac2.60.0027.mpkg.zip

Also, here is the link to the previous version of the software that I was using without any problems:

http://www6.sprint.com/downloads/sprint_smartview/executables/SSV_Mac2.50.0097.mpkg.zip

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee
Jump to solution

Hi angelsfan1,

The mystery part of the backtrace is revealed:

0xffffff80fda23be0 : 0xffffff80002b3777 lck_mtx_lock + 7 (page fault, causing panic, occurs here)


0xffffff80fda23ce0 : 0xffffff7f81190401
   0x12401 com_beceem_BeceemAppleWiMAXAdapter::rwm(unsigned char, unsigned int, unsigned int*, unsigned int)+119


0xffffff80fda23d20 : 0xffffff7f811904d9
   0x124d9 com_beceem_BeceemAppleWiMAXAdapter::rawWrm(unsigned int, unsigned int*, unsigned int)+55


0xffffff80fda23d50 : 0xffffff7f8119051c
   0x1251c com_beceem_BeceemAppleWiMAXAdapter::wrm(unsigned int, unsigned int*, unsigned int)+64


0xffffff80fda23d80 : 0xffffff7f81190d2f
   0x12d2f com_beceem_BeceemAppleWiMAXAdapter::putOnChipProcInReset(unsigned char)+1343

0xffffff80fda23dc0 : 0xffffff7f81199c8e
   0x1bc8e com_beceem_BeceemAppleWiMAXAdapter::stop(IOService*)+222

(At about this point, it logs a message "Start: Could not open the device exiting!!!!!")

0xffffff80fda23df0 : 0xffffff7f81197133
   0x19133 com_beceem_BeceemAppleWiMAXAdapter::start(IOService*)+835

0xffffff80fda23e60 : 0xffffff80006267dd IOService::startCandidate(IOService*) + 105 (The kernel found a driver for a device/service and is connecting them)

So the Beceem driver tries to start managing the device, but it fails (for reasons as yet unknown).  From your description, this is probably directly precipitated by Fusion's presence.  It doesn't directly cause the panic, but it does cause the driver take a different path...

Once the Beceem driver decides that it can't manage the device, it logs "Start: Could not open the device exiting!!!!!", and then immediately tries to stop the device, but it does not correctly handle shutting down a device that it has not yet finished starting – the driver tries to ensure mutual exclusion using a lock, but it looks like the lock it tries to use had not been initialized.  This is a relatively common software coding error, and is definitely a Beceem bug.  This is the direct cause of the host kernel panic.

So, there's the two-stage answer: We cause the Beceem driver to fail to start the device (details as yet unknown), which triggers a coding error in the Beceem driver that causes the host kernel panic while trying to shut down the device.

Now we have to figure out what to to about it...  The panic itself is something that Beceem will have to fix if they want their driver to be robust, but I'll look into this a bit deeper from our side and see if I can figure out why the Beceem driver specifically doesn't like starting the device while Fusion is running.

Cheers,

--

Darius

jkozlow3
Enthusiast
Enthusiast
Jump to solution

I am having this exact same issue with Kernel panics and the Beceem driver using my Sprint air card and Fusion 4.1.1. 

Are there any updates yet?  Or perhaps a way to tell Fusion to never see the card when it's inserted?

Thanks!

Reply
0 Kudos
angelsfan1
Contributor
Contributor
Jump to solution

I knew I was not the only one! I had an open support request in to them, but I let it lapse due to being out of town and not being able to fully collect all of the data they were asking for. There is deffinately a problem with the Beceem aircard driver for Sprint and running Fusion 4.1.1 at the same time.

There has to be way to tell Fusion to ignore certain USB devices or aircards, or a way needs to be developed. Either that or to make the two compatible and not enter into an endless loop of kernal panics. This problem cost me an entire Virtual Machine OS, and all of my data and files for 3 months of work. Hopefully someone at VMWare will look at these logs I have already posted and find the problem with this! We will see.

Reply
0 Kudos
dariusd
VMware Employee
VMware Employee
Jump to solution

Hi everyone,

We've now observed this panic in-house with a Sierra Wireless 250U 3G/4G Sprint USB aircard and Sprint SmartView 2.60.0027 64bit.  We will continue to investigate Fusion's role in precipitating the crash.  I currently have no information on the timeframe in which a fix might become available.  I'll post back here if I have any updates or workarounds I can share.  You should probably also remain observant for updated drivers for the 250U, since those drivers are the ones directly responsible for this panic, and it is possible that updated drivers from the hardware vendor would address the problem directly.

Thanks again for reporting this issue, and sorry for the trouble it's caused.

Cheers,

--

Darius

Reply
0 Kudos
jkozlow3
Enthusiast
Enthusiast
Jump to solution

Thanks for the update Darius.  I have the exact same issue with my Sprint U301 card which also uses the Beceem drivers.  I've tried a version of the driver from the Sprint Smartview 2.60 software as well as the beta drivers for OS X 10.7 provided by Clear on their website for their 4G air cards and both versions of the driver cause the kernel panics.

Please keep us updated if you find any workarounds.

Thanks again.

Reply
0 Kudos
haihong
Contributor
Contributor
Jump to solution

Hi Everyone,

Here's finding from investigation of our developer.

Our developer just had a look at the kernel driver that's panicing (BeceemAppleWiMAXAdapter.kext), the reason the driver is starting and  failing to start is because it doesn't use the standard IOMatchCategory,  it sets its IOMatchCategory to com_beceem_BeceemAppleWiMAXAdapter. This  means that when we grab the device (which we always do initially when  it's plugged in and the Fusion UI is running), IOKit stats both our  driver and the Beceem one. We then open() the device, and then the  Beceem one fails to start and unfortunately panics when stopping.

There's not much we can do from Fusion side.

Beceem should fix their driver  (just removing the IOMatchCategory entry from their plist would  probably be good enough).

Perhaps Apple can suggest a way of stopping all other drivers from being  start()ed, not just ones with the default IOMatchCategory.

We are in the process of contacting Apple and Sprint to report the problem.

Sprint SmartView 2.60.0027 64bit are not published on sprint website, there's a possibility Sprint is aware of this Beceem driver has issue. Before there's a solid version available, is the 32-bit version still works for you?

Thanks!

Haihong

Reply
0 Kudos
jkozlow3
Enthusiast
Enthusiast
Jump to solution

Thanks for the detailed update Haihong!

The 2.60 drivers *are* available on Sprint's site.  For some reason, they are only listed under *some* of their air cards however and not others despite the fact that many of the cards use the same Beceem driver.

For example, the 64-bit 2.60 drivers are available under the following link if you choose OS X Lion as the OS:

http://shop2.sprint.com/en/software_downloads/mobile_broadband/sierra_aircard_402.shtml

The 32-bit drivers included in SmartView 2.5 won't work under OS 10.7 which both the original poster and I am using, so we can't test those out.  As an FYI, Clear (clear.com) has some "beta" OS X 10.7 drivers listed for all of their 4G air cards and the Beceem driver posted on their site also causes the same kernel panic issue when Fusion is running.  I have not been able to find any other 64-bit Beceem drivers on the web to try however.

Thanks again.  Please let us know if you find out any additional info or workarounds.

Reply
0 Kudos
angelsfan1
Contributor
Contributor
Jump to solution

Thanks Haihong for all of the work researching the problem. Like the previous poster stated, the kernel panics happen with both the Sprint drivers and with the Clear drivers for 4g, 64bit, USB aircards. This is happening with different drivers from different companies. This did not happen with the 32bit drivers. The problem with the 32bit drivers is that they are out of date, and are only made to run 3g and not 4g on 64bit OSX Lion. They warn that the drivers were never tested on 64bit systems, so the 4g is disabled. There are no problems with any of these drivers (64bit and 32bit) unless Fusion is running. The 32bit driver will run 3g only just fine on 64bit OSX Lion with and without Fusion running. The 64bit driver works fine with no problems as well, until Fusion is opened, or is already opened when you insert the aircard. This would point that it might be more a Fusion problem than the different drivers problem. You said that Fusion will grab the inserted USB and try and start it automatically, when the OSX Lion is already doing that just fine. Is there, could there, and should there be a way that we can select the Fusion to not grab inserted USB devices unless we select it to? That would seem to be a step in the right direction. I have never had a problem with the 64bit OSX Lion running 4g, until Fusion does this "grabbing and opening" of the USB aircard and kernel panics the system. Thanks again for all of the help in a possible resolution!

Reply
0 Kudos
jkozlow3
Enthusiast
Enthusiast
Jump to solution

I have fixed the problem on my Macbook thanks to Haihong's tip!!  I'm writing this post now using my Sprint air card which I plugged in while Fusion was running - no kernel panic!

Angelsfan1, here's what you need to do:

  1. Install SmartView 2.6
  2. Open Finder, and go to System>Library>Extensions
  3. Copy the Beceem kext to your desktop and then trash the installed one in the Extensions directory
  4. Right-click on the kext you copied and "show package contents"
  5. Open the info.plist file with TextEdit or another editor
  6. Carefully delete all 28 entries of the following 2 lines:
  7. <key>IOMatchCategory</key>
    <string>com_beceem_BeceemAppleWiMAXAdapter</string>
  8. Save the file.
  9. Install the modified kext.  For this you'll need a free kext installer.  I used Kext Drop, but there are tons of free ones out there.
  10. The newly installed kext should now appear in the System>Library>Extensions directory.
  11. Reboot
  12. Open up SmartView and connect!

Thanks again for the tip and all the research done by your team Haihong!  And thank-you for starting this thread Angelsfan1!

Reply
0 Kudos
jkozlow3
Enthusiast
Enthusiast
Jump to solution

Just for clarification, do not delete the similar strings which are found underneath the CFBundleIdentifier or IOClass keys.  The string value goes with the key value directly above it and the IOMatchCategory pair is what you want to remove.

Before & after examples from one section of the file (to be repeated for a total of 28 times):

Before:

<key>CFBundleIdentifier</key>
<string>com.beceem.BeceemAppleWiMAXAdapter</string>
<key>IOClass</key>
<string>com_beceem_BeceemAppleWiMAXAdapter</string>
<key>IOMatchCategory</key>
<string>com_beceem_BeceemAppleWiMAXAdapter</string>

<key>IOProviderClass</key>
<string>IOUSBDevice</string>
<key>idProduct</key>
<integer>768</integer>
<key>idVendor</key>
<integer>6543</integer>

After:

<key>CFBundleIdentifier</key>
<string>com.beceem.BeceemAppleWiMAXAdapter</string>
<key>IOClass</key>
<string>com_beceem_BeceemAppleWiMAXAdapter</string>
<key>IOProviderClass</key>
<string>IOUSBDevice</string>
<key>idProduct</key>
<integer>768</integer>
<key>idVendor</key>
<integer>6543</integer>

Reply
0 Kudos
angelsfan1
Contributor
Contributor
Jump to solution

I have as well FINALLY fixed the problem on my MacBook Pro thanks to Haihong and jkozlow3! I followed the perfect work around instructions provided by jkozlow3 after the great information provided by Haihong and the developers, and I am up and running 4g on my 250U USB Sprint aircard on OSX Lion 64bit while Fusion is running! This has been a long and costly road for me, but I am glad for the help from vmware to find the problem that was being caused by the driver conflicting with Fusion! You guys went above and beyond for us suffering from the conflict! I followed the instructions from jkozlow3 and all is well so far! I have been doing everything that caused kernel panics before (pluggin in and removing the aircard as well as starting Fusion), and have not found a problem yet (hopefully never)! Thanks again for all of the hard work everyone, and hopefully the drivers get updated for everyone soon!

Reply
0 Kudos
jkozlow3
Enthusiast
Enthusiast
Jump to solution

Agreed - the research from the Fusion team, including the level of detail and the tip they provided really was above and beyond!

Kudos to the Fusion team.  Yet another reason I stick with Fusion over the alternative.

Reply
0 Kudos
haihong
Contributor
Contributor
Jump to solution

Thanks so much for your support to VMware product. jkozlow3, thanks a lot for trying out our developer's suggestion and posting detailed instruction of the fix! And many thanks for angelsfan1 to report the problem and verifying the solution. You guys are awsome Smiley Happy

Reply
0 Kudos