VMware

This Question is Possibly Answered

1 "correct" answer available (10 pts) 2 "helpful" answers available (6 pts)
1 2 3 4 Previous Next 46 Replies Last post: Jun 2, 2009 12:53 AM by dan81  

ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded) posted: Jul 30, 2008 5:16 AM

Click to view bigdee's profile Enthusiast
bigdee
54 posts since
Aug 15, 2006

Hi gents

I updated four of our ESX hosts to 3.5 Update 2. The hosts all had ESX 3.5 Update 1 installed with the latest patches. After a reboot of the host the following entries are added to the vmkernel logfile for all physical CPUs:

Jul 30 13:29:57 svstr18003 vmkernel: 0:00:02:23.455 cpu2:1035)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Jul 30 13:29:57 svstr18003 vmkernel: 0:00:02:24.414 cpu2:1036)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Jul 30 13:29:57 svstr18003 vmkernel: 0:00:02:24.504 cpu2:1035)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Jul 30 13:29:57 svstr18003 vmkernel: 0:00:02:24.626 cpu3:1037)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
...

Beside of that I have noticed no other issues. The hosts seem to run fine!

Has anyone experienced something similar or knows what this messages mean?

Thanks

CU


Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

1. Jul 30, 2008 6:39 AM in response to: bigdee
Click to view Texiwill's profile Guru
Texiwill
10,212 posts since
Jan 13, 2004
Hello,

What are the messages surrounding these? Generally they relate to SCSI Reservation limits exceeded, but do not always do so.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354
As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

2. Jul 30, 2008 7:20 AM in response to: bigdee
Click to view aworkman's profile Novice
aworkman
19 posts since
Nov 28, 2007
I'm having the same issues, however no SCSI Reservation messages come up near these messages.

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

3. Jul 30, 2008 7:36 AM in response to: aworkman
Click to view Texiwill's profile Guru
Texiwill
10,212 posts since
Jan 13, 2004
Hello,

How many files are on your VMFS? If its close to 32K or so, then you have reached the file limit. Otherwise I would start investigating your Data stores remote and local for fullness and errors on the remote storage devices.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354
As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

4. Jul 30, 2008 7:46 AM in response to: Texiwill
Click to view aworkman's profile Novice
aworkman
19 posts since
Nov 28, 2007
Jul 30 00:03:26 nau-esx2 vmkernel: 0:08:21:10.231 cpu11:1472)<6>Debug scsi underrun
Jul 30 01:48:33 nau-esx2 vmkernel: 0:10:06:18.113 cpu5:1620)Net: 4211: unicastAddr 00:50:56:ad:44:27;
Jul 30 01:48:33 nau-esx2 last message repeated 5 times
Jul 30 01:48:34 nau-esx2 vmkernel: 0:10:06:19.023 cpu5:1619)Net: 4211: unicastAddr 00:50:56:ad:44:27;
Jul 30 01:48:34 nau-esx2 vmkernel: 0:10:06:19.041 cpu5:1619)Net: 4211: unicastAddr 00:50:56:ad:44:27;
Jul 30 01:48:41 nau-esx2 vmkernel: 0:10:06:25.871 cpu6:1619)Net: 4211: unicastAddr 00:50:56:ad:44:27;
Jul 30 01:48:46 nau-esx2 vmkernel: 0:10:06:30.778 cpu12:1586)Net: 4211: unicastAddr 00:50:56:ad:43:36;
Jul 30 01:48:46 nau-esx2 last message repeated 5 times
Jul 30 01:48:46 nau-esx2 vmkernel: 0:10:06:30.857 cpu7:1610)Net: 4211: unicastAddr 00:50:56:ad:13:1a;
Jul 30 01:48:46 nau-esx2 last message repeated 5 times
Jul 30 01:48:47 nau-esx2 vmkernel: 0:10:06:31.651 cpu13:1583)Net: 4211: unicastAddr 00:50:56:ad:43:36;
Jul 30 01:48:47 nau-esx2 vmkernel: 0:10:06:31.677 cpu13:1583)Net: 4211: unicastAddr 00:50:56:ad:43:36;
Jul 30 01:48:47 nau-esx2 vmkernel: 0:10:06:31.774 cpu7:1609)Net: 4211: unicastAddr 00:50:56:ad:13:1a;
Jul 30 01:48:47 nau-esx2 vmkernel: 0:10:06:31.795 cpu6:1609)Net: 4211: unicastAddr 00:50:56:ad:13:1a;
Jul 30 01:48:50 nau-esx2 vmkernel: 0:10:06:34.346 cpu3:1196)<6>Debug scsi underrun
Jul 30 01:48:58 nau-esx2 vmkernel: 0:10:06:42.316 cpu14:1586)Net: 4211: unicastAddr 00:50:56:ad:43:36;
Jul 30 01:48:59 nau-esx2 vmkernel: 0:10:06:43.813 cpu5:1610)Net: 4211: unicastAddr 00:50:56:ad:13:1a;
Jul 30 04:43:47 nau-esx2 vmkernel: 0:13:01:31.333 cpu2:1281)<6>Debug scsi underrun
Jul 30 07:53:48 nau-esx2 vmkernel: 0:16:11:32.420 cpu14:1304)<6>Debug scsi underrun
Jul 30 08:23:08 nau-esx2 vmkernel: 0:16:40:52.361 cpu12:1155)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.364 cpu14:1155)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.835 cpu15:1155)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 last message repeated 5 times
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.874 cpu13:1154)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.879 cpu13:1154)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.879 cpu13:1154)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.881 cpu13:1154)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:23:09 nau-esx2 vmkernel: 0:16:40:53.896 cpu13:1154)Net: 4211: unicastAddr 00:50:56:ad:70:cc;
Jul 30 08:24:56 nau-esx2 vmkernel: 0:16:42:40.736 cpu15:1516)VSCSI: 2803: Reset request on handle 8452 (0 outstanding commands)
Jul 30 08:24:56 nau-esx2 vmkernel: 0:16:42:40.736 cpu11:1069)VSCSI: 3019: Resetting handle 8452 0/0
Jul 30 08:24:56 nau-esx2 vmkernel: 0:16:42:40.736 cpu11:1069)VSCSI: 2871: Completing reset on handle 8452 (0 outstanding commands)
Jul 30 08:34:23 nau-esx2 vmkernel: 0:16:52:07.988 cpu1:1516)VSCSI: 2803: Reset request on handle 8452 (0 outstanding commands)
Jul 30 08:34:23 nau-esx2 vmkernel: 0:16:52:07.988 cpu11:1069)VSCSI: 3019: Resetting handle 8452 0/0
Jul 30 08:34:23 nau-esx2 vmkernel: 0:16:52:07.988 cpu11:1069)VSCSI: 2871: Completing reset on handle 8452 (0 outstanding commands)
Jul 30 09:18:23 nau-esx2 vmkernel: 0:17:36:07.520 cpu7:1281)<6>Debug scsi underrun


This is on 4Gb FCP with Qlogic HBA's. Also has those scsi underruns listed. I recently upped the queue depth to 64 on the 3x esx hosts and 1x exchange server which is also using luns from the same SAN. There is no way there is 32k files on any of the vmfs luns. I have it split out into os/data/swap datastores/luns. 3x os luns 400GB each, 2x data luns 400GB each, 2x swap luns 150GB each. This works out to be c/d/e in the OS to keep our netapp snapshot size reduced by separating those different kinds of data into their own volumes(containing the luns). There is up to 20~ vm's on some of the luns(not sure if this is a big deal).

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

5. Jul 30, 2008 10:47 PM in response to: Texiwill
Click to view bigdee's profile Enthusiast
bigdee
54 posts since
Aug 15, 2006

Hi

There are no other messages around them.

Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.706 cpu0:1024)Host: 2587: irq 6 vector 0x39 (host 0x51)
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.706 cpu0:1024)Host: 2655: irq 6 vector 0x39 (host 0x51)
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.706 cpu0:1024)Host: 2587: irq 6 vector 0x39 (host 0x51)
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.710 cpu0:1024)Host: 2655: irq 4 vector 0x31 (host 0x41)
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.710 cpu0:1024)Host: 2587: irq 4 vector 0x31 (host 0x41)
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.710 cpu0:1024)Host: 2599: irq 4 happened while being masked
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.803 cpu1:1036)Config: 414: "VMOverheadGrowthLimit" = -1, Old Value: -1, (Status: 0x0)
Jul 30 12:56:03 svmhg10011 vmkernel: 0:00:02:24.985 cpu3:1035)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Jul 30 12:56:04 svmhg10011 vmkernel: 0:00:02:25.102 cpu2:1037)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)

...


These FileIO messages only appear after a reboot and after the installation of Update 2. Before that we never saw those messages. The strange thing is we have these messages in four different VMware Cluster which means four different environments and only after installing Update 2. The number of files should not be a problem. I just checked the VMFS and the numbers are beetween 35 and about 200 files. Free space on VMFS is at least 10%. I have also checked the local filesystems. They have plenty of free space too.

I am also waiting for an answer of VMware support.

Thanks.

CU


Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

6. Jul 31, 2008 5:06 AM in response to: bigdee
Click to view Texiwill's profile Guru
Texiwill
10,212 posts since
Jan 13, 2004
Hello,

I think the SR approach is the best one. They could refer to a limit in the advanced options for the host, that currently is unknown. Please let us know what VMware states.


Best regards,
Edward L. Haletky
VMware Communities User Moderator
====
Author of the book 'VMWare ESX Server in the Enterprise: Planning and Securing Virtualization Servers', Copyright 2008 Pearson Education.
CIO Virtualization Blog: http://www.cio.com/blog/index/topic/168354
As well as the Virtualization Wiki at http://www.astroarch.com/wiki/index.php/Virtualization

FileIO failed with 0x0xbad0006 (Limit exceeded) on server without any VMFS3 parition.

7. Jul 31, 2008 5:25 AM in response to: Texiwill
Click to view Erik Bussink's profile Expert
Erik Bussink
423 posts since
Jan 10, 2004

I'm also monitoring this FileIO failed with 0x0xbad0006 (Limit exceeded) error on a brand fresh install on a HP ML150 G5 (Xeon 5410). The server has a HP SmartArray E200 controller with a RAID1.

Part of my troubleshooting involved: disabling the VMFS2 vmkload_mod module; disconnecting from any iSCSI VMFS3 storage, total removal of any virtual machines (clean VMFS3) and in the end disabling the local disk VMFS3 partition.

The errors are still present even on a fresh ESX 3.5.0 Update 2 (Build 103908) install, without any VMFS3 volumes.


Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

8. Jul 31, 2008 7:46 AM in response to: bigdee
Click to view fgth's profile Lurker
fgth
1 posts since
Oct 30, 2006

Hi,

did you read the Release Notes??? (vi3_esx35u2_vc25u2_rel_notes)

Under Known Issues and Miscellaneous are some kb Article:

Benign Error Message in the /var/log/messages File of the ESX Server Host (KB 1004499)

Is it similar to your problem???


FileIO failed with 0x0xbad0006(Limit exceeded)

9. Jul 31, 2008 9:41 AM in response to: fgth
Click to view Erik Bussink's profile Expert
Erik Bussink
423 posts since
Jan 10, 2004

I did read the release notes, but I didn't see or didn't inprint the FileIO KB article.

The follow-up question, is the BC: 810: FileIO failed with 0x0xbad001f(Not supported) message the same cause/effect/warning as a BC: 814: FileIO failed with 0x0xbad0006 (Limit exceeded) or not ?


Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

10. Aug 1, 2008 4:42 AM in response to: fgth
Click to view bigdee's profile Enthusiast
bigdee
54 posts since
Aug 15, 2006
I have read the release notes but did not saw this. For us this message only occurred after an update to ESX Update 2.

We have also some of the following messages now:
Jul 31 00:14:59 esx31 vmkernel: 1:10:29:51.326 cpu0:1024)<6>Debug scsi underrun
Jul 31 00:14:59 esx31 vmkernel: 1:10:29:51.326 cpu0:1024)<6>Debug scsi underrun

I am still waiting for an answer of VMware support.

CU

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

11. Aug 1, 2008 5:32 AM in response to: bigdee
Click to view J-D's profile Hot Shot
J-D
175 posts since
Feb 22, 2007

Got the same message on 2 fresh installation on HP DL380's with management agents 8.1.0 and connected to the new HP MSA2000 SAN.

Thought it had something to do with this new type of SAN but after reading this thread probably caused by Update 2 which I also used.

Does anyone with Update 2 NOT have this issue?


Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

12. Aug 1, 2008 5:58 AM in response to: J-D
Click to view dkfbp's profile Master
dkfbp
577 posts since
May 31, 2007
I am seeing this issue on my HP BL460c.

Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:02:59.753 cpu6:1041)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:00.426 cpu4:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:00.651 cpu7:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:00.734 cpu6:1041)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:00.847 cpu7:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:06.260 cpu6:1041)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:07.033 cpu7:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:07.163 cpu7:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:07.243 cpu4:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:21.732 cpu4:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:33:19 vmcpkro55 vmkernel: 0:00:03:22.430 cpu6:1041)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:24 vmcpkro55 vmkernel: 0:00:04:41.684 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:25 vmcpkro55 vmkernel: 0:00:04:41.797 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:25 vmcpkro55 vmkernel: 0:00:04:41.809 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:25 vmcpkro55 vmkernel: 0:00:04:41.819 cpu5:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:25 vmcpkro55 vmkernel: 0:00:04:41.828 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:26 vmcpkro55 vmkernel: 0:00:04:43.476 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:27 vmcpkro55 vmkernel: 0:00:04:43.902 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:27 vmcpkro55 vmkernel: 0:00:04:44.004 cpu1:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:27 vmcpkro55 vmkernel: 0:00:04:44.074 cpu1:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:28 vmcpkro55 vmkernel: 0:00:04:45.236 cpu1:1040)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:34:28 vmcpkro55 vmkernel: 0:00:04:45.760 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:38:28 vmcpkro55 vmkernel: 0:00:08:45.591 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:38:28 vmcpkro55 vmkernel: 0:00:08:45.747 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:38:29 vmcpkro55 vmkernel: 0:00:08:45.898 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:38:29 vmcpkro55 vmkernel: 0:00:08:45.980 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)
Aug 1 14:38:29 vmcpkro55 vmkernel: 0:00:08:46.013 cpu1:1039)BC: 814: FileIO failed with 0x0xbad0006(Limit exceeded)

Best regards
Frank Brix Pedersen

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

13. Aug 1, 2008 10:12 AM in response to: J-D
Click to view Evilmikey's profile Lurker
Evilmikey
2 posts since
Jul 14, 2008
Hi,

i have an FSC BX600 S3 with 3 BX620S4 ESX 3.5 U2 with the same Problem

Have someone called the vmware support?

Attachments:

Re: ESX 3.5 Update 2 - vmkernel messages FileIO failed with 0x0xbad0006(Limit exceeded)

14. Aug 1, 2008 11:41 AM in response to: Evilmikey
Click to view BigHug's profile Enthusiast
BigHug
74 posts since
Aug 24, 2006
FYI, I have the same message on our ESX 3.5 (70356) and 3.5 U1 (98103). However, mine happens every 2 hours or so. I also have SCSI reservation conflicts problem, too. They could be related.
1 2 3 4 Previous Next Go to original post

Actions

VMware Developer

SDKs, APIs, Videos, Learn and much more in the Developer community.

Learn More

Developer Sample Code

Increase your developer productivity with VMware API sample code.

Learn More

VMworld Sessions & Labs

Online access to the latest VMworld Sessions & Labs and online services.

Learn more

Purchase PSO Credits Online

Purchase credits to redeem training and consulting services online.

Buy Now

Community Hardware Software

View reported configurations or report your own.

Learn More

VMware vSphere

Come witness the next giant leap in virtualization.

Register Today

Communities