dheerajms
Enthusiast
Enthusiast

DL585 & VI 3.0.1 & TS Cluster - Repeated PSOD's.

Hi,

I'm running ESX 3.0.1 on 2 nos of DL585 G2, each having - 2 Opteron Dual Core CPU(NUMA, Node Balanced), 16GB RAM(HP), P400 Smart Array Controller with Dual SAS HDD in RAID-1, QLogic 2340 2 Gbps HBA & Intel NC360T Dual Port NIC.

I'm running 3 Terminal Server VM's(Windows Server 2003 R2 Enterprise x64 Edition) on each ESX1 & ESX2 with NLB. Session Directory Server is running on ESX1. All Terminal Server VM's are having Dual NIC, one for NLB & other for inter-host communication. I have carved them out of vmnic1 and called those switches as NLB & TSLink. SD is on vmnic2. NLB is configured for unicast as the multicast is not possible with existing switches.

SAN consists of DS4300 connected to EXP810 and IBM 16 Port SAN Switch.

I have all firmware & ROMPAQs (DL585 BIOS, HDD, HBA, DS4300, EXP810) latest, VI 3.0.1 patches till date(25 patches), VirtualCenter 2.0.1 Patch 2.

VirtualCenter(VC) Server manages ESX hosts and the license server on VC manages the licensing. It's an VI3 Enterprise License.

No HP Insight Agents running on the ESX Hosts.

7 VM's, 7 LUN's, and one VMFS volume per LUN.

Previously, i had done Storage Partitioning and hence both ESX1 & ESX2 were able to seen the LUN's. Hence VMotion, HA & DRS were all happening with no issues.

With all this in place, both the ESX Servers crash almost together with a PSOD once in 20 - 40 hours from past couple of days.

To isolate the problem, i've separated the hosts in terms of Storage(no storage partition now), Zoning & Licensing. Not using VC Server. Disabled VMotion, HA & DRS.

Advanced configuration include:

QLogic HBA Level:

Enable LIP Reset: No

Enable LIP Full Login: Yes

Enable Target Reset: Yes

DS4300 Level:

Host type is set to LNXCL

ADT Disabled

DS4300 SP Sense Data setting applied

SAN Switch:

RSCN Suppressed on both Storage & Host Ports given to ESX Servers(Ports 0-3 in SAN Switch, 0 & 3 are DS4300 SPA1 & SPB2; 1 & 2 are ESX1 & ESX2)

ESX 3.0.1:

Disk.UseDeviceReset to 0

Disk.UseLunReset to 1

Using 2 GBPS HBA Driver

QLogic Queue Depth - 64

Some Logs:

\[root@esx01 root]# cat /var/log/vmkwarning

Apr 1 00:13:35 esx01 vmkernel: 0:06:32:13.155 cpu3:1027)WARNING: CpuSched: 7161: time went backwards by 12 usec

Apr 2 05:38:25 esx01 vmkernel: 1:11:57:03.117 cpu1:1025)WARNING: CpuSched: 7161: time went backwards by 23 usec

Apr 2 16:43:17 esx01 vmkernel: 0:00:07:35.429 cpu2:1065)WARNING: CpuSched: 7161: time went backwards by 10 usec

Apr 2 17:52:33 esx01 vmkernel: 0:01:16:51.916 cpu0:1060)WARNING: CpuSched: 7161: time went backwards by 26 usec

Apr 2 17:52:33 esx01 vmkernel: 0:01:16:51.948 cpu0:1024)WARNING: CpuSched: 7161: time went backwards by 40 usec

Apr 2 20:20:16 esx01 vmkernel: 0:00:02:19.509 cpu1:1058)WARNING: CpuSched: 7161: time went backwards by 10 usec

Apr 2 21:10:56 esx01 vmkernel: 0:00:52:59.547 cpu0:1067)WARNING: CpuSched: 7161: time went backwards by 39 usec

Apr 2 21:25:55 esx01 vmkernel: 0:01:07:58.343 cpu1:1068)WARNING: CpuSched: 7161: time went backwards by 40 usec

Apr 3 15:03:19 esx01 vmkernel: 0:00:08:51.126 cpu0:1067)WARNING: CpuSched: 7161: time went backwards by 10 usec

This warning keeps repeating often. Should i include that TSC Setting "host.TSC.noForceSync = TRUE" in each VM .vmx file?

PSOD Final Message is:

[45m[33;1mVMware ESX Server \[Releasebuild-42368][0m

Exception type 14 in world 1058:vmm0:TS04 @ 0x6bbd47

gate=0xe frame=0x348b8f0 eip=0x6bbd47 cr2=0x2807000 cr3=0x4086f000 cr4=0x668

eax=0xffffffee ebx=0xc ecx=0x3ffffc11 edx=0x0 es=0x4028 ds=0x4028

fs=0x0 gs=0x4041 ebp=0x348b950 esi=0x741fcb8 edi=0x2c07000 err=2 ef=0x11213

cpu 0 1024 console: cpu 1 1068 vmm0:TS06: cpu 2 1063 vmm0:TS05: CPU 3 1058 vmm0:TS04:

@BlueScreen: Exception type 14 in world 1058:vmm0:TS04 @ 0x6bbd47

0x348b950:\[0x6bbd47]Pkt_CopyOutMappedAndCsum+0x4f(0x2c0604c, 0x741ed04, 0xfffffffa)

0x348b9b8:\[0x6bbb5d]Pkt_CopyOutAndCsumVerify+0x271(0x2c0600a, 0x61fb3c8, 0x0)

0x348b9e0:\[0x6bae65]E1000ReceivePacket+0x231(0x66e8438, 0x348ba8c, 0x348bae8)

0x348bab0:\[0x6b6d75]E1000PollRxRing+0x1bd(0x66e8438, 0x348bae8, 0x3d76e008)

0x348bacc:\[0x6b991d]E1000DevRx+0x55(0x66e8438, 0x0, 0x348bae8)

0x348bb04:\[0x67e3be]IOChain_Resume+0x6e(0x66e8438, 0x66e8444, 0x0)

0x348bb1c:\[0x68741d]PortOutput+0x39(0x66e8438, 0x0, 0x348bbf8)

0x348bc34:\[0x68689c]EtherswitchPortDispatch+0x368(0x640b6a0, 0x348bc98, 0x66e6248)

0x348bc58:\[0x6770be]Port_Input+0x96(0x66e6248, 0x348bc98, 0x0)

0x348bcc4:\[0x670669]NetBH+0x2c1(0x0, 0x8, 0x91)

0x348bcfc:\[0x60d3c8]BHCallHandlers+0x68(0x40400010, 0x6fbc94, 0x2000000)

0x348bd1c:\[0x60d498]BH_Check+0x6c(0x1, 0x44e, 0x1935a1e)

0x348bd38:\[0x61b8bb]IDT_HandleInterrupt+0x77(0x348bd84, 0x0, 0x14a97e4)

0x348bd4c:\[0x61b9ee]IDTIntrHandler+0x52(0x348bd84, 0x4041, 0x4041)

0x348bdb0:\[0x667e4c]CommonIntr+0xc(0x3, 0x6276b820, 0x7ee2)

0x348bdcc:\[0x6fb4bb]CpuSchedIdleLoopInt+0xaf(0xca99c0, 0x0, 0x5ec)

0x348bdf8:\[0x6fb578]CpuSchedBusyWait+0x54(0x0, 0x7, 0x0)

0x348bef0:\[0x6f5722]CpuSchedDispatch+0x296(0x0, 0x0, 0xca9a4c)

0x348bf2c:\[0x6f8c98]CpuSchedWait+0x188(0x18fa450, 0xca9a4c, 0x6)

0x348bf50:\[0x6f8e7d]CpuSched_WaitDirectedYield+0x39(0xca9a4c, 0x6, 0xffffffff)

VMK uptime: 0:17:32:01.977 TSC: 139510789393287

Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1... using slot 1 of 1... log

SD, TS01 to TS03 in ESX1 & TS04 to TS06 in ESX2 are Session Directory Server & Terminal Servers.

I power on Windows VM's and i see in their System Log that it would have mentioned that the last shutdown was unexpected.

Have logged a call with VMware via HP. Yet to hear from them.

64-Bit Windows Server 2003 R2 causing problems? Have checked the RAM modules, they are fine. My worry is even after separating the hosts, both crash together! Is that because i have 3 TS running in one ESX & other 3 running out of other? I have similar switch configuration but still they are physically different?

With this info, any pointers to overcome this problem? Please let me know.

Thanks,

Dheeraj.

0 Kudos
7 Replies
dheerajms
Enthusiast
Enthusiast

Some updates to my above post...

I stopped NLB across ESX1 & ESX2. Isolated NLB hosts only on to ESX1(3 of them with Session Directory Server). Ran 3 IIS hosts on ESX2. Made TSLink(Terminal Server inter-communication link) private, not using any physical NIC's. In the next 11 hours, both the ESX Servers crashed with PSOD.

16 GB RAM in each server is paired and NUMA balanced. I would like to highlight that the following RAM modules are being used in each of the ESX Servers:

Per CPU(8 GB):

512 x 4

1 GB x 2

2 GB x 2

Like this for the II CPU. Memory is in increasing order away from the CPU.

Is the presence of Single HBA on each of these ESX Servers a problem? Should i seriously have a II HBA in place?

I'm planning to try with NUMA disabled on just one Host.

Plus, i tried pulling FC cables given to ESX1 & ESX2 one after the other. ESX & VM's were stable. Pulled both once and put it back after 30 secs, it didn't crash. VM resumed pinging.

Would common IRQs between Local HDD & NIC cause any trouble? That's what i'm noticing:

\# cat /proc/vmware/pci says...

Bus:Sl.F Vend:Dvid Subv:Subd Type Vendor ISA/irq/Vec P M Module Name Spawned bus

008:00.0 103c:3230 103c:3234 RAID 0x103c 10/ 18[/b]/0x71 A V cciss vmhba0065:01.0 14e4:164a 103c:1709 Ethernet Broadcom 10/ 24/0x81 A V bnx2 vmnic1

065:02.0 14e4:164a 103c:1709 Ethernet Broadcom 11/ 25/0x89 A V bnx2 vmnic2

066:01.0 1077:2312 1077:0100 FC QLogic 10/ 20/0x91 A V qla2300_7xx vmhba1

073:00.0 8086:105e 103c:7044 Ethernet Intel 11/ 19/0x79 A V e1000 vmnic0

073:00.1 8086:105e 103c:7044 Ethernet Intel 10/ 18[/b]/0x71 B V e1000 vmnic3

</sniped>

Posting some important lines which has 'error' term in today's CoreDump file...

0:00:00:45.924 cpu1:1034)Mod: 501: mainHeap avail after: 12920504

0:00:00:45.924 cpu1:1034)Mod: 509: no private ID set

0:00:00:46.501 cpu1:1033)Config: 384: "ShowProgress" = 0, Old Value: 0, (Status: 0x0)

0:00:00:47.724 cpu0:1024)Tcpip_Support: 2326: dst = 0xfea9, netmask = 0xffff, gw = 0x0

0:00:00:47.724 cpu0:1024)Tcpip: 2444: change route failed

0:00:00:47.909 cpu1:1033)Net: 795: Enabling NIC in the shadow vmkernel tcpip stack

0:00:00:47.909 cpu1:1033)Tcpip: 2289: Opening COSSharedPort port = 0x66da5e0

0:00:00:47.909 cpu1:1033)Tcpip_Support: 2705: No NIC support for TSO

0:00:00:47.909 cpu1:1033)Tcpip_Support: 2712: No NIC support for checksum offloading

0:00:00:47.909 cpu1:1033)Tcpip_Support: 2718: No NIC support for Scatter-Gather DMA

0:00:00:47.909 cpu1:1033)Tcpip_Support: 2756: ether attach complete

0:00:00:47.909 cpu1:1033)Tcpip_Support: 2260: index = 8834544, ip_addr = 0x0, netmask = 0x0

0:00:00:47.909 cpu1:1033)Tcpip_Support: 2306: error = 0

0:00:00:49.908 cpu0:1024)Tcpip_Support: 2260: index = 8834544, ip_addr = 0x10a0a0a, netmask = 0xffffff

0:00:00:49.908 cpu0:1024)Tcpip_Support: 2306: error = 0

0:00:00:49.913 cpu0:1024)VMNIX: NetCos: 199: checking fea9,ffff,0

0:00:00:49.913 cpu0:1024)Tcpip_Support: 2326: dst = 0x0, netmask = 0x0, gw = 0x640a0a0a

0:00:00:49.917 cpu0:1024)VMNIX: NetCos: 199: checking fea9,ffff,0

0:00:00:49.917 cpu0:1024)VMNIX: NetCos: 201: dup

0:00:00:49.917 cpu0:1024)Tcpip_Support: 2326: dst = 0xfea9, netmask = 0xffff, gw = 0x0

Any suggestions with this updated input? Please let me know.

Thanks,

Dheeraj.

0 Kudos
Paul_Lalonde
Commander
Commander

HP should be all over this. They know all about the stability issues with the DL585 G2 and ESX 3.0.x.

Get them back on the phone and keep pushing them to resolve this for you.

Paul

dheerajms
Enthusiast
Enthusiast

Some more updates for this problem...

Isolation of ESX1 & ESX2 continued. Had connected ESX1 to SAN and running 3 Windows Server 2003 R2 x86 TS VM's & 1 SD VM. ESX2 was disconnected from the SAN and running IIS on 2 Windows Server 2003 R2 x64 VM on Local HDD VMFS Volume. So vmhba2 is the Local VMFS Volume.

Setup ran without any problem for about 20 hours. Then i plugged in the FC Cable to ESX2 and hence connected it to SAN. Powered on x64 VMs on ESX2 & ESX1. After this, Servers lasted for about 1hr 15mins and crashed together.

ESX1 & ESX2 were running in Debug Mode. Could get more info from the Crash Dump as below:

ESX1[/b]

0:19:43:04.414 cpu2:1034)Resv: UndoRefCount:828: Reservation on vmhba1:0:1 held for 125 msecs for 1 reserve/release pairs!

0:19:43:36.619 cpu2:1033)World: vm 1093: Init:693: Starting world vmware-vmx with flags 4

0:19:43:36.951 cpu3:1093)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:37.453 cpu2:1093)MemSched: vm 1093: AdmitUserOverhead:3075: attempting to reserve 72K overhead memory when overhead hasn't been configured

0:19:43:37.453 cpu2:1093)MemSched: vm 1093: SetUserOverheadInt:2993: client user-overhead: 18 delta was: 18 status: 0x0

0:19:43:37.474 cpu2:1093)World: vm 1094: Init:693: Starting world vmm0:SDS_x64_SAN with flags 8

0:19:43:37.474 cpu2:1093)Sched: vm 1094: SetupVmGroup:4836: adding 'vmm0:SDS_x64_SAN': group 'host/user': cpu: shares=-1 min=-1 max=-1

0:19:43:37.474 cpu2:1093)Sched: vm 1094: SetupVmGroup:4849: renamed group 22 to vm.1093

0:19:43:37.474 cpu2:1093)Sched: vm 1094: SetupVmGroup:4863: moved group 22 to be under group 4

0:19:43:37.488 cpu2:1093)MemSched: vm 1093: SetUserOverheadInt:2993: client user-overhead: 7913 delta was: 7895 status: 0x0

0:19:43:37.488 cpu2:1093)Alloc: vm 1094: WorldInit:1682: numPhysPages=524288, numAnonPages=28762,

0:19:43:37.488 cpu2:1093)Swap: vm 1094: Extend:1426: extending swap to 2097152 KB

0:19:43:37.647 cpu1:1093)MemSched: vm 1094: AdmitSystem:5240: heap OK: avail=8349K, need=1024K

0:19:43:38.012 cpu2:1093)VSCSI: CreateDevice:2585: Creating Virtual Device for world 1094 vscsi0:0

0:19:43:38.012 cpu2:1093)SCSI: SetTargetShares:1271: Set shares value for world 1094 to 0x3e8

0:19:43:38.098 cpu2:1093)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:38.187 cpu1:1049)NUMASched: Epoch:3770: snapshot failed: Busy

0:19:43:38.200 cpu2:1093)World: vm 1095: Init:693: Starting world vmware-vmx with flags 44

0:19:43:38.201 cpu3:1095)World: vm 1096: Init:693: Starting world vmware-vmx with flags 44

0:19:43:38.202 cpu3:1095)World: vm 1097: Init:693: Starting world vmware-vmx with flags 44

0:19:43:38.203 cpu2:1094)CpuSched: vm 1094: StartWorld:13422: VMK IDT offset = 0x528000, pte = 11328021, stackTop = 0x351c000

0:19:43:38.203 cpu2:1094)Init: nit:677: Received INIT from world 1094

0:19:43:38.238 cpu3:1094)VSCSI: RegisterVMMDevice:1352: ai = 16, vAdapt = 0, vTarget = 0

0:19:43:38.292 cpu3:1094)MemSched: vm 1094: MonitorStarted:6038: valid=1, vmmStarted=1, mapped=0/570246, overhd=8033

0:19:43:38.294 cpu3:1094)Net: Net_VMMConnect:51: switch name portgroup5 from world 1094

0:19:43:38.294 cpu3:1094)Net: Portset_ConnectPort:1076: newID 0x3000011, newIDIdx 0x11, psMask 0x3f, newPort 0x7f1b3c4, portsInUse 4, portCfg 0x7f2e708

0:19:43:38.294 cpu3:1094)Net: Port_AssociateVmmWorldGroup:645: world 1094 vmm0:SDS_x64_SAN ---> port 0x3000011 on vSwitch2

0:19:43:38.294 cpu3:1094)Net: L2Sec_SetFixedClientHWID:48: 0x3000011: 00:00:00:00:00:00 -> 00:50:56:a6:77:de

0:19:43:38.294 cpu3:1094)Net: Net_Connect:852: connected to net portgroup5, portset 0x7c111e8, PortID = 0x3000011, status 0x0

0:19:43:38.295 cpu3:1094)E1000: E1000Dev_Enable:3660: 0x3000011: 12 0

0:19:43:38.295 cpu3:1094)Net: EnablePort:1596: 0x3000011 on vSwitch2 portgroup5

0:19:43:42.144 cpu1:1033)World: vm 1098: Init:693: Starting world vmware-vmx with flags 4

0:19:43:42.188 cpu0:1098)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:42.629 cpu2:1098)MemSched: vm 1098: AdmitUserOverhead:3075: attempting to reserve 72K overhead memory when overhead hasn't been configured

0:19:43:42.629 cpu2:1098)MemSched: vm 1098: SetUserOverheadInt:2993: client user-overhead: 18 delta was: 18 status: 0x0

0:19:43:42.650 cpu1:1098)World: vm 1099: Init:693: Starting world vmm0:TS01_x64_SAN with flags 8

0:19:43:42.651 cpu1:1098)Sched: vm 1099: SetupVmGroup:4836: adding 'vmm0:TS01_x64_SAN': group 'host/user': cpu: shares=-1 min=-1 max=-1

0:19:43:42.651 cpu1:1098)Sched: vm 1099: SetupVmGroup:4849: renamed group 23 to vm.1098

0:19:43:42.651 cpu1:1098)Sched: vm 1099: SetupVmGroup:4863: moved group 23 to be under group 4

0:19:43:42.663 cpu1:1098)MemSched: vm 1098: SetUserOverheadInt:2993: client user-overhead: 12009 delta was: 11991 status: 0x0

0:19:43:42.663 cpu1:1098)Alloc: vm 1099: WorldInit:1682: numPhysPages=1048576, numAnonPages=28770,

0:19:43:42.663 cpu1:1098)Swap: vm 1099: Extend:1426: extending swap to 4194304 KB

0:19:43:42.965 cpu0:1098)MemSched: vm 1099: AdmitSystem:5240: heap OK: avail=8345K, need=1024K

0:19:43:43.231 cpu0:1098)VSCSI: CreateDevice:2585: Creating Virtual Device for world 1099 vscsi0:0

0:19:43:43.232 cpu0:1098)SCSI: SetTargetShares:1271: Set shares value for world 1099 to 0x3e8

0:19:43:43.260 cpu0:1098)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:43.534 cpu3:1098)World: vm 1100: Init:693: Starting world vmware-vmx with flags 44

0:19:43:43.535 cpu0:1100)World: vm 1101: Init:693: Starting world vmware-vmx with flags 44

0:19:43:43.536 cpu0:1100)World: vm 1102: Init:693: Starting world vmware-vmx with flags 44

0:19:43:43.537 cpu0:1099)CpuSched: vm 1099: StartWorld:13422: VMK IDT offset = 0x528000, pte = 11328021, stackTop = 0x3530000

0:19:43:43.537 cpu0:1099)Init: nit:677: Received INIT from world 1099

0:19:43:43.606 cpu1:1099)VSCSI: RegisterVMMDevice:1352: ai = 16, vAdapt = 0, vTarget = 0

0:19:43:43.646 cpu1:1099)MemSched: vm 1099: MonitorStarted:6038: valid=1, vmmStarted=1, mapped=0/1099662, overhd=12154

0:19:43:43.647 cpu1:1099)Net: Net_VMMConnect:51: switch name portgroup7 from world 1099

0:19:43:43.647 cpu1:1099)Net: Portset_ConnectPort:1076: newID 0x400001d, newIDIdx 0x1d, psMask 0x3f, newPort 0x7f3a684, portsInUse 3, portCfg 0x7f48b20

0:19:43:43.647 cpu1:1099)Net: Port_AssociateVmmWorldGroup:645: world 1099 vmm0:TS01_x64_SAN ---> port 0x400001d on vSwitch3

0:19:43:43.647 cpu1:1099)Net: L2Sec_SetFixedClientHWID:48: 0x400001d: 00:00:00:00:00:00 -> 00:50:56:a6:01:ca

0:19:43:43.647 cpu1:1099)Net: Net_Connect:852: connected to net portgroup7, portset 0x7c168c0, PortID = 0x400001d, status 0x0

0:19:43:43.648 cpu1:1099)E1000: E1000Dev_Enable:3660: 0x400001d: 13 1

0:19:43:43.649 cpu1:1099)Net: EnablePort:1596: 0x400001d on vSwitch3 portgroup7

0:19:43:43.649 cpu1:1099)Net: Net_VMMConnect:51: switch name portgroup3 from world 1099

0:19:43:43.649 cpu1:1099)Net: Portset_ConnectPort:1076: newID 0x2000020, newIDIdx 0x20, psMask 0x3f, newPort 0x7f06cb8, portsInUse 5, portCfg 0x7f14098

0:19:43:43.649 cpu1:1099)Net: Port_AssociateVmmWorldGroup:645: world 1099 vmm0:TS01_x64_SAN ---> port 0x2000020 on vSwitch1

0:19:43:43.649 cpu1:1099)Net: L2Sec_SetFixedClientHWID:48: 0x2000020: 00:00:00:00:00:00 -> 00:50:56:a6:1c:84

0:19:43:43.650 cpu1:1099)Net: Net_Connect:852: connected to net portgroup3, portset 0x7c0bb10, PortID = 0x2000020, status 0x0

0:19:43:43.650 cpu1:1099)E1000: E1000Dev_Enable:3660: 0x2000020: 12 0

0:19:43:43.650 cpu1:1099)Net: EnablePort:1596: 0x2000020 on vSwitch1 portgroup3

0:19:43:45.711 cpu2:1034)World: vm 1103: Init:693: Starting world vmware-vmx with flags 4

0:19:43:45.755 cpu2:1103)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:46.037 cpu2:1097)World64: GetVMM64InitInfo:933: vmm64->cr3 = 0x3ee3e000, vmkContext = 0x8e1db8c, vmkCR3 = 0x7ce11000switchPageMPN = 0x1 switchPageL4E = 0x386f43003

0:19:43:46.149 cpu0:1103)MemSched: vm 1103: AdmitUserOverhead:3075: attempting to reserve 72K overhead memory when overhead hasn't been configured

0:19:43:46.149 cpu0:1103)MemSched: vm 1103: SetUserOverheadInt:2993: client user-overhead: 18 delta was: 18 status: 0x0

0:19:43:46.169 cpu2:1103)World: vm 1104: Init:693: Starting world vmm0:TS02_x64_SAN with flags 8

0:19:43:46.169 cpu2:1103)Sched: vm 1104: SetupVmGroup:4836: adding 'vmm0:TS02_x64_SAN': group 'host/user': cpu: shares=-1 min=-1 max=-1

0:19:43:46.169 cpu2:1103)Sched: vm 1104: SetupVmGroup:4849: renamed group 24 to vm.1103

0:19:43:46.169 cpu2:1103)Sched: vm 1104: SetupVmGroup:4863: moved group 24 to be under group 4

0:19:43:46.173 cpu2:1103)Res3: RecalculateNFCR:4898: Patching free resources (14) with calculated value 15

0:19:43:46.181 cpu2:1103)FS3: DiskLockLock:1714: Checking if lock holders are live for lock [type 10c00001 offset 69369856 v 21, hb offset 3767296

gen 20, mode 1, owner 4612725e-2b31b224-0adf-0016358216c0 mtime 213]

0:19:43:46.182 cpu2:1103)FS3: CheckHostPulseAndLock:752: Clearing stale owner for lock[type 10c00001 offset 69369856 v 21, hb offset 3767296

gen 20, mode 1, owner 4612725e-2b31b224-0adf-0016358216c0 mtime 213]

0:19:43:46.183 cpu2:1103)MemSched: vm 1103: SetUserOverheadInt:2993: client user-overhead: 12009 delta was: 11991 status: 0x0

0:19:43:46.184 cpu2:1103)Alloc: vm 1104: WorldInit:1682: numPhysPages=1048576, numAnonPages=28770,

0:19:43:46.184 cpu2:1103)Swap: vm 1104: Extend:1426: extending swap to 4194304 KB

0:19:43:46.184 cpu2:1103)MemSched: vm 1104: AdmitSystem:5240: heap OK: avail=8341K, need=1024K

0:19:43:46.191 cpu0:1049)NUMASched: Epoch:3770: snapshot failed: Busy

0:19:43:46.368 cpu2:1103)FS3: DiskLockLock:1714: Checking if lock holders are live for lock [type 10c00001 offset 69357568 v 24, hb offset 3767296

gen 20, mode 1, owner 4612725e-2b31b224-0adf-0016358216c0 mtime 213]

0:19:43:46.369 cpu2:1103)FS3: CheckHostPulseAndLock:752: Clearing stale owner for lock[type 10c00001 offset 69357568 v 24, hb offset 3767296

gen 20, mode 1, owner 4612725e-2b31b224-0adf-0016358216c0 mtime 213]

0:19:43:46.419 cpu0:1103)VSCSI: CreateDevice:2585: Creating Virtual Device for world 1104 vscsi0:0

0:19:43:46.419 cpu0:1103)SCSI: SetTargetShares:1271: Set shares value for world 1104 to 0x3e8

0:19:43:46.424 cpu2:1103)FS3: DiskLockLock:1714: Checking if lock holders are live for lock [type 10c00001 offset 69361664 v 26, hb offset 3767296

gen 20, mode 1, owner 4612725e-2b31b224-0adf-0016358216c0 mtime 213]

0:19:43:46.425 cpu2:1103)FS3: CheckHostPulseAndLock:752: Clearing stale owner for lock[type 10c00001 offset 69361664 v 26, hb offset 3767296

gen 20, mode 1, owner 4612725e-2b31b224-0adf-0016358216c0 mtime 213]

0:19:43:46.456 cpu0:1103)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:46.733 cpu0:1103)World: vm 1105: Init:693: Starting world vmware-vmx with flags 44

0:19:43:46.734 cpu2:1105)World: vm 1106: Init:693: Starting world vmware-vmx with flags 44

0:19:43:46.736 cpu2:1105)World: vm 1107: Init:693: Starting world vmware-vmx with flags 44

0:19:43:46.736 cpu0:1104)CpuSched: vm 1104: StartWorld:13422: VMK IDT offset = 0x528000, pte = 11328021, stackTop = 0x3544000

0:19:43:46.737 cpu0:1104)Init: nit:677: Received INIT from world 1104

0:19:43:46.834 cpu2:1104)VSCSI: RegisterVMMDevice:1352: ai = 16, vAdapt = 0, vTarget = 0

0:19:43:46.889 cpu2:1104)MemSched: vm 1104: MonitorStarted:6038: valid=1, vmmStarted=1, mapped=0/1099662, overhd=12153

0:19:43:46.890 cpu2:1104)Net: Net_VMMConnect:51: switch name portgroup7 from world 1104

0:19:43:46.890 cpu2:1104)Net: Portset_ConnectPort:1076: newID 0x400001e, newIDIdx 0x1e, psMask 0x3f, newPort 0x7f3ace0, portsInUse 4, portCfg 0x7f48b20

0:19:43:46.890 cpu2:1104)Net: Port_AssociateVmmWorldGroup:645: world 1104 vmm0:TS02_x64_SAN ---> port 0x400001e on vSwitch3

0:19:43:46.891 cpu2:1104)Net: L2Sec_SetFixedClientHWID:48: 0x400001e: 00:00:00:00:00:00 -> 00:50:56:a6:3e:1a

0:19:43:46.891 cpu2:1104)Net: Net_Connect:852: connected to net portgroup7, portset 0x7c168c0, PortID = 0x400001e, status 0x0

0:19:43:46.891 cpu2:1104)E1000: E1000Dev_Enable:3660: 0x400001e: 13 1

0:19:43:46.892 cpu2:1104)Net: EnablePort:1596: 0x400001e on vSwitch3 portgroup7

0:19:43:46.892 cpu2:1104)Net: Net_VMMConnect:51: switch name portgroup3 from world 1104

0:19:43:46.893 cpu2:1104)Net: Portset_ConnectPort:1076: newID 0x2000021, newIDIdx 0x21, psMask 0x3f, newPort 0x7f07314, portsInUse 6, portCfg 0x7f14098

0:19:43:46.893 cpu2:1104)Net: Port_AssociateVmmWorldGroup:645: world 1104 vmm0:TS02_x64_SAN ---> port 0x2000021 on vSwitch1

0:19:43:46.893 cpu2:1104)Net: L2Sec_SetFixedClientHWID:48: 0x2000021: 00:00:00:00:00:00 -> 00:50:56:a6:10:d5

0:19:43:46.893 cpu2:1104)Net: Net_Connect:852: connected to net portgroup3, portset 0x7c0bb10, PortID = 0x2000021, status 0x0

0:19:43:46.894 cpu3:1104)E1000: E1000Dev_Enable:3660: 0x2000021: 12 0

0:19:43:46.895 cpu3:1104)Net: EnablePort:1596: 0x2000021 on vSwitch1 portgroup3

0:19:43:49.638 cpu2:1094)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:43:50.066 cpu0:1033)World: vm 1108: Init:693: Starting world vmware-vmx with flags 4

0:19:43:50.120 cpu0:1108)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:50.625 cpu0:1108)MemSched: vm 1108: AdmitUserOverhead:3075: attempting to reserve 72K overhead memory when overhead hasn't been configured

0:19:43:50.625 cpu0:1108)MemSched: vm 1108: SetUserOverheadInt:2993: client user-overhead: 18 delta was: 18 status: 0x0

0:19:43:50.652 cpu0:1108)World: vm 1109: Init:693: Starting world vmm0:TS03_x64_SAN with flags 8

0:19:43:50.652 cpu0:1108)Sched: vm 1109: SetupVmGroup:4836: adding 'vmm0:TS03_x64_SAN': group 'host/user': cpu: shares=-1 min=-1 max=-1

0:19:43:50.652 cpu0:1108)Sched: vm 1109: SetupVmGroup:4849: renamed group 25 to vm.1108

0:19:43:50.652 cpu0:1108)Sched: vm 1109: SetupVmGroup:4863: moved group 25 to be under group 4

0:19:43:50.674 cpu0:1108)MemSched: vm 1108: SetUserOverheadInt:2993: client user-overhead: 12009 delta was: 11991 status: 0x0

0:19:43:50.674 cpu0:1108)Alloc: vm 1109: WorldInit:1682: numPhysPages=1048576, numAnonPages=28770,

0:19:43:50.674 cpu0:1108)Swap: vm 1109: Extend:1426: extending swap to 4194304 KB

0:19:43:51.028 cpu1:1108)MemSched: vm 1109: AdmitSystem:5240: heap OK: avail=8337K, need=1024K

0:19:43:51.198 cpu1:1102)World64: GetVMM64InitInfo:933: vmm64->cr3 = 0x41820000, vmkContext = 0x8f1ecd4, vmkCR3 = 0x7a616000switchPageMPN = 0x1 switchPageL4E = 0x39b646003

0:19:43:51.671 cpu0:1108)VSCSI: CreateDevice:2585: Creating Virtual Device for world 1109 vscsi0:0

0:19:43:51.671 cpu0:1108)SCSI: SetTargetShares:1271: Set shares value for world 1109 to 0x3e8

0:19:43:51.709 cpu0:1108)mmap: LinuxMem: Mmap2:230: UNSUPPORTED flags (0x1) -> EINVAL

0:19:43:52.048 cpu3:1108)World: vm 1110: Init:693: Starting world vmware-vmx with flags 44

0:19:43:52.051 cpu0:1110)World: vm 1111: Init:693: Starting world vmware-vmx with flags 44

0:19:43:52.052 cpu0:1110)World: vm 1112: Init:693: Starting world vmware-vmx with flags 44

0:19:43:52.054 cpu0:1109)CpuSched: vm 1109: StartWorld:13422: VMK IDT offset = 0x528000, pte = 11328021, stackTop = 0x3558000

0:19:43:52.054 cpu0:1109)Init: nit:677: Received INIT from world 1109

0:19:43:52.136 cpu0:1109)VSCSI: RegisterVMMDevice:1352: ai = 16, vAdapt = 0, vTarget = 0

0:19:43:52.379 cpu0:1109)MemSched: vm 1109: MonitorStarted:6038: valid=1, vmmStarted=1, mapped=0/624906, overhd=12159

0:19:43:52.380 cpu0:1109)Net: Net_VMMConnect:51: switch name portgroup7 from world 1109

0:19:43:52.380 cpu0:1109)Net: Portset_ConnectPort:1076: newID 0x400001f, newIDIdx 0x1f, psMask 0x3f, newPort 0x7f3b33c, portsInUse 5, portCfg 0x7f48b20

0:19:43:52.380 cpu0:1109)Net: Port_AssociateVmmWorldGroup:645: world 1109 vmm0:TS03_x64_SAN ---> port 0x400001f on vSwitch3

0:19:43:52.380 cpu0:1109)Net: L2Sec_SetFixedClientHWID:48: 0x400001f: 00:00:00:00:00:00 -> 00:50:56:a6:3b:31

0:19:43:52.380 cpu0:1109)Net: Net_Connect:852: connected to net portgroup7, portset 0x7c168c0, PortID = 0x400001f, status 0x0

0:19:43:52.381 cpu0:1109)E1000: E1000Dev_Enable:3660: 0x400001f: 13 1

0:19:43:52.381 cpu0:1109)Net: EnablePort:1596: 0x400001f on vSwitch3 portgroup7

0:19:43:52.419 cpu1:1109)Net: Net_VMMConnect:51: switch name portgroup3 from world 1109

0:19:43:52.419 cpu1:1109)Net: Portset_ConnectPort:1076: newID 0x2000022, newIDIdx 0x22, psMask 0x3f, newPort 0x7f07970, portsInUse 7, portCfg 0x7f14098

0:19:43:52.419 cpu1:1109)Net: Port_AssociateVmmWorldGroup:645: world 1109 vmm0:TS03_x64_SAN ---> port 0x2000022 on vSwitch1

0:19:43:52.419 cpu1:1109)Net: L2Sec_SetFixedClientHWID:48: 0x2000022: 00:00:00:00:00:00 -> 00:50:56:a6:58:45

0:19:43:52.419 cpu1:1109)Net: Net_Connect:852: connected to net portgroup3, portset 0x7c0bb10, PortID = 0x2000022, status 0x0

0:19:43:52.447 cpu0:1109)E1000: E1000Dev_Enable:3660: 0x2000022: 12 0

0:19:43:52.448 cpu0:1109)Net: EnablePort:1596: 0x2000022 on vSwitch1 portgroup3

0:19:43:56.039 cpu0:1099)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:43:57.830 cpu3:1107)World64: GetVMM64InitInfo:933: vmm64->cr3 = 0x7d68f000, vmkContext = 0x4031ed54, vmkCR3 = 0x3d1e9000switchPageMPN = 0x1 switchPageL4E = 0x457e93003

0:19:44:02.507 cpu0:1112)World64: GetVMM64InitInfo:933: vmm64->cr3 = 0x7ad77000, vmkContext = 0x9024cf4, vmkCR3 = 0x3ea3c000switchPageMPN = 0x1 switchPageL4E = 0x387d42003

0:19:44:02.754 cpu3:1104)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:44:07.109 cpu1:1109)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:45:19.992 cpu3:1094)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:30.627 cpu3:1079)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:30.673 cpu3:1079)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:32.971 cpu2:1079)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:34.634 cpu2:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:38.437 cpu2:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:38.454 cpu2:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:38.793 cpu2:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:39.053 cpu2:1108)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:39.143 cpu2:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:45.727 cpu1:1086)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:45.919 cpu1:1099)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:46.036 cpu1:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:46.607 cpu3:1104)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:46.768 cpu3:1074)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:47.189 cpu3:1094)E1000: CheckIPv4Checksum:1143: NOT_TESTED /build/mts/release/bora-42368/bora/vmkernel/net/e1000_vmkdev.c:1143

0:19:45:50.172 cpu0:1099)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:52.276 cpu0:1109)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:45:57.608 cpu0:1072)E1000: E1000PollRxRing:2237: PollRxRing: need to set rx empty interrupt

0:19:46:51.036 cpu3:1104)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:46:51.265 cpu1:1109)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:48:54.151 cpu1:1099)FS: DoAsyncIO:271: done delaying; exiting to fastpath

0:19:49:06.899 cpu0:1099)FS: DoAsyncIO:271: done delaying; exiting to fastpath

0:19:51:01.126 cpu3:1094)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:51:24.732 cpu2:1096)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:27.466 cpu2:1094)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:37.670 cpu2:1104)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:37.677 cpu2:1079)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:39.291 cpu3:1096)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:42.082 cpu0:1072)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:43.222 cpu2:1093)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:43.251 cpu2:1093)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:43.285 cpu2:1093)Alloc: vm 1094: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:51:49.563 cpu2:1034)LVM: ValidateDevice:1618: \[vmhba2:0:0:8] Mismatch between stored disk ID and actual disk ID

0:19:51:49.718 cpu2:1034)LVM: ValidateDevice:1618: \[vmhba2:0:0:8] Mismatch between stored disk ID and actual disk ID

0:19:52:43.572 cpu1:1109)FS: DoAsyncIO:271: done delaying; exiting to fastpath

0:19:53:20.129 cpu2:1034)LVM: ValidateDevice:1618: \[vmhba2:0:0:8] Mismatch between stored disk ID and actual disk ID

0:19:53:20.147 cpu2:1034)LVM: ValidateDevice:1618: \[vmhba2:0:0:8] Mismatch between stored disk ID and actual disk ID

0:19:53:48.546 cpu3:1033)LVM: ValidateDevice:1618: \[vmhba2:0:0:8] Mismatch between stored disk ID and actual disk ID

0:19:53:48.691 cpu3:1033)LVM: ValidateDevice:1618: \[vmhba2:0:0:8] Mismatch between stored disk ID and actual disk ID

0:19:54:28.007 cpu3:1104)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:54:39.592 cpu1:1099)FS: DoAsyncIO:271: done delaying; exiting to fastpath

0:19:55:15.710 cpu2:1034)Resv: UndoRefCount:828: Reservation on vmhba1:0:3 held for 146 msecs for 1 reserve/release pairs!

0:19:57:03.022 cpu0:1099)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:57:18.533 cpu2:1079)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:24.850 cpu0:1099)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:24.852 cpu0:1072)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:24.853 cpu0:1072)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:32.364 cpu1:1099)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:33.971 cpu0:1099)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:33.971 cpu0:1099)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:33.971 cpu0:1099)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:57:33.972 cpu0:1099)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:01.699 cpu3:1109)SCSI: InitialErrorCheckOfCommand:97: INQUIRY request with EVPD set

0:19:58:04.641 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:04.641 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:04.641 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:10.673 cpu0:1049)MemSched: vm 1109: MigrateVMMCallback:6129: migrated vmm to new node

0:19:58:19.672 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.672 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.672 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.673 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.673 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.673 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.673 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.674 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.674 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.674 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.674 cpu2:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.681 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.682 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.682 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.682 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.683 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.683 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.683 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.684 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.684 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.684 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.685 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:19:58:19.685 cpu3:1109)Alloc: vm 1099: PageFaultInt:3361: swapin conflict detected: non-blocking case

0:20:05:32.763 cpu2:1079)Net: Net_VMMPortDisable:400: port 0x400001c from world 1079

0:20:05:32.763 cpu2:1079)Net: Vmxnet2VMKDevDisable:3505: saved ring indices: rxRings: 15,0 txRing: 84

0:20:05:32.763 cpu2:1079)Net: Vmxnet2VMKDevUnpinTxBuffers:2395: unmapping tx buffers: 50 pages

0:20:05:32.763 cpu2:1079)Net: Vmxnet2UnmapPA:2263: unmapping PA: 129 pages

0:20:05:32.763 cpu2:1079)Net: DisablePort:1799: 0x400001c on vSwitch3 portgroup7

0:20:05:32.766 cpu2:1079)Net: Net_VMMDisconnect:79: port 0x400001c from world 1079

0:20:05:32.767 cpu2:1079)Net: PortDisassociateVmmWorld:733: world 1079 vmm0:TS2_x86 -X-> port 0x400001c on vSwitch3

0:20:05:32.767 cpu2:1079)Net: PortDisassociateVmmWorldGroup:683: world 1079 vmm0:TS2_x86 -X-> port 0x400001c on vSwitch3

0:20:05:32.767 cpu2:1079)Net: NetDisconnect:934: disconnected from net vSwitch3, PortID = 0x400001c

0:20:05:32.767 cpu2:1079)Net: Net_VMMConnect:51: switch name portgroup7 from world 1079

0:20:05:32.767 cpu2:1079)Net: Portset_ConnectPort:1076: newID 0x4000020, newIDIdx 0x20, psMask 0x3f, newPort 0x7f3b998, portsInUse 5, portCfg 0x7f48b20

0:20:05:32.768 cpu2:1079)Net: Port_AssociateVmmWorldGroup:645: world 1079 vmm0:TS2_x86 ---> port 0x4000020 on vSwitch3

0:20:05:32.768 cpu2:1079)Net: L2Sec_SetFixedClientHWID:48: 0x4000020: 00:00:00:00:00:00 -> 00:0c:29:16:8b:1e

0:20:05:32.768 cpu2:1079)Net: Net_Connect:852: connected to net portgroup7, portset 0x7c168c0, PortID = 0x4000020, status 0x0

0:20:05:32.810 cpu2:1079)Net: VlanceVMKDev_Enable:614: 0x4000020: 13

0:20:05:32.810 cpu2:1079)Net: EnablePort:1596: 0x4000020 on vSwitch3 portgroup7

0:20:05:32.870 cpu3:1079)Net: Net_VMMPortDisable:400: port 0x200001f from world 1079

0:20:05:32.870 cpu3:1079)Net: Vmxnet2VMKDevDisable:3505: saved ring indices: rxRings: 22,0 txRing: 87

0:20:05:32.870 cpu3:1079)Net: Vmxnet2VMKDevUnpinTxBuffers:2395: unmapping tx buffers: 50 pages

0:20:05:32.870 cpu3:1079)Net: Vmxnet2UnmapPA:2263: unmapping PA: 129 pages

0:20:05:32.871 cpu3:1079)Net: DisablePort:1799: 0x200001f on vSwitch1 portgroup3

0:20:05:32.874 cpu3:1079)Net: Net_VMMDisconnect:79: port 0x200001f from world 1079

0:20:05:32.874 cpu3:1079)Net: PortDisassociateVmmWorld:733: world 1079 vmm0:TS2_x86 -X-> port 0x200001f on vSwitch1

0:20:05:32.874 cpu3:1079)Net: PortDisassociateVmmWorldGroup:683: world 1079 vmm0:TS2_x86 -X-> port 0x200001f on vSwitch1

0:20:05:32.874 cpu3:1079)Net: NetDisconnect:934: disconnected from net vSwitch1, PortID = 0x200001f

0:20:05:32.875 cpu3:1079)Net: Net_VMMConnect:51: switch name portgroup3 from world 1079

0:20:05:32.875 cpu3:1079)Net: Portset_ConnectPort:1076: newID 0x2000023, newIDIdx 0x23, psMask 0x3f, newPort 0x7f07fcc, portsInUse 7, portCfg 0x7f14098

0:20:05:32.875 cpu3:1079)Net: Port_AssociateVmmWorldGroup:645: world 1079 vmm0:TS2_x86 ---> port 0x2000023 on vSwitch1

0:20:05:32.875 cpu3:1079)Net: L2Sec_SetFixedClientHWID:48: 0x2000023: 00:00:00:00:00:00 -> 00:0c:29:16:8b:14

0:20:05:32.875 cpu3:1079)Net: Net_Connect:852: connected to net portgroup3, portset 0x7c0bb10, PortID = 0x2000023, status 0x0

0:20:05:32.917 cpu3:1079)Net: VlanceVMKDev_Enable:614: 0x2000023: 12

0:20:05:32.917 cpu3:1079)Net: EnablePort:1596: 0x2000023 on vSwitch1 portgroup3

0:20:05:34.034 cpu3:1079)World: vm 1079: Exit:3867: Killing self with status=0x0:Success

0:20:05:34.034 cpu1:1034)Net: Net_WorldPreCleanup:463: worldID 1079 has 2 associated ports

0:20:05:34.034 cpu2:1082)World: vm 1082: Exit:3867: Killing self with status=0x0:Success

0:20:05:34.034 cpu1:1034)Net: VlanceVMKDevDisable:575: 0x4000020

0:20:05:34.034 cpu1:1034)Net: DisablePort:1799: 0x4000020 on vSwitch3 portgroup7

0:20:05:34.035 cpu1:1034)Net: PortDisassociateVmmWorld:733: world 1079 vmm0:TS2_x86 -X-> port 0x4000020 on vSwitch3

0:20:05:34.035 cpu1:1034)Net: PortDisassociateVmmWorldGroup:683: world 1079 vmm0:TS2_x86 -X-> port 0x4000020 on vSwitch3

0:20:05:34.035 cpu1:1034)Net: NetDisconnect:934: disconnected from net vSwitch3, PortID = 0x4000020

0:20:05:34.035 cpu1:1034)Net: VlanceVMKDevDisable:575: 0x2000023

0:20:05:34.035 cpu1:1034)Net: DisablePort:1799: 0x2000023 on vSwitch1 portgroup3

0:20:05:34.035 cpu1:1034)Net: PortDisassociateVmmWorld:733: world 1079 vmm0:TS2_x86 -X-> port 0x2000023 on vSwitch1

0:20:05:34.035 cpu1:1034)Net: PortDisassociateVmmWorldGroup:683: world 1079 vmm0:TS2_x86 -X-> port 0x2000023 on vSwitch1

0:20:05:34.035 cpu0:1033)World: VMKStackCleanup:5723: least stack avail=6296 (world 1082:vcpu-0:TS2_x86)

0:20:05:34.036 cpu1:1034)Net: NetDisconnect:934: disconnected from net vSwitch1, PortID = 0x2000023

0:20:05:34.036 cpu1:1034)World: vm 1079: Reap:4368: reapCount = 1, refCount = 0, hostCount = 1, scsiActive = 0

0:20:05:34.144 cpu2:1081)World: vm 1081: Exit:3867: Killing self with status=0x0:Success

0:20:05:34.169 cpu3:1078)VSCSI: DeallocHandle:3685: Can't find handleID 8193 in world with ID 1079.

0:20:05:34.182 cpu3:1083)World: vm 1083: Exit:3867: Killing self with status=0x0:Success

0:20:05:34.183 cpu3:1084)World: vm 1084: Exit:3867: Killing self with status=0x0:Success

0:20:05:36.854 cpu3:1080)World: vm 1080: Exit:3867: Killing self with status=0x0:Success

0:20:05:36.875 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x4b ra=0x642f12

0:20:05:36.875 cpu2:1078)World: vm 1078: Exit:3867: Killing self with status=0x0:Success

0:20:05:36.876 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x4c ra=0x642f12

0:20:05:36.910 cpu1:1049)NUMASched: Epoch:3770: snapshot failed: Busy

0:20:05:40.437 cpu0:1072)Net: Net_VMMPortDisable:400: port 0x400001a from world 1072

0:20:05:40.437 cpu0:1072)Net: Vmxnet2VMKDevDisable:3505: saved ring indices: rxRings: 70,0 txRing: 65

0:20:05:40.437 cpu0:1072)Net: Vmxnet2VMKDevUnpinTxBuffers:2395: unmapping tx buffers: 50 pages

0:20:05:40.437 cpu0:1072)Net: Vmxnet2UnmapPA:2263: unmapping PA: 129 pages

0:20:05:40.437 cpu0:1072)Net: DisablePort:1799: 0x400001a on vSwitch3 portgroup7

0:20:05:40.441 cpu0:1072)Net: Net_VMMDisconnect:79: port 0x400001a from world 1072

0:20:05:40.441 cpu0:1072)Net: PortDisassociateVmmWorld:733: world 1072 vmm0:TS1_x86 -X-> port 0x400001a on vSwitch3

0:20:05:40.441 cpu0:1072)Net: PortDisassociateVmmWorldGroup:683: world 1072 vmm0:TS1_x86 -X-> port 0x400001a on vSwitch3

0:20:05:40.441 cpu0:1072)Net: NetDisconnect:934: disconnected from net vSwitch3, PortID = 0x400001a

0:20:05:40.442 cpu0:1072)Net: Net_VMMConnect:51: switch name portgroup7 from world 1072

0:20:05:40.442 cpu0:1072)Net: Portset_ConnectPort:1076: newID 0x4000021, newIDIdx 0x21, psMask 0x3f, newPort 0x7f3bff4, portsInUse 4, portCfg 0x7f48b20

0:20:05:40.442 cpu0:1072)Net: Port_AssociateVmmWorldGroup:645: world 1072 vmm0:TS1_x86 ---> port 0x4000021 on vSwitch3

0:20:05:40.442 cpu0:1072)Net: L2Sec_SetFixedClientHWID:48: 0x4000021: 00:00:00:00:00:00 -> 00:0c:29:f2:71:10

0:20:05:40.442 cpu0:1072)Net: Net_Connect:852: connected to net portgroup7, portset 0x7c168c0, PortID = 0x4000021, status 0x0

0:20:05:40.443 cpu0:1072)Net: VlanceVMKDev_Enable:614: 0x4000021: 13

0:20:05:40.443 cpu0:1072)Net: EnablePort:1596: 0x4000021 on vSwitch3 portgroup7

0:20:05:40.501 cpu0:1072)Net: Net_VMMPortDisable:400: port 0x200001d from world 1072

0:20:05:40.501 cpu0:1072)Net: Vmxnet2VMKDevDisable:3505: saved ring indices: rxRings: 17,0 txRing: 24

0:20:05:40.501 cpu0:1072)Net: Vmxnet2VMKDevUnpinTxBuffers:2395: unmapping tx buffers: 50 pages

0:20:05:40.502 cpu0:1072)Net: Vmxnet2UnmapPA:2263: unmapping PA: 129 pages

0:20:05:40.502 cpu0:1072)Net: DisablePort:1799: 0x200001d on vSwitch1 portgroup3

0:20:05:40.505 cpu0:1072)Net: Net_VMMDisconnect:79: port 0x200001d from world 1072

0:20:05:40.505 cpu0:1072)Net: PortDisassociateVmmWorld:733: world 1072 vmm0:TS1_x86 -X-> port 0x200001d on vSwitch1

0:20:05:40.505 cpu0:1072)Net: PortDisassociateVmmWorldGroup:683: world 1072 vmm0:TS1_x86 -X-> port 0x200001d on vSwitch1

0:20:05:40.505 cpu0:1072)Net: NetDisconnect:934: disconnected from net vSwitch1, PortID = 0x200001d

0:20:05:40.506 cpu0:1072)Net: Net_VMMConnect:51: switch name portgroup3 from world 1072

0:20:05:40.506 cpu0:1072)Net: Portset_ConnectPort:1076: newID 0x2000024, newIDIdx 0x24, psMask 0x3f, newPort 0x7f08628, portsInUse 6, portCfg 0x7f14098

0:20:05:40.506 cpu0:1072)Net: Port_AssociateVmmWorldGroup:645: world 1072 vmm0:TS1_x86 ---> port 0x2000024 on vSwitch1

0:20:05:40.506 cpu0:1072)Net: L2Sec_SetFixedClientHWID:48: 0x2000024: 00:00:00:00:00:00 -> 00:0c:29:f2:71:06

0:20:05:40.506 cpu0:1072)Net: Net_Connect:852: connected to net portgroup3, portset 0x7c0bb10, PortID = 0x2000024, status 0x0

0:20:05:40.507 cpu0:1072)Net: VlanceVMKDev_Enable:614: 0x2000024: 12

0:20:05:40.507 cpu0:1072)Net: EnablePort:1596: 0x2000024 on vSwitch1 portgroup3

0:20:05:41.489 cpu1:1072)World: vm 1072: Exit:3867: Killing self with status=0x0:Success

0:20:05:41.490 cpu1:1033)Net: Net_WorldPreCleanup:463: worldID 1072 has 2 associated ports

0:20:05:41.490 cpu1:1033)Net: VlanceVMKDevDisable:575: 0x4000021

0:20:05:41.490 cpu1:1033)Net: DisablePort:1799: 0x4000021 on vSwitch3 portgroup7

0:20:05:41.490 cpu1:1033)Net: PortDisassociateVmmWorld:733: world 1072 vmm0:TS1_x86 -X-> port 0x4000021 on vSwitch3

0:20:05:41.490 cpu0:1075)World: vm 1075: Exit:3867: Killing self with status=0x0:Success

0:20:05:41.490 cpu1:1033)Net: PortDisassociateVmmWorldGroup:683: world 1072 vmm0:TS1_x86 -X-> port 0x4000021 on vSwitch3

0:20:05:41.490 cpu1:1033)Net: NetDisconnect:934: disconnected from net vSwitch3, PortID = 0x4000021

0:20:05:41.490 cpu1:1033)Net: VlanceVMKDevDisable:575: 0x2000024

0:20:05:41.490 cpu1:1033)Net: DisablePort:1799: 0x2000024 on vSwitch1 portgroup3

0:20:05:41.491 cpu1:1033)Net: PortDisassociateVmmWorld:733: world 1072 vmm0:TS1_x86 -X-> port 0x2000024 on vSwitch1

0:20:05:41.491 cpu1:1033)Net: PortDisassociateVmmWorldGroup:683: world 1072 vmm0:TS1_x86 -X-> port 0x2000024 on vSwitch1

0:20:05:41.491 cpu1:1033)Net: NetDisconnect:934: disconnected from net vSwitch1, PortID = 0x2000024

0:20:05:41.491 cpu1:1033)World: vm 1072: Reap:4368: reapCount = 1, refCount = 0, hostCount = 1, scsiActive = 0

0:20:05:41.507 cpu1:1074)World: vm 1074: Exit:3867: Killing self with status=0x0:Success

0:20:05:41.523 cpu0:1071)VSCSI: DeallocHandle:3685: Can't find handleID 8192 in world with ID 1072.

0:20:05:41.526 cpu1:1077)World: vm 1077: Exit:3867: Killing self with status=0x0:Success

0:20:05:41.527 cpu1:1076)World: vm 1076: Exit:3867: Killing self with status=0x0:Success

0:20:05:44.271 cpu1:1073)World: vm 1073: Exit:3867: Killing self with status=0x0:Success

0:20:05:44.294 cpu0:1071)World: vm 1071: Exit:3867: Killing self with status=0x0:Success

0:20:05:44.295 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x5 ra=0x642f12

0:20:05:44.295 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x6 ra=0x642f12

0:20:05:44.914 cpu1:1049)NUMASched: Epoch:3770: snapshot failed: Busy

0:20:05:54.863 cpu0:1086)Net: Net_VMMPortDisable:400: port 0x3000010 from world 1086

0:20:05:54.863 cpu0:1086)Net: Vmxnet2VMKDevDisable:3505: saved ring indices: rxRings: 40,0 txRing: 9

0:20:05:54.863 cpu0:1086)Net: Vmxnet2VMKDevUnpinTxBuffers:2395: unmapping tx buffers: 50 pages

0:20:05:54.864 cpu0:1086)Net: Vmxnet2UnmapPA:2263: unmapping PA: 129 pages

0:20:05:54.864 cpu0:1086)Net: DisablePort:1799: 0x3000010 on vSwitch2 portgroup5

0:20:05:54.867 cpu0:1086)Net: Net_VMMDisconnect:79: port 0x3000010 from world 1086

0:20:05:54.867 cpu0:1086)Net: PortDisassociateVmmWorld:733: world 1086 vmm0:SD_x86 -X-> port 0x3000010 on vSwitch2

0:20:05:54.868 cpu0:1086)Net: PortDisassociateVmmWorldGroup:683: world 1086 vmm0:SD_x86 -X-> port 0x3000010 on vSwitch2

0:20:05:54.868 cpu0:1086)Net: NetDisconnect:934: disconnected from net vSwitch2, PortID = 0x3000010

0:20:05:54.868 cpu0:1086)Net: Net_VMMConnect:51: switch name portgroup5 from world 1086

0:20:05:54.868 cpu0:1086)Net: Portset_ConnectPort:1076: newID 0x3000012, newIDIdx 0x12, psMask 0x3f, newPort 0x7f1ba20, portsInUse 4, portCfg 0x7f2e708

0:20:05:54.868 cpu0:1086)Net: Port_AssociateVmmWorldGroup:645: world 1086 vmm0:SD_x86 ---> port 0x3000012 on vSwitch2

0:20:05:54.869 cpu0:1086)Net: L2Sec_SetFixedClientHWID:48: 0x3000012: 00:00:00:00:00:00 -> 00:0c:29:c1:74:b8

0:20:05:54.869 cpu0:1086)Net: Net_Connect:852: connected to net portgroup5, portset 0x7c111e8, PortID = 0x3000012, status 0x0

0:20:05:54.908 cpu0:1086)Net: VlanceVMKDev_Enable:614: 0x3000012: 12

0:20:05:54.908 cpu0:1086)Net: EnablePort:1596: 0x3000012 on vSwitch2 portgroup5

0:20:05:55.655 cpu0:1086)World: vm 1086: Exit:3867: Killing self with status=0x0:Success

0:20:05:55.655 cpu2:1034)Net: Net_WorldPreCleanup:463: worldID 1086 has 1 associated ports

0:20:05:55.655 cpu2:1034)Net: VlanceVMKDevDisable:575: 0x3000012

0:20:05:55.656 cpu2:1034)Net: DisablePort:1799: 0x3000012 on vSwitch2 portgroup5

0:20:05:55.656 cpu0:1089)World: vm 1089: Exit:3867: Killing self with status=0x0:Success

0:20:05:55.656 cpu2:1034)Net: PortDisassociateVmmWorld:733: world 1086 vmm0:SD_x86 -X-> port 0x3000012 on vSwitch2

0:20:05:55.656 cpu2:1034)Net: PortDisassociateVmmWorldGroup:683: world 1086 vmm0:SD_x86 -X-> port 0x3000012 on vSwitch2

0:20:05:55.656 cpu2:1034)Net: NetDisconnect:934: disconnected from net vSwitch2, PortID = 0x3000012

0:20:05:55.656 cpu2:1034)World: vm 1086: Reap:4368: reapCount = 1, refCount = 0, hostCount = 1, scsiActive = 0

0:20:05:55.758 cpu0:1088)World: vm 1088: Exit:3867: Killing self with status=0x0:Success

0:20:05:55.771 cpu1:1085)VSCSI: DeallocHandle:3685: Can't find handleID 8194 in world with ID 1086.

0:20:05:55.776 cpu1:1091)World: vm 1091: Exit:3867: Killing self with status=0x0:Success

0:20:05:55.776 cpu0:1090)World: vm 1090: Exit:3867: Killing self with status=0x0:Success

0:20:05:57.527 cpu1:1087)World: vm 1087: Exit:3867: Killing self with status=0x0:Success

0:20:05:57.541 cpu1:1085)World: vm 1085: Exit:3867: Killing self with status=0x0:Success

0:20:05:57.541 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x91 ra=0x642f12

0:20:05:57.542 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x92 ra=0x642f12

0:20:05:57.542 cpu1:1033)World: VMKStackCleanup:5723: least stack avail=6012 (world 1085:vmware-vmx)

[45m [33;1mVMware ESX Server \[BETAbuild-42368] [0m

Exception type 14 in world 1101:mks:TS01_x64 @ 0x749a4a

gate=0xe frame=0x35376f4 eip=0x749a4a cr2=0x2841000 cr3=0x7d70f000 cr4=0x168

eax=0x216 ebx=0x3cd3a360 ecx=0x3ffffc11 edx=0xffffff2e es=0x4028 ds=0x4028

fs=0x0 gs=0x0 ebp=0x3537758 esi=0x3cd3b248 edi=0x2c41000 err=2 ef=0x10213

cpu 0 1099 vmm0:TS01: CPU 1 1101 mks:TS01_: cpu 2 1109 vmm0:TS03: cpu 3 1094 vmm0:SDS_:

@BlueScreen: Exception type 14 in world 1101:mks:TS01_x64 @ 0x749a4a

0x3537758:\[0x749a4a]Pkt_CopyOutMappedAndCsum+0x66(0x2c4004c, 0x3cd3a294, 0xfffffffa)

0x35377c4:\[0x74977b]Pkt_CopyOutAndCsumVerify+0x29b(0x2c4000a, 0x79c8578, 0x7f5dc78)

0x3537804:\[0x747de8]E1000ReceivePacket+0x408(0x7f1b3c4, 0x35378b0, 0x3537924)

0x35378d4:\[0x73f62c]E1000PollRxRing+0x43c(0x7f1b3c4, 0x3537924, 0x8e6d9b4)

0x35378fc:\[0x744f63]E1000DevRx+0xbb(0x7f1b3c4, 0x0, 0x3537924)

0x3537940:\[0x6d7a05]IOChain_Resume+0x12d(0x7f1b3c4, 0x7f1b3d0, 0x0)

0x3537970:\[0x6e6a8f]PortOutput+0xdb(0x7f1b3c4, 0x0, 0x3537a10)

0x3537a4c:\[0x6e5eed]EtherswitchPortDispatch+0x419(0x7c111e8, 0x3537ae8, 0x7f14e04)

0x3537a7c:\[0x6c8d24]Port_Input+0x198(0x7f14e04, 0x3537ae8, 0x0)

0x3537b14:\[0x6bc62e]NetBH+0x5ca(0x0, 0x3537b2c, 0x88eb35)

0x3537b48:\[0x617f89]BHCallHandlers+0xa5(0x40400010, 0x0, 0xe8000000)

0x3537b74:\[0x618192]BH_Check+0x14e(0x1, 0x1cedd98, 0xc627c0)

0x3537b9c:\[0x62ef22]IDT_HandleInterrupt+0x8a(0x3537bf4, 0x3537bd0, 0x3537bc4)

0x3537bbc:\[0x62f18c]IDTIntrHandler+0x90(0x3537bf4, 0x4028, 0x4028)

0x3537c2c:\[0x6ace3c]CommonIntr+0xc(0x1, 0x2c423c48, 0x961f)

0x3537c50:\[0x7ef083]CpuSchedIdleLoopInt+0x117(0x14e71e0, 0x0, 0x0)

0x3537c88:\[0x7ef298]CpuSchedBusyWait+0x138(0x0, 0x7, 0x0)

0x3537da4:\[0x7e59e3]CpuSchedDispatch+0x373(0x0, 0x0, 0x3eba6efd)

0x3537dec:\[0x7eade0]CpuSchedWait+0x478(0x1969bc8, 0x3eba6efd, 0x1f)

0x3537e10:\[0x7eb0fd]CpuSched_WaitDirectedYield+0x39(0x3eba6efd, 0x1f, 0x0)

VMK uptime: 0:20:44:42.862 TSC: 165060650377171

lastClrIntrRA = 0x6a2174

Starting coredump to disk Starting coredump to disk Dumping using slot 1 of 1... using slot 1 of 1... log

ESX2[/b]...after connecting FC

0:18:50:45.101 cpu1:1082)

0:20:05:55.771 cpu1:1085)VSCSI: DeallocHandle:3685: Can't find handleID 8194 in world with ID 1086.

0:20:05:55.776 cpu1:1091)World: vm 1091: Exit:3867: Killing self with status=0x0:Success

0:20:05:55.776 cpu0:1090)World: vm 1090: Exit:3867: Killing self with status=0x0:Success

0:20:05:57.527 cpu1:1087)World: vm 1087: Exit:3867: Killing self with status=0x0:Success

0:20:05:57.541 cpu1:1085)World: vm 1085: Exit:3867: Killing self with status=0x0:Success

0:20:05:57.541 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x91 ra=0x642f12

0:20:05:57.542 cpu0:1024)RPC: FindAndLockCnx:707: Invalid connection ID: 0x92 ra=0x642f12

Soon after this, ESX1 has crashed.

IRQs are different for HBA/NIC/P400 SAS Controller at VI3 Level. But they share with other components as below:

P400 SAS Controller is having IRQ 17

Ethernet controller(Intel Corp.) is having IRQ 19

Ethernet controller(Intel Corp.) is having IRQ 18 ... Dual Port NC360T NIC.

Ethernet controller(Broadcom Corporation) is having IRQ 24

Ethernet controller(Broadcom Corporation) is having IRQ 25 ... OnBoard Broadcom NICs.

Fibre Channel: QLogic Corp. QLA2312 Fibre Channel Adapter is having IRQ 20

USB Controller: PCI device 10de:005a (nVidia Corporation) (rev 162) is having IRQ 16.

VGA compatible controller: PCI device 1002:515e (ATI Technologies Inc) is having IRQ 17.

System peripheral: PCI device 0e11:b203 (Compaq Computer Corporation) is having IRQ 18.

System peripheral: PCI device 0e11:b204 (Compaq Computer Corporation) is having IRQ 19.

USB Controller: PCI device 103c:3300 (Hewlett-Packard Company) is having IRQ 19.

Class 0c07: PCI device 103c:3302 (Hewlett-Packard Company) is having IRQ 18.

But at the BIOS level, some of these IRQs are shared.

P400 is IRQ 5 sharing with Video Controller.

Intel NC360T Port 1 is IRQ 10

Intel NC360T Port 2 is IRQ 11

OnBoard Broadcom(NC371i) Port 1 is 10

OnBoard Broadcom(NC371i) Port 2 is 11

All NICs share same IRQ with HP iLO(4 components).

DL585 G2 RBSU User Guide says "Multiple PCI devices can share an interrupt". I'm thinking of changing the slots for P400, HBA & NIC and seeing whether anything would help. By doing this, vmnic & vmhba references will go mad. They need to be reconfigured.

So, with this whole lot debug mode output, what can be pin pointed looking at the unusual error message posted above?

Anyone can please help me in understanding the situation?

0 Kudos
dheerajms
Enthusiast
Enthusiast

I also would like to verify Zoning in this case. Let me explain. DS4300 is the Storage Array. I have a 16 Port SAN Switch, 8 ports activated and used up by DS4300(SPA1, SPA2, SPB1, SPB2), ESX Servers(ESX1_1, ESX2_1), TS1 and TS2 Servers. Ports in SAN Switch used as follows:

0 - SPA1

1 - ESX1_1

2 - ESX2_1

3 - SPB2

4 - SPA2

5 - TS1

6 - TS2

7 - SPB1

Previously, before PSOD's i had the following Zone in place for ESX Servers and Physical Terminal Servers,

Zone ESX(0,1,2,3)

Zone TS1(4,5,7)

Zone TS2(4,6,7)

After PSOD's, i tried to isolate various components and hence changed the Zoning to:

Zone ESX1(0,1,3)

Zone ESX2(0,2,3)

Zone TS1(4,5,7)

Zone TS2(4,6,7)

How good is this? TS1 & TS2 work perfectly. Even otherwise, ESX Servers too work perfectly but crash frequently.

What is the best zoning preferred for Two Hosts with Single HBA each, SAN Switch(8 Ports) & DS4300 for VI3.

Would be of great help if someone can share this valuable information Smiley Happy TIA!

0 Kudos
mangzeus
Contributor
Contributor

Hi,

I'm running ESX 3.0.1 on 2 nos of DL585 G2, each

having - 2 Opteron Dual Core CPU(NUMA, Node

Balanced), 16GB RAM(HP), P400 Smart Array Controller

with Dual SAS HDD in RAID-1, QLogic 2340 2 Gbps HBA &

Intel NC360T Dual Port NIC.

I'm running 3 Terminal Server VM's(Windows Server

2003 R2 Enterprise x64 Edition) on each ESX1 & ESX2

with NLB. Session Directory Server is running on

ESX1. All Terminal Server VM's are having Dual NIC,

one for NLB & other for inter-host communication. I

have carved them out of vmnic1 and called those

switches as NLB & TSLink. SD is on vmnic2. NLB is

configured for unicast as the multicast is not

possible with existing switches.

SAN consists of DS4300 connected to EXP810 and IBM 16

Port SAN Switch.

I have all firmware & ROMPAQs (DL585 BIOS, HDD, HBA,

DS4300, EXP810) latest, VI 3.0.1 patches till date(25

patches), VirtualCenter 2.0.1 Patch 2.

VirtualCenter(VC) Server manages ESX hosts and the

license server on VC manages the licensing. It's an

VI3 Enterprise License.

No HP Insight Agents running on the ESX Hosts.

7 VM's, 7 LUN's, and one VMFS volume per LUN.

Previously, i had done Storage Partitioning and hence

both ESX1 & ESX2 were able to seen the LUN's. Hence

VMotion, HA & DRS were all happening with no issues.

With all this in place, both the ESX Servers crash

almost together with a PSOD once in 20 - 40 hours

from past couple of days.

To isolate the problem, i've separated the hosts in

terms of Storage(no storage partition now), Zoning &

Licensing. Not using VC Server. Disabled VMotion, HA

& DRS.

Advanced configuration include:

QLogic HBA Level:

Enable LIP Reset: No

Enable LIP Full Login: Yes

Enable Target Reset: Yes

4300 Level:

Host type is set to LNXCL

ADT Disabled

DS4300 SP Sense Data setting applied

N Switch:

RSCN Suppressed on both Storage & Host

Ports given to ESX Servers(Ports 0-3 in SAN Switch,

0 & 3 are DS4300 SPA1 & SPB2; 1 & 2 are ESX1 &

ESX2)

ESX 3.0.1:

Disk.UseDeviceReset to 0

Disk.UseLunReset to 1

Using 2 GBPS HBA Driver

QLogic Queue Depth - 64

Some Logs:

\[root@esx01 root]# cat /var/log/vmkwarning

Apr 1 00:13:35 esx01 vmkernel: 0:06:32:13.155

cpu3:1027)WARNING: CpuSched: 7161: time went

backwards by 12 usec

Apr 2 05:38:25 esx01 vmkernel: 1:11:57:03.117

cpu1:1025)WARNING: CpuSched: 7161: time went

backwards by 23 usec

Apr 2 16:43:17 esx01 vmkernel: 0:00:07:35.429

cpu2:1065)WARNING: CpuSched: 7161: time went

backwards by 10 usec

Apr 2 17:52:33 esx01 vmkernel: 0:01:16:51.916

cpu0:1060)WARNING: CpuSched: 7161: time went

backwards by 26 usec

Apr 2 17:52:33 esx01 vmkernel: 0:01:16:51.948

cpu0:1024)WARNING: CpuSched: 7161: time went

backwards by 40 usec

Apr 2 20:20:16 esx01 vmkernel: 0:00:02:19.509

cpu1:1058)WARNING: CpuSched: 7161: time went

backwards by 10 usec

Apr 2 21:10:56 esx01 vmkernel: 0:00:52:59.547

cpu0:1067)WARNING: CpuSched: 7161: time went

backwards by 39 usec

Apr 2 21:25:55 esx01 vmkernel: 0:01:07:58.343

cpu1:1068)WARNING: CpuSched: 7161: time went

backwards by 40 usec

Apr 3 15:03:19 esx01 vmkernel: 0:00:08:51.126

cpu0:1067)WARNING: CpuSched: 7161: time went

backwards by 10 usec

This warning keeps repeating often. Should i include

that TSC Setting "host.TSC.noForceSync = TRUE" in

each VM .vmx file?

PSOD Final Message is:

[45m[33;1mVMware ESX Server \[Releasebuild-42368][0m

Exception type 14 in world 1058:vmm0:TS04 @ 0x6bbd47

gate=0xe frame=0x348b8f0 eip=0x6bbd47 cr2=0x2807000

cr3=0x4086f000 cr4=0x668

eax=0xffffffee ebx=0xc ecx=0x3ffffc11 edx=0x0

es=0x4028 ds=0x4028

fs=0x0 gs=0x4041 ebp=0x348b950 esi=0x741fcb8

edi=0x2c07000 err=2 ef=0x11213

cpu 0 1024 console: cpu 1 1068 vmm0:TS06: cpu 2 1063

vmm0:TS05: CPU 3 1058 vmm0:TS04:

@BlueScreen: Exception type 14 in world

1058:vmm0:TS04 @ 0x6bbd47

0x348b950:\[0x6bbd47]Pkt_CopyOutMappedAndCsum+0x4f(0x2c

0604c, 0x741ed04, 0xfffffffa)

0x348b9b8:\[0x6bbb5d]Pkt_CopyOutAndCsumVerify+0x271(0x2

c0600a, 0x61fb3c8, 0x0)

0x348b9e0:\[0x6bae65]E1000ReceivePacket+0x231(0x66e8438

, 0x348ba8c, 0x348bae8)

0x348bab0:\[0x6b6d75]E1000PollRxRing+0x1bd(0x66e8438,

0x348bae8, 0x3d76e008)

0x348bacc:\[0x6b991d]E1000DevRx+0x55(0x66e8438, 0x0,

0x348bae8)

0x348bb04:\[0x67e3be]IOChain_Resume+0x6e(0x66e8438,

0x66e8444, 0x0)

0x348bb1c:\[0x68741d]PortOutput+0x39(0x66e8438, 0x0,

0x348bbf8)

0x348bc34:\[0x68689c]EtherswitchPortDispatch+0x368(0x64

0b6a0, 0x348bc98, 0x66e6248)

0x348bc58:\[0x6770be]Port_Input+0x96(0x66e6248,

0x348bc98, 0x0)

0x348bcc4:\[0x670669]NetBH+0x2c1(0x0, 0x8, 0x91)

0x348bcfc:\[0x60d3c8]BHCallHandlers+0x68(0x40400010,

0x6fbc94, 0x2000000)

0x348bd1c:\[0x60d498]BH_Check+0x6c(0x1, 0x44e,

0x1935a1e)

0x348bd38:\[0x61b8bb]IDT_HandleInterrupt+0x77(0x348bd84

, 0x0, 0x14a97e4)

0x348bd4c:\[0x61b9ee]IDTIntrHandler+0x52(0x348bd84,

0x4041, 0x4041)

0x348bdb0:\[0x667e4c]CommonIntr+0xc(0x3, 0x6276b820,

0x7ee2)

0x348bdcc:\[0x6fb4bb]CpuSchedIdleLoopInt+0xaf(0xca99c0,

0x0, 0x5ec)

0x348bdf8:\[0x6fb578]CpuSchedBusyWait+0x54(0x0, 0x7,

0x0)

0x348bef0:\[0x6f5722]CpuSchedDispatch+0x296(0x0, 0x0,

0xca9a4c)

0x348bf2c:\[0x6f8c98]CpuSchedWait+0x188(0x18fa450,

0xca9a4c, 0x6)

0x348bf50:\[0x6f8e7d]CpuSched_WaitDirectedYield+0x39(0x

ca9a4c, 0x6, 0xffffffff)

VMK uptime: 0:17:32:01.977 TSC: 139510789393287

Starting coredump to disk Starting coredump to disk

Dumping using slot 1 of 1... using slot 1 of 1...

log

SD, TS01 to TS03 in ESX1 & TS04 to TS06 in ESX2 are

Session Directory Server & Terminal Servers.

I power on Windows VM's and i see in their System Log

that it would have mentioned that the last shutdown

was unexpected.

Have logged a call with VMware via HP. Yet to hear

from them.

64-Bit Windows Server 2003 R2 causing problems? Have

checked the RAM modules, they are fine. My worry is

even after separating the hosts, both crash together!

Is that because i have 3 TS running in one ESX &

other 3 running out of other? I have similar switch

configuration but still they are physically

different?

With this info, any pointers to overcome this

problem? Please let me know.

Thanks,

Dheeraj.

Hello my name is marco, excused for my English but I do not know it very well; i write from Verona (italy) and I have a problem of connection between ESXServer and LUN in the storage the 4300, in the sense that I do not succeed to add more than 2 partitions inside of the VC for every host. How I make to exceed this limit? I know that it exists “features storage partitioning” that it is visualized through the Storage program subsystem. mine it would only seem qualified that in “list” I see visualized enabled: 1 of 4. How I make for being able to exceed the limit of 2 partitions for every host? thanks to all

0 Kudos
mangzeus
Contributor
Contributor

Hi,

I'm running ESX 3.0.1 on 2 nos of DL585 G2, each

having - 2 Opteron Dual Core CPU(NUMA, Node

Balanced), 16GB RAM(HP), P400 Smart Array Controller

with Dual SAS HDD in RAID-1, QLogic 2340 2 Gbps HBA &

Intel NC360T Dual Port NIC.

I'm running 3 Terminal Server VM's(Windows Server

2003 R2 Enterprise x64 Edition) on each ESX1 & ESX2

with NLB. Session Directory Server is running on

ESX1. All Terminal Server VM's are having Dual NIC,

one for NLB & other for inter-host communication. I

have carved them out of vmnic1 and called those

switches as NLB & TSLink. SD is on vmnic2. NLB is

configured for unicast as the multicast is not

possible with existing switches.

SAN consists of DS4300 connected to EXP810 and IBM 16

Port SAN Switch.

I have all firmware & ROMPAQs (DL585 BIOS, HDD, HBA,

DS4300, EXP810) latest, VI 3.0.1 patches till date(25

patches), VirtualCenter 2.0.1 Patch 2.

VirtualCenter(VC) Server manages ESX hosts and the

license server on VC manages the licensing. It's an

VI3 Enterprise License.

No HP Insight Agents running on the ESX Hosts.

7 VM's, 7 LUN's, and one VMFS volume per LUN.

Previously, i had done Storage Partitioning and hence

both ESX1 & ESX2 were able to seen the LUN's. Hence

VMotion, HA & DRS were all happening with no issues.

With all this in place, both the ESX Servers crash

almost together with a PSOD once in 20 - 40 hours

from past couple of days.

To isolate the problem, i've separated the hosts in

terms of Storage(no storage partition now), Zoning &

Licensing. Not using VC Server. Disabled VMotion, HA

& DRS.

Advanced configuration include:

QLogic HBA Level:

Enable LIP Reset: No

Enable LIP Full Login: Yes

Enable Target Reset: Yes

4300 Level:

Host type is set to LNXCL

ADT Disabled

DS4300 SP Sense Data setting applied

N Switch:

RSCN Suppressed on both Storage & Host

Ports given to ESX Servers(Ports 0-3 in SAN Switch,

0 & 3 are DS4300 SPA1 & SPB2; 1 & 2 are ESX1 &

ESX2)

ESX 3.0.1:

Disk.UseDeviceReset to 0

Disk.UseLunReset to 1

Using 2 GBPS HBA Driver

QLogic Queue Depth - 64

Some Logs:

\[root@esx01 root]# cat /var/log/vmkwarning

Apr 1 00:13:35 esx01 vmkernel: 0:06:32:13.155

cpu3:1027)WARNING: CpuSched: 7161: time went

backwards by 12 usec

Apr 2 05:38:25 esx01 vmkernel: 1:11:57:03.117

cpu1:1025)WARNING: CpuSched: 7161: time went

backwards by 23 usec

Apr 2 16:43:17 esx01 vmkernel: 0:00:07:35.429

cpu2:1065)WARNING: CpuSched: 7161: time went

backwards by 10 usec

Apr 2 17:52:33 esx01 vmkernel: 0:01:16:51.916

cpu0:1060)WARNING: CpuSched: 7161: time went

backwards by 26 usec

Apr 2 17:52:33 esx01 vmkernel: 0:01:16:51.948

cpu0:1024)WARNING: CpuSched: 7161: time went

backwards by 40 usec

Apr 2 20:20:16 esx01 vmkernel: 0:00:02:19.509

cpu1:1058)WARNING: CpuSched: 7161: time went

backwards by 10 usec

Apr 2 21:10:56 esx01 vmkernel: 0:00:52:59.547

cpu0:1067)WARNING: CpuSched: 7161: time went

backwards by 39 usec

Apr 2 21:25:55 esx01 vmkernel: 0:01:07:58.343

cpu1:1068)WARNING: CpuSched: 7161: time went

backwards by 40 usec

Apr 3 15:03:19 esx01 vmkernel: 0:00:08:51.126

cpu0:1067)WARNING: CpuSched: 7161: time went

backwards by 10 usec

This warning keeps repeating often. Should i include

that TSC Setting "host.TSC.noForceSync = TRUE" in

each VM .vmx file?

PSOD Final Message is:

[45m[33;1mVMware ESX Server \[Releasebuild-42368][0m

Exception type 14 in world 1058:vmm0:TS04 @ 0x6bbd47

gate=0xe frame=0x348b8f0 eip=0x6bbd47 cr2=0x2807000

cr3=0x4086f000 cr4=0x668

eax=0xffffffee ebx=0xc ecx=0x3ffffc11 edx=0x0

es=0x4028 ds=0x4028

fs=0x0 gs=0x4041 ebp=0x348b950 esi=0x741fcb8

edi=0x2c07000 err=2 ef=0x11213

cpu 0 1024 console: cpu 1 1068 vmm0:TS06: cpu 2 1063

vmm0:TS05: CPU 3 1058 vmm0:TS04:

@BlueScreen: Exception type 14 in world

1058:vmm0:TS04 @ 0x6bbd47

0x348b950:\[0x6bbd47]Pkt_CopyOutMappedAndCsum+0x4f(0x2c

0604c, 0x741ed04, 0xfffffffa)

0x348b9b8:\[0x6bbb5d]Pkt_CopyOutAndCsumVerify+0x271(0x2

c0600a, 0x61fb3c8, 0x0)

0x348b9e0:\[0x6bae65]E1000ReceivePacket+0x231(0x66e8438

, 0x348ba8c, 0x348bae8)

0x348bab0:\[0x6b6d75]E1000PollRxRing+0x1bd(0x66e8438,

0x348bae8, 0x3d76e008)

0x348bacc:\[0x6b991d]E1000DevRx+0x55(0x66e8438, 0x0,

0x348bae8)

0x348bb04:\[0x67e3be]IOChain_Resume+0x6e(0x66e8438,

0x66e8444, 0x0)

0x348bb1c:\[0x68741d]PortOutput+0x39(0x66e8438, 0x0,

0x348bbf8)

0x348bc34:\[0x68689c]EtherswitchPortDispatch+0x368(0x64

0b6a0, 0x348bc98, 0x66e6248)

0x348bc58:\[0x6770be]Port_Input+0x96(0x66e6248,

0x348bc98, 0x0)

0x348bcc4:\[0x670669]NetBH+0x2c1(0x0, 0x8, 0x91)

0x348bcfc:\[0x60d3c8]BHCallHandlers+0x68(0x40400010,

0x6fbc94, 0x2000000)

0x348bd1c:\[0x60d498]BH_Check+0x6c(0x1, 0x44e,

0x1935a1e)

0x348bd38:\[0x61b8bb]IDT_HandleInterrupt+0x77(0x348bd84

, 0x0, 0x14a97e4)

0x348bd4c:\[0x61b9ee]IDTIntrHandler+0x52(0x348bd84,

0x4041, 0x4041)

0x348bdb0:\[0x667e4c]CommonIntr+0xc(0x3, 0x6276b820,

0x7ee2)

0x348bdcc:\[0x6fb4bb]CpuSchedIdleLoopInt+0xaf(0xca99c0,

0x0, 0x5ec)

0x348bdf8:\[0x6fb578]CpuSchedBusyWait+0x54(0x0, 0x7,

0x0)

0x348bef0:\[0x6f5722]CpuSchedDispatch+0x296(0x0, 0x0,

0xca9a4c)

0x348bf2c:\[0x6f8c98]CpuSchedWait+0x188(0x18fa450,

0xca9a4c, 0x6)

0x348bf50:\[0x6f8e7d]CpuSched_WaitDirectedYield+0x39(0x

ca9a4c, 0x6, 0xffffffff)

VMK uptime: 0:17:32:01.977 TSC: 139510789393287

Starting coredump to disk Starting coredump to disk

Dumping using slot 1 of 1... using slot 1 of 1...

log

SD, TS01 to TS03 in ESX1 & TS04 to TS06 in ESX2 are

Session Directory Server & Terminal Servers.

I power on Windows VM's and i see in their System Log

that it would have mentioned that the last shutdown

was unexpected.

Have logged a call with VMware via HP. Yet to hear

from them.

64-Bit Windows Server 2003 R2 causing problems? Have

checked the RAM modules, they are fine. My worry is

even after separating the hosts, both crash together!

Is that because i have 3 TS running in one ESX &

other 3 running out of other? I have similar switch

configuration but still they are physically

different?

With this info, any pointers to overcome this

problem? Please let me know.

Thanks,

Dheeraj.

Hello my name is marco, excused for my English but I do not know it very well; i write from Verona (italy) and I have a problem of connection between ESXServer and LUN in the storage the 4300, in the sense that I do not succeed to add more than 2 partitions inside of the VC for every host. How I make to exceed this limit? I know that it exists “features storage partitioning” that it is visualized through the Storage program subsystem. mine it would only seem qualified that in “list” I see visualized enabled: 1 of 4. How I make for being able to exceed the limit of 2 partitions for every host? thanks to all

0 Kudos
dheerajms
Enthusiast
Enthusiast

Well, posting back some good results after about 4-5 months. Now it's all stable and rocking.

Finally, i gave a fresh reinstall for the Servers, with all the patches. This was done after PCI - IRQ - Vector clashes were removed. Zoning was cleaned up, note that duplicate entries should not be made even if SAN Switch Zoning Tool complains that some entities are not included although you have included in some or the other form. After this, it was the turn of 64-Bit Windows Server 2003 R2 to trouble me in the form of BSOD's. Answer was to install SP2 and issue is here http://support.microsoft.com/kb/911028/en-us. In short, when you use a 32-bit program to print a document from a 64-bit version of Microsoft Windows, you receive a Stop error message.

It's all going good now. Stable from last 3-4 months. Thanks to all who have read this post and helped in out.

Dheeraj.

0 Kudos