VMware Cloud Community
joernc
Enthusiast
Enthusiast

infrequent boot to Recovery Console, Windows Server 2008 R2

Hi!  When booting VMs running Windows Server 2008 R2, sometime they end up in the Recovery Console. This seems to happen most often after installing Windows updates (but then again, this is probably the dominant reason for reboots). Simply aborting the first requester in the Recovery Console leads to a successful reboot of the VM. A Minidump is created, which points to a fault in the Windows kernel (see below).  Does anybody have similar problems? Does anybody have an explanation and maybe even a fix for this? I asked VMware Support about this, but they weren't aware of other people seeing this happen. I am using a rather out-of-the-box installation, with VMware Tools installed and VMXNET3 driver, nothing fancy. Environment is ESX4.1 (U3, but this problem started earlier).   This is the analysis of the minidump. The german error message roughly translates to "The instruction in 0x%08lx points to memory 0x%08lx. Unable to execute task %s in memory". Whatever that actually means...  EXCEPTION_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.  FAULTING_IP:  +16 00004449`485f2e43 ??              ???  EXCEPTION_RECORD:  fffff88001f57a38 -- (.exr 0xfffff88001f57a38) ExceptionAddress: 00004449485f2e43    ExceptionCode: c0000005 (Access violation)   ExceptionFlags: 00000000 NumberParameters: 2    Parameter[0]: 0000000000000008    Parameter[1]: 00004449485f2e43 Attempt to execute non-executable address 00004449485f2e43  CONTEXT:  fffff88001f57290 -- (.cxr 0xfffff88001f57290) rax=fffffa8001cb72b0 rbx=fffffa8001cd7830 rcx=0000000000000000 rdx=fffffa8001cb72b0 rsi=fffffa800189e660 rdi=fffffa8001cd5720 rip=00004449485f2e43 rsp=fffff88001f57c78 rbp=fffff8000186c280  r8=fffffa800188d0c8  r9=0000000000000000 r10=fffffffffffffffe r11=fffff80001841e80 r12=fffffa8001cd5720 r13=0000000000000000 r14=0000000000000000 r15=0000000000000001 iopl=0         nv up ei pl zr na po nc cs=0010  ss=0018  ds=002b  es=002b  fs=0053  gs=002b             efl=00010246 00004449`485f2e43 ??              ??? Resetting default scope  DEFAULT_BUCKET_ID:  DRIVER_FAULT_SERVER_MINIDUMP  PROCESS_NAME:  System  CURRENT_IRQL:  0  ERROR_CODE: (NTSTATUS) 0xc0000005 - Die Anweisung in 0x%08lx verweist auf Speicher 0x%08lx. Der Vorgang %s konnte nicht im Speicher durchgef hrt werden.  EXCEPTION_PARAMETER1:  0000000000000008  EXCEPTION_PARAMETER2:  00004449485f2e43  WRITE_ADDRESS: GetPointerFromAddress: unable to read from fffff800018fe100  00004449485f2e43   FOLLOWUP_IP:  nt!IopProcessWorkItem+23 fffff800`019c1633 488bcb          mov     rcx,rbx  FAILED_INSTRUCTION_ADDRESS:  +23 00004449`485f2e43 ??              ???  BUGCHECK_STR:  0x7E  LAST_CONTROL_TRANSFER:  from fffff800019c1633 to 00004449485f2e43  STACK_TEXT:   fffff880`01f57c78 fffff800`019c1633 : 00000000`00000001 00000000`00000000 fffffa80`01896830 fffffa80`0189e660 : 0x4449`485f2e43 fffff880`01f57c80 fffff800`016d8841 : fffffa80`01896800 fffff800`019c1600 fffff800`018cd800 00000000`00000000 : nt!IopProcessWorkItem+0x23 fffff880`01f57cb0 fffff800`01965e6a : 00000000`00000000 fffffa80`0189e660 00000000`00000080 fffffa80`0188d040 : nt!ExpWorkerThread+0x111 fffff880`01f57d40 fffff800`016bfec6 : fffff800`01841e80 fffffa80`0189e660 fffffa80`0189eb50 00000000`00000000 : nt!PspSystemThreadStartup+0x5a fffff880`01f57d80 00000000`00000000 : fffff880`01f58000 fffff880`01f52000 fffff880`01f579e0 00000000`00000000 : nt!KxStartSystemThread+0x16   SYMBOL_STACK_INDEX:  1  SYMBOL_NAME:  nt!IopProcessWorkItem+23  FOLLOWUP_NAME:  MachineOwner  MODULE_NAME: nt  IMAGE_NAME:  ntkrnlmp.exe  DEBUG_FLR_IMAGE_TIMESTAMP:  4fa390f3  STACK_COMMAND:  .cxr 0xfffff88001f57290 ; kb  FAILURE_BUCKET_ID:  X64_0x7E_BAD_IP_nt!IopProcessWorkItem+23  BUCKET_ID:  X64_0x7E_BAD_IP_nt!IopProcessWorkItem+23  Followup: MachineOwner

0 Kudos
3 Replies
joernc
Enthusiast
Enthusiast

stupid online editor removing all linebreaks... I have attached the debugger output as file.

0 Kudos
TOMJP
Contributor
Contributor

We have also been experiencing this problem.  I'd guess it is about 1% of boots, mabye 1/2 that. 
We get both bccode 7e and 44.

No resolution yet, we haven't nailed down the offending driver.  Nothing to do but cancel the recovery console and start again.

0 Kudos
joernc
Enthusiast
Enthusiast

We have set up a VM that reboots every hour to provoke the problem. The machine happily rebooted for weeks, until new Windows updates were available. We configured the machine to create a complete memory dump and sent it to Microsoft for analysis. They suggested this fix

http://support.microsoft.com/kb/2591271

which points to problems with the IPv6 driver. After installation of the Hotfix, the problem has not shown again, although I think it is too early to call this case closed.

0 Kudos