1 2 Previous Next 21 Replies Latest reply on Apr 8, 2014 7:13 AM by TarasSY

    Bluescreen 7b after restore

    goppi Enthusiast

      Hi.

       

      This is not directly releated to GhettoVCB as we have modified the script to our needs,

      however I suspect the problems we encounter will affect users of original GhettoVCB as well.

       

      We have two installations of VSphere 5.1 where all restored Win2008R2 machines are

      unusable as they produce a Bluescreen with code 7b on boot.

       

      This is reproducable to both installations and all Win2008R2 machines we backup.

       

      On further testing it seems that the Quiesce option is the cause of the problem.

       

      We tried to reduce complexity to find the cause and had the following steps

      which failed with the bluescreen mentioned.

       

      1. Plain vanilla VM type 8 with Win2008R2 Enterprise and VMware Tools installed
      2. Taking a Snapshot via VSphere client (Quiesce option) checked.
      3. vmkfstools -i testserver.vmdk -d 2gbsparse backup.vmdk
      4. Shutdown VM via VSphere client
      5. Deleting snapshot via VSphere client
      6. delete testserver.vmdk via datastore browser
      7. vmkfstools -i backup.vmdk -d thin testserver.vmdk
      8. Start machine via VSphere client
      9. Bluescreen.

       

      I would be very interested in feedback if others using VSphere 5.1 and Win2008R2 are

      able to successfully restore VMs backed up with Quiesce option and GhettoVCB.

       

      Cheers.

        • 1. Re: Bluescreen 7b after restore
          goppi Enthusiast

          Be careful when running Windows 2008 - Your backups might be corrupt.


          Further investigation shows that on 5.1 when doing a Quiesce snapshot two snapshot vmdk are created when running Win2008R2.

          Using the original vmdk in this state for cloing/backup (like GhettoVCB does) can lead to unusable backups at least for Win2008R2.

          First tests show that when using the -000002.vmdk instead things seem to work but who knows.

           

          Anyone else who has additional knowledge on that?

           

          Cheers.

          • 2. Re: Bluescreen 7b after restore
            goppi Enthusiast

            "Designing Backup Solutions for VMware vSphere" states the following:

             

            Windows 2008 application level quiescing is performed using a hardware snapshot provider. After quiescing
            the virtual machine, the hardware snapshot provider creates two redo logs per disk: one for the live virtual
            machine writes and another for the VSS and writers in the guest to modify the disks after the snapshot
            operation as part of the quiescing operations. The snapshot configuration information reports this second redo
            log as part of the snapshot. This redo log represented the quiesced state of all the applications in the guest. This
            redo log must be opened for backup with VDDK 1.2. The older VDDK 1.1 software cannot open the second
            redo log for backup.

             

            So be warned that you may get a non working backup if you refer to the initial vmdk file for backup/cloning

            as you would normally do with non Win2008 guests.

            • 3. Re: Bluescreen 7b after restore
              InsertCoin Novice

              I'm struggling with the same problem: ghettoVCB on ESXi 5.1, VM is a Windows Server 2008R2 machine, Quiesce option is ON. After restoring the backup: always a BSOD with 7B "stop error". I can restore a non-quiesced backup of a running WS2008R2 machine without a BSOD.

              Is a ghettoVCB backup always unreliable when used with WS2008R2? Or is this a ESXi 5.1 problem? Is backing up a running WS2008R2 VM with Quiesce set to OFF reliable, using ghettoVCB?

              • 4. Re: Bluescreen 7b after restore
                goppi Enthusiast

                InsertCoin schrieb:

                 

                I'm struggling with the same problem: ghettoVCB on ESXi 5.1, VM is a Windows Server 2008R2 machine, Quiesce option is ON. After restoring the backup: always a BSOD with 7B "stop error". I can restore a non-quiesced backup of a running WS2008R2 machine without a BSOD.

                Is a ghettoVCB backup always unreliable when used with WS2008R2? Or is this a ESXi 5.1 problem? Is backing up a running WS2008R2 VM with Quiesce set to OFF reliable, using ghettoVCB?

                I tried to be as clear as possible on what I have posted.

                 

                The way ghettoVCB works it produces unusable backups when the following conditions are met:

                 

                • OS is Windows 2008 or up
                • VSS provider of VMware tools is used
                • ESXi 4.1 or up is used
                • Quiesec option is set to ON

                 

                The reason is it simply picks the wong redo-log (snapshot file) in this scenario as there a two of them created.

                You can simply verify this.

                 

                I was suprised in getting nearly no response to my posting, as I'm sure a lot of people will use ghettoVCB in this way, and they all will be in for a rude awakening if they ever had to restore what they backed up before.

                 

                Cheers.

                • 5. Re: Bluescreen 7b after restore
                  scowse Enthusiast

                  Thanks for all your efforts on documenting this one but I would never strike this scenario because I do not quiesce.

                  When I require absolutely bullet-proof backups for mission critical vms, which I have had for several years thanks to ghetto, I leave nothing to chance and shut the vm down before backup.

                  If you are unable to do so then you should pay a larger sum of money for your backup solution.

                  I have always viewed quiescing as a smoke and mirrors black art and studiously avoided it - hence no problems!

                  • 6. Re: Bluescreen 7b after restore
                    goppi Enthusiast

                    scowse schrieb:

                     

                    Thanks for all your efforts on documenting this one but I would never strike this scenario because I do not quiesce.

                    When I require absolutely bullet-proof backups for mission critical vms, which I have had for several years thanks to ghetto, I leave nothing to chance and shut the vm down before backup.

                    If you are unable to do so then you should pay a larger sum of money for your backup solution.

                    I have always viewed quiescing as a smoke and mirrors black art and studiously avoided it - hence no problems!

                     

                    Hi scowse.

                     

                    I partially agree to what you say.

                    We also have switched our backup stragtegy to have backups once per week for all machines in an switched-off state. That's not a big deal. Shutdown, take snapshot, restart, backup.

                     

                    On the other side I have to comment that the so called "expencive backup solutions" are expencive in the first place. For some customers we swtiched back to a ghettoVCB like approach because it seems that some of the vendors meanwhile have problems in handling the compelxity of their own software applications.

                     

                    And those solutions use the same ecosystem as ghettoVCB at least if they do not implement their own VSS provider (as e.g. Symantec does) and so are susceptible to the same kind of problems. Don't know if you have watched the flame war between Veem and Symantec regarding vStorage API 5.1 and the chances in getting a reliable backup solution at all.

                     

                    So there is nothing wrong with quiescing in general if:

                     

                    • you trust the VSS provider and all VSS writers for your applications
                    • use the correct redo-log for the backup process.

                     

                    Cheers.

                    • 7. Re: Bluescreen 7b after restore
                      InsertCoin Novice

                      The question that remains for me is: can (the highly praised) ghettoVCB be altered so it will use the correct redo-log (and have a correct quiesced snapshot backup of a WS2008 VM)?

                      • 8. Re: Bluescreen 7b after restore
                        goppi Enthusiast

                        InsertCoin schrieb:

                         

                        The question that remains for me is: can (the highly praised) ghettoVCB be altered so it will use the correct redo-log (and have a correct quiesced snapshot backup of a WS2008 VM)?

                         

                        I have no idea how because things like guest OS and VSS provider involved seem to have an influence on this.

                         

                        For example if you backup a Linux or Windows 2003 box everything is ok.

                        The same applies if you backup a Windows 2008 R2 box and use the Symantec VSS provider instead of the VMware one.

                        When doing a backup of Windows 2008 R2 and using VMware VSS it seems you have to choose the -000002.vmdk file for cloning instead.

                         

                        Cheers.

                        • 9. Re: Bluescreen 7b after restore
                          chrtho1 Novice

                          Are there very few who actually test restore of their backups - or is this problem limited to only a few of us?

                          Where are the redo-logs actually stored? there is very little information regarding this that i can find..

                           

                          More info would be appriciated

                          • 10. Re: Bluescreen 7b after restore
                            Phatsta Enthusiast

                            idag tisdag 30/4 är jag ledig. Återkommer igen på torsdag. Vid brådskande ärenden, ring vår tillfälliga växel på nr 070-279 71 17. Ha en trevlig dag!

                            /Daniel Jokinen

                            Linford Communication AB

                            • 11. Re: Bluescreen 7b after restore
                              gwijsman Lurker

                              Long live the new Dutch King too!

                              Kind Regards,

                              Gert

                              ( http://www.guardian.co.uk/world/video/2013/apr/30/queen-beatrix-abdicates-son-dutch-king-video )

                               

                              Van: Phatsta communities-emailer@vmware.com

                              Verzonden: dinsdag 30 april 2013 13:24

                              Aan: gert@wijsman.net

                              Onderwerp: New message: "Bluescreen 7b after restore"

                               

                              VMware Communities<http://communities.vmware.com/index.jspa>

                               

                              Bluescreen 7b after restore

                              reply from Phatsta<http://communities.vmware.com/people/Phatsta> in ghettoVCB - View the full discussion<http://communities.vmware.com/message/2234751#2234751

                              • 12. Re: Bluescreen 7b after restore
                                lunadesign Novice

                                UGH!

                                 

                                I just had a drive crash on my ESXI 5.0 host and am encountering exactly this problem with my Win 2008 R2 VMs.  Am I completely hosed?  IE, is there any way to get a useful restore out of my quiesced backups?

                                 

                                I *did* do a handful of restore tests when I started using GhettoVCB but none of them happened to be Win 2008 R2 VMs.  I made the mistake of assuming that if it worked for a few Windows OSes that it would work for all Windows OSes.

                                • 13. Re: Bluescreen 7b after restore
                                  Phatsta Enthusiast

                                  I'm not sure I understand your problem, but if you have backups of your guest machines, you can always use VMware Converter to restore them if you can't use the restore function in ghetto. That will also allow you to change the vm's settings / hard drive config and size if you need to.

                                   

                                  If the crashed drive has caused your guest backups to contain file system errors then I'd say you'd probably want to use the Windows DVD / ISO to do a "repair" install of the OS.

                                   

                                  I hope that at least helps some. Good luck.

                                   

                                   

                                  Med vänliga hälsningar

                                   

                                  Daniel Jokinen

                                  Linford Communication AB

                                  019-277 22 73

                                   

                                   

                                  22 sep 2013 kl. 08:41 skrev "lunadesign" <communities-emailer@vmware.com<mailto:communities-emailer@vmware.com>>:

                                   

                                  VMware Communities<https://communities.vmware.com/index.jspa>

                                   

                                  Bluescreen 7b after restore

                                  created by lunadesign<https://communities.vmware.com/people/lunadesign> in ghettoVCB - View the full discussion<https://communities.vmware.com/message/2290462#2290462>

                                  • 14. Re: Bluescreen 7b after restore
                                    lunadesign Novice

                                    I *do* have backups of my VMs on a separate non-failed hard drive.  However, based on my partial understanding of how VSS works and goppi's excellent post, I believe the way GhettoVCB made the backups renders them unusable for restores.

                                    1 2 Previous Next