jasonboche's Posts

Confirmed solution to this issue.
Unfortunately I'm in the same boat across 4 environments all sharing a content library of templates. Software defined. Gotta love it.
Thanks for this. I summarized my experience here:  http://www.boche.net/blog/index.php/2015/08/15/vcloud-director-vdnscope-1-could-not-be-found/
I've seen similar with RHEL7 guests in vSphere 6, with the following notes and workarounds: -Installing a 'minimalist' version of RHEL7 and then installing open-vm-tools (from the RHEL7 DVD yu... See more...
I've seen similar with RHEL7 guests in vSphere 6, with the following notes and workarounds: -Installing a 'minimalist' version of RHEL7 and then installing open-vm-tools (from the RHEL7 DVD yum repo) as well as the deployPkg Tools Plug-in (from VMware following KB2075048), guest customization doesn't work properly. It appears to run but the resulting clone is not unique. -Installing RHEL7 along with the Guest Agents Add-On, open-vm-tools is automatically installed (from the RHEL7 DVD) as well as the deployPkg Tools Plug-in (from VMware following KB2075048), as well as some maybe other package(s) bundled with Guest Agents whereby guest customization and resulting clones are unique. The difference between failure and success points to the delta between manually installing the open-vm-tools package from RHEL7 which is failure prone and deploying RHEL7 in the Infrastructure Server role along with the Guest Agents Add-On which works. (In either method, for guest customization to have a chance to work, you must install the deployPkg Tools Plug-in from VMware following KB2075048 before sealing the template and attempting customization or cloning) There may be more to it than that but I'm still researching in the lab via trial and error. EDIT: I've got it sorted. I updated my findings and many other personal tidbits in the following blog post. A RHEL 7 Minimal Install is missing PERL which guest customization requires. The installation of deployPkg Tools Plug-in from VMware doesn't check for the existence of PERL. http://www.boche.net/blog/index.php/2015/08/09/rhel-7-open-vm-tools-and-guest-customization/
vCenter Server 5.5 Update 2 on Windows Server 2008 R2. Indeed, attempting the installation using the local Administrator account instead of a domain account (domain admin) resolved the issue. ... See more...
vCenter Server 5.5 Update 2 on Windows Server 2008 R2. Indeed, attempting the installation using the local Administrator account instead of a domain account (domain admin) resolved the issue. I hadn't seen this problem in the past with previous versions of vCenter Server on Windows. Thank you for the tip on this one. Jas
I wouldn’t mind seeing this as a solution as long as VMware backs it up and it’s used solely to escape from a hung state. Obviously underlying storage problems may exist & would need to be resolv... See more...
I wouldn’t mind seeing this as a solution as long as VMware backs it up and it’s used solely to escape from a hung state. Obviously underlying storage problems may exist & would need to be resolved as a separate measure. Jas
I ran into a similar issue recently, worked with VMware support, and documented the resolution here: vCloud Director and vCenter Proxy Service Failure
ian0x0r wrote: Did you ver get this reolved Jason? I've got the same situation now with Dell Equallogic kit and I dont really want to go down the root of uninstalling and re-installing SRM to... See more...
ian0x0r wrote: Did you ver get this reolved Jason? I've got the same situation now with Dell Equallogic kit and I dont really want to go down the root of uninstalling and re-installing SRM to fix this. Thanks, Ian Lee and a few others were able to spend considerable time with me on this.  In my instance, I was able to solve the problem without uninstalling/reinstalling the environment.  I did so by cleaning up the storage and replication at both sites which SRM managed.  One or more of the volumes at one of more of the sites was in a precarious state from a replication standpoint.  This was causing the SRA to return a status to SRM which SRM did not like and thus would not proceed until the precarious state was resolved and a more appropriate/proper status code could be returned by the SRA.  Once that underlying storage issue was remedied, the force cleanup worked instantly as one might expect it to and I was then able to tear down the recovery plan and the hung protection group, fix the direction of replication as needed for the LUN(s), then simply recreate the protection group and recovery plan. As I cross posted in the other thread linked above, I think there needs to be an ability to cut away the protection group & recovery plan without reinstalling the environment.  From the looks at the other thread, that person will not be able to forcefully remove the protection group.  SRM should not allow the SRA to interfere with the integrity & protection of the rest of the environment, particularly where there could be other array models involved.  This is an Achilles Heel from an architecture standpoint. Thank you, Jas
Lee Dilworth wrote: i've seen this once before on dell EQ and it was caused by a device on the array being in a weird state that was causing the SRA to report and incorrect results when disco... See more...
Lee Dilworth wrote: i've seen this once before on dell EQ and it was caused by a device on the array being in a weird state that was causing the SRA to report and incorrect results when discoverDevices was called. this basically meant the offending device state could never be changed and reprotect could never complete. the "weird" device was caused because on the target array the same id was being used for a source and target device at the same time and the SRA couldn't handle that. once the device id was correct the customer created a new recovery plan for that protection group, ran the reprotect again, this failed as expected, ran the reprotect again with "force cleanup" ticked, this then ran to completetion and got the protection group back into a normal state. Lee, If you're referring to the case you and I worked together on here, it was a Dell Compellent SAN, not EQL.  I'm going to follow up on that discussion since in the end I was able to remove the protection group without reinstalling but I'm not sure that solution will do much good in the case in this thread.  In that vein, it reinforces my opinion that within SRM there needs to be an ability to cut away the protection group & recovery plan without reinstalling the environment.  SRM should not allow the SRA to interfere with the integrity & protection of the rest of the environment, particularly where there could be other array models involved.  This is an Achilles Heel from an architecture standpoint. Thank you, Jas
I wrote a blog article on Expanding Transfer Server Storage  yesterday which you may or may not be interested in.  You can find that  article at the link below: http://www.boche.net/blog/index... See more...
I wrote a blog article on Expanding Transfer Server Storage  yesterday which you may or may not be interested in.  You can find that  article at the link below: http://www.boche.net/blog/index.php/2011/12/05/expanding-vcloud-director-transfer-server-storage/
I haven't found a lot of detailed information about Transfer Server Storage.  I wrote a blog article on Expanding Transfer Server Storage yesterday which you may or may not be interested in.  You... See more...
I haven't found a lot of detailed information about Transfer Server Storage.  I wrote a blog article on Expanding Transfer Server Storage yesterday which you may or may not be interested in.  You can find that article at the link below: http://www.boche.net/blog/index.php/2011/12/05/expanding-vcloud-director-transfer-server-storage/
I just noticed a 2nd error in vCD about this VM - sysprep failed to run.  The time stamp appears to be based on the 2nd power on of the VM where guest recustomization was forced.  The error messa... See more...
I just noticed a 2nd error in vCD about this VM - sysprep failed to run.  The time stamp appears to be based on the 2nd power on of the VM where guest recustomization was forced.  The error message is: "Guest customization failed on this virtual machine. Error is : SID change was not successful" As a further test, I powered off the VM, then powered it back on with force recustomization & that did not work. I'm back to the point where I cannot log on with the administrator account.
Power on and Force Recustomization did indeed invoke customization on the XP x64 vApp on the subsequent power on. During this customization, it halted with an error message which I grabbed a s... See more...
Power on and Force Recustomization did indeed invoke customization on the XP x64 vApp on the subsequent power on. During this customization, it halted with an error message which I grabbed a screen of and posted to this thread.  After clicking OK, it rebooted. I logged on and Windows proceeded to run guest customization, reboot, complete customization, reboot.  This time it appeared to complete successfully with no errors and once the VM rebooted the last time it was back to waiting at the Windows XP guest OS logon screen.  At this point the guest OS has been customization with the change of the host name & the specified Administrator password. I took a look at c:\windows\temp\customize-guest.log and it appears to have all the information from the 2nd customization but not the first customization which incurred the failure. The guest customization tools are working but not at first power on in vCD with Windows XP x64. Jas
vSphere 5.0 GA vCD 1.5 GA Single vCD Cell Server I'm curious if anyone is seeing any guest customization issues with legacy Windows guest operating systems requiring sysprep.  I've got a van... See more...
vSphere 5.0 GA vCD 1.5 GA Single vCD Cell Server I'm curious if anyone is seeing any guest customization issues with legacy Windows guest operating systems requiring sysprep.  I've got a vanilla Windows XP x64 template I've imported into vCD.  When I deploy a vApp using this template, I can see in the guest console that guest cuztomization isn't running.  Entering the administrator credentials at logon quickly returns me to the logon prompt. I did validate the sysprep files were correct and functional when I used them with guest customization in conjunction with a vCenter Server. I've run through the procedure of copying over the sysprep files to the cell server and generating the sysprep package twice with the same results. Windows Server 2008 R2, Windows 7 x64, Windows Server 2003 R2 x64, and RHEL 5.6 x64 have no guest customization/sysprep issues. Thank you, Jas
I ran into a similar situation where a vApp would not start because vCloud Director 1.5 couldn't find the right sysprep files for Windows XP x64.  I had previously provided the correct sysprep fi... See more...
I ran into a similar situation where a vApp would not start because vCloud Director 1.5 couldn't find the right sysprep files for Windows XP x64.  I had previously provided the correct sysprep files based on http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1005593 and then ran createSysprepPackage.sh which completed successfully. What's odd is that the MS site http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=8287 talks about that version of sysprep being for Windows Server 2003 SP2 x64 (and not XP x64). So then I grabbed an XP x64 CD and copied the contents of deploy.cab to the vCD server and reran createSysprepPackage.sh which completed successfully.  Still had the same error trying to power on the vApp. I then tried service vmware-vcd restart but it failed to shut down the service. I then rebooted the vCD server which resolved the issue.  Unfortunately, I don't know if the fix was the server reboot, the replacement sysprep files from the XP x64 CD, or both. Jas
Ryan wrote: Hi Jason, It sounds like it's failing in the section of code I suspected then.  Feel free to continue working with Lee on this and sending him whatever logs/screenshots you hav... See more...
Ryan wrote: Hi Jason, It sounds like it's failing in the section of code I suspected then.  Feel free to continue working with Lee on this and sending him whatever logs/screenshots you have as I can get them from him if an SR can't be filed. I'm curious about a couple things when the group is in this state and just want some confirmation.  If you get a chance and don't mind trying some things out upon reproducing this then can you answer the following for me: 1) Can you edit the recovery plan to remove the group?  I think the plan may be in the Incomplete Reprotect state and it should allow editing.  (Maybe you addressed this already in your original post.)  This would allow other groups to proceed with reprotect.  Otherwise creating a new plan with working groups may be another option. 2) Can you unprotect VMs from the group in the "Protection Group" "Virtual Machines" tab?  This shoud be enabled.  If so then does it work?  If you remove all the VMs in the group and then run Reprotect again does it now work?  (Given where it's failing I suspect it won't work, but it's worth a shot.) 3) Is the ability to remove the protection group just not available when it's in this state? I suspect it may be given our UI specification.  Otherwise, if it is actually available then with what error does it fail when invoked? Thanks, -Ryan As luck would have it, I wasn't able to reproduce the problem earlier this morning for 2 hours, then this afternoon it showed up in a live customer demo   Reprotect and Reprotect with Force Cleanup does not complete; instantly fails.  This time due to step 1.0/1.1 failing "Error - Failed to reverse replication for failed over devices. Cannot process device '21666' with role 'target' when expected device with role 'promotedTarget'." 1)  Yes.  After I remove the PG, the RP goes into an error state about "This plan cannot be run because it doesn not contain any protection groups.".  On the PG side, the PG is still in "Reprotecting..." state & it cannot be edited or deleted.  I then ran a Reprotect against the Protection Group after the VMs were removed from the PG individually. This resulted in an immediate failure as did the Force Cleanup option. 2)  Yes.I can remove all VMs from the protection group.  Then at that point, the PG still exists in a "Reprotecting..." state and cannot be edited or deleted. 3)  I've only seen the inability to remove the PG when it's hung in the "Reprotecting..." state. Screenshots and logs sent to Lee Dilworth via FTP/email followup. Thank you, Jas Message was edited by: jasonboche  Added additional step/reponse to 1)
Thanks for the reply Ryan. I can pretty consistently reproduce the issues; I'm unable to open an SR with my current personal or Global Technology Alliance Partner account but I'll definately get ... See more...
Thanks for the reply Ryan. I can pretty consistently reproduce the issues; I'm unable to open an SR with my current personal or Global Technology Alliance Partner account but I'll definately get those into VMware if there is a way that I can (perhaps upload to the root of the FTP site or email to Lee as the logs should be small enough for a single run). In your 2nd paragraph, you've described precisely what I'm seeing with the protection group (Reprotecting...) You may be correct on the exact point of failure - I've sent a screenshot tonight to Lee's email address. "Configure Protection to Reverse Direction" is the step that fails in the screenshot. Once in this state of "Reprotectiong...", Force Cleanup doesn't resolve the issue and I can't manually clean up by deleting protection groups or recovery plans since the UI sees them as actively running.
vmwnelson wrote: Have you double-checked that the storage you're using is at the minimum supported firmware rev. for the latest SRA?  I have seen this when I was working with a falconstor VSA... See more...
vmwnelson wrote: Have you double-checked that the storage you're using is at the minimum supported firmware rev. for the latest SRA?  I have seen this when I was working with a falconstor VSA that was not at the min MP4 patch level.  Same results as you where I needed to reinitialize to clear that protection group and successfully go through with the reprotect.  Putting in the upgrade fixed the issue in my case. Supported/certified storage: Yes To me this is an SRM framework/workflow issue.  Storage management is SRA's responsibility.  SRM should leave its problems with the SRA and those issues should not impact protection groups, and recovery plans to the point that the SRM application needs to be uninstalled which could expose a RTO vulnerability to protection groups which are based on other SRAs/array pairs.
Lee, Pure vSphere 5.0 GA & SRM 5.0 GA environment. I have some older logs but I'll just jump in an alternate lab and reproduce fresh logs. I'll take the storage specific questions to email... See more...
Lee, Pure vSphere 5.0 GA & SRM 5.0 GA environment. I have some older logs but I'll just jump in an alternate lab and reproduce fresh logs. I'll take the storage specific questions to email.
Stefan Tsonev wrote: Did SRM or VC crash during reprotect operation? If you go to the offending Protection Group and into the Virtual Machines list, do you see any VMs with errors? If so, ... See more...
Stefan Tsonev wrote: Did SRM or VC crash during reprotect operation? If you go to the offending Protection Group and into the Virtual Machines list, do you see any VMs with errors? If so, could you try to "Remove Protection" on these VMs. Thanks Neither crashes that I'm aware of.  The reprotect fails, throwing some errors in the process.  The root cause has something to do with failing to reverse replication via the SRA. At that point, the protection group is left in a pseudo state of a reprotection in progress (mandating that a successful reprotect or cleanup be compelted, but force cleanup doesn't work) such that anything it depends on cannot be removed (ie. Array pairs).  I'm not able to edit or remove the protection group itself when it is in this state and I don't recall being able to remove individual VMs but I'll take a look at that next time it happens. Message was edited by: jasonboche