PowerCLI backup of ESXi configuration no more working after upgrade ESXi to 6.5 update1

[AntonioMP]

Hi,

    this week we started upgrading our ESXi hosts from 6.5 to 6.5 update 1, using update manager.

For many years we made the ESXi hosts configuration backup using the vSphere CLI, by the vicfg-cfgbackup command, but after the 6.5 release we have same

errors that seems related to nonpresent Perl libraries.

So we started backupping ESXi configuration using the following PowerCLI command:

Get-VMHostFirmware -VMHost ESXi_host_IP_address -BackupConfiguration -DestinationPath output_directory

Now, after upgrading the first 4 host to ESXi 6.5 update 1, this backup no works anymore.

We obtain the following error:

PowerCLI C:\> Get-VMHostFirmware -VMHost srvesx013.xxxxxx -BackupConfiguration -DestinationPath C:\BCK

Get-VMHostFirmware : 23/08/2017 15:41:05        Get-VMHostFirmware              A general system error occurred: Internal error

At line:1 char:1

+ Get-VMHostFirmware -VMHost srvesx013.xxxxxx -BackupConfigura ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], SystemError

    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_ViError,VMware.VimAutomation.V

   iCore.Cmdlets.Commands.Host.GetVMHostFirmware

While for an host that is still at 6.5 release, the command works well, as usual:

PowerCLI C:\> Get-VMHostFirmware -VMHost srvesx012.xxxxxxx -BackupConfiguration -DestinationPath C:\BCK

Host            Data

----            ----

srvesx012... C:\BCK\configBundle-srvesx012.xxxxxxx.tgz

Any hints?

[LucD]

Seems to be working for me.

Which PowerCLI version are you using?

Do a Get-PowerCLIversion

[AntonioMP]

Hi Luc,

thanks for you reply, on my notebook I have this PowerCLI version:

PowerCLI C:\> Get-PowerCLIversion

PowerCLI Version

----------------

   VMware PowerCLI 6.5 Release 1 build 4624819

---------------

Component Versions

---------------

   VMware Cis Core PowerCLI Component 6.5 build 4624453

   VMware VimAutomation Core PowerCLI Component 6.5 build 4624450

   VMWare ImageBuilder PowerCLI Component 6.5 build 4561891

   VMWare AutoDeploy PowerCLI Component 6.5 build 4561891

   VMware HA PowerCLI Component 6.0 build 4525225

   VMware Licensing PowerCLI Component 6.5 build 4624822

   VMware Storage PowerCLI Component 6.5 build 4624820

   VMware Vds PowerCLI Component 6.5 build 4624695

PowerCLI C:\>

This is the same version that can be downloaded also today from myware.com site, under "Driver & Tools", "Automation Tools and SDK(s)".

I've noticed it now, after your replica, that the release date is still November 2016.

I also tried to install (not on My notebook, but on a 2012R2 VM) a more recent PowerCLI version following this tutorial:

VMware PowerCLI 6.5.1 Installation Walkthrough - YouTube

I certainly preferred the old offline installer and in fact I must have done something wrong:

After installing the module from PSGallery, the module appears as available:

PS C:\Windows\system32> Get-Module -Name VMware* -ListAvailable

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version    Name                                ExportedCommands

---------- -------    ----                                ----------------

Binary     6.5.1.5... VMware.DeployAutomation             {Add-DeployRule, Add-ProxyServer, Add-ScriptBundle, Copy-D...

Binary     6.5.1.5... VMware.ImageBuilder                 {Add-EsxSoftwareDepot, Add-EsxSoftwarePackage, Compare-Esx...

Manifest   6.5.2.6... VMware.PowerCLI

Binary     6.5.2.6... VMware.VimAutomation.Cis.Core       {Connect-CisServer, Disconnect-CisServer, Get-CisService}

Binary     6.5.1.5... VMware.VimAutomation.Cloud          {Add-CIDatastore, Connect-CIServer, Disconnect-CIServer, G...

Manifest   6.5.1.5... VMware.VimAutomation.Common

Binary     6.5.2.6... VMware.VimAutomation.Core           {Add-PassthroughDevice, Add-VirtualSwitchPhysicalNetworkAd...

Binary     6.0.0.5... VMware.VimAutomation.HA             Get-DrmInfo

Binary     7.1.0.5... VMware.VimAutomation.HorizonView    {Connect-HVServer, Disconnect-HVServer}

Binary     6.5.1.5... VMware.VimAutomation.License        Get-LicenseDataManager

Binary     6.5.1.5... VMware.VimAutomation.PCloud         {Connect-PIServer, Disconnect-PIServer, Get-PIComputeInsta...

Manifest   1.0.0.5... VMware.VimAutomation.Sdk            {Get-PSVersion, Get-InstallPath}

Binary     6.5.1.5... VMware.VimAutomation.Srm            {Connect-SrmServer, Disconnect-SrmServer}

Binary     6.5.1.5... VMware.VimAutomation.Storage        {Copy-VDisk, Export-SpbmStoragePolicy, Get-NfsUser, Get-Sp...

Script     1.0        VMware.VimAutomation.StorageUtility Update-VmfsDatastore

Binary     6.5.1.5... VMware.VimAutomation.Vds            {Add-VDSwitchPhysicalNetworkAdapter, Add-VDSwitchVMHost, E...

Binary     6.5.1.5... VMware.VimAutomation.vROps          {Connect-OMServer, Disconnect-OMServer, Get-OMAlert, Get-O...

Binary     6.5.1.5... VMware.VumAutomation                {Add-EntityBaseline, Copy-Patch, Get-Baseline, Get-Complia...

But can't be loaded when I try to use them:

PS C:\Windows\system32> Get-PowerCliVersion

Get-PowerCliVersion : The 'Get-PowerCliVersion' command was found in the module 'VMware.VimAutomation.Core', but the

module could not be loaded. For more information, run 'Import-Module VMware.VimAutomation.Core'.

At line:1 char:1

+ Get-PowerCliVersion

+ ~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : ObjectNotFound: (Get-PowerCliVersion:String) [], CommandNotFoundException

    + FullyQualifiedErrorId : CouldNotAutoloadMatchingModule

PS C:\Windows\system32> Import-Module VMware.VimAutomation.Core

Import-Module : Could not load file or assembly 'log4net, Version=1.2.10.0, Culture=neutral,

PublicKeyToken=692fbea5521e1304' or one of its dependencies. The system cannot find the file specified.

At line:1 char:1

+ Import-Module VMware.VimAutomation.Core

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException

    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand

Thanks in advance for any suggestions.

Regards

[LucD]

You did uninstall the old version, like the tutorial states?

Can you try

Get-Module -Name VMware* -ListAvailable | Import-Module

And which PowerShell verson are you running?

Display the content of $PSVersionTable

[AntonioMP]

Hi Luc,

   thank again for your support, on the 2012R2 VM there wasn't any previous version of PowerCLi, while

Powershell version is 5.0.10586.117

PS C:\Windows\system32> $PSVersionTable

Name                           Value

----                           -----

PSVersion                      5.0.10586.117

PSCompatibleVersions           {1.0, 2.0, 3.0, 4.0...}

BuildVersion                   10.0.10586.117

CLRVersion                     4.0.30319.34014

WSManStackVersion              3.0

PSRemotingProtocolVersion      2.3

SerializationVersion           1.1.0.1

PS C:\Windows\system32>

While the command Get-Module -Name VMware* -ListAvailable

give me a list of modules as for my previous post, the command you suggested give me an error:

PS C:\Windows\system32> Get-Module -Name VMware* -ListAvailable | Import-Module

Import-Module : Could not load file or assembly 'log4net, Version=1.2.10.0, Culture=neutral,

PublicKeyToken=692fbea5521e1304' or one of its dependencies. The system cannot find the file specified.

At line:1 char:1

+ Get-Module -Name VMware* -ListAvailable | Import-Module

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Import-Module], FileNotFoundException

    + FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.ImportModuleCommand

It seems the same issue of this other communities thread:

https://communities.vmware.com/thread/547725

I've read here that Powershell x86 is not affected from this bug.

So I've tried just now the command get-vmhost | get-vmhostfirmware -BackupConfiguration -DestinationPath “C:\Download”

on a x86 powershell windows and it works well but only for host non yet upgraded to 6.5 update 1 (they are 6.5.0 Build 5224529)

The others (6.5.0 Build 5969303) give this error:

get-vmhost | get-vmhostfirmware -BackupConfiguration -DestinationPath “C:\Download”

get-vmhostfirmware : 24/08/2017 19:06:11    Get-VMHostFirmware        A general system error occurred: Internal error

At line:1 char:14

+ ... et-vmhost | get-vmhostfirmware -BackupConfiguration -DestinationPath  ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], SystemError

    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_ViError,VMware.VimAutomation.V

   iCore.Cmdlets.Commands.Host.GetVMHostFirmware

Other commands, like for example get-vmhost work great.

[LucD]

Yes, looks indeed like that the issue from that other thread.
Let me see if I can ping someone.

[LucD]

Are you a local admin on that station?
If not, can you try to import the modules as an administrator?

Also make sure no files are not blocked by your AV.

[jakerpowercli]

Do you by chance have Crystal Reports installed on your client machine?

[AntonioMP]

Hi,

        I'm a Domain Administrator and always opened the powershell windows with the "Run as administrator" option,

so also the import of PowerCLI was made with elevated privileges.

I'll try to made a new installation on a clean workstation and let you know as it goes.

Regards,

[AntonioMP]

Nice point..., yes, on the 2012R2 VM where I tried to made the backup, there is also Crystal Reports installed.

So I'll try reinstall everything from scratch on a "clean" machine and I'll let you know as it goes.

[AntonioMP]

Hi everybody and thanks for your help.

I've tried to reinstall the latest PowerCLI from scratch,

both on my laptop and on a clean workstation at office.

Results:

1. LOG4NET ERROR ON X64 POWERSHELL

   The log4net error on x64 Powershell seems to be related to the Crystal Reports

   installation on the 2012R2 VM where I tried yesterday.

   Thanks to jakerpowercli for this suggestion.

   No more this error on the new installations.

2. ON A CLEAN WINDOWS 8.1 WORKSTATION

   Installed the latest Windows Management Framework to update Powershell to latest version -->OK

   Installed PowerCLI 6.5.2 --> OK

   Both from an x86 than x64 powershell Windows (Run as Administrator) tried to made a configuration

   backup with the command (after Connect-ViServer):

   get-vmhost | get-vmhostfirmware -BackupConfiguration -DestinationPath "c:\BCK"

   For host 6.5.0 Build 5224529 it all OK, while for ESXi 6.5 update 1 hosts, except one, I have the same

   error of the previous post:

   get-vmhostfirmware : 25/08/2017 12:14:23        Get-VMHostFirmware              A general system error occurred: Internal error

In riga:1 car:14

+ ... et-vmhost | get-vmhostfirmware -BackupConfiguration -DestinationPath  ...

+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    + CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], SystemError

    + FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_ViError,VMware.VimAutomation.V

   iCore.Cmdlets.Commands.Host.GetVMHostFirmware

3. ON A WINDOWS 10 LAPTOP WITH ALL THE LATEST UPDATES

   Installed PowerCLI 6.5.2 -->OK

   Same error as above for host 6.5 Update 1, except one, works ok for 6.5 hosts without this update.

4. IMPORTANT NEWS

   Just yesterday I upgraded another host to 6.5 update 1 and, because of a disconnected NIC on the remote site, Update Manager failed to upgrade,

   so I made and inplace (by IDRAC) upgrade using ISO file. This host is the only upgraded host where the backup of configuration as above

   works well. Very strange behaviour...

[EricJamesWL]

I had this exact problem on some hosts in my lab on which I recently did a fresh install of 6.5 Update 1 using the HP July 2017 custom ISO. The error was related to an issue with the "scratch" storage location for the hosts. The Get-VMHostFirmware command writes the tgz file to /scratch/downloads before outputting to the location specified in the -DestinationPath parameter. If the host cannot write to /scratch/downloads, the command fails.

The issue with the scratch location was related to some faulty Emulex vibs packaged with the HP custom ISO (see more on this thread: Configured scratch location doesn't work after upgrade to ESXi 6.5.) After removing the vib on my hosts I was able to run the Get-VMHostFirmware command with no issues.

To test if this is the issue with your hosts, SSH into one of them and check the contents of your scratch location with the "ls scratch" command. What is the output? If you don't see the "downloads" folder in there, that's the problem. CD into the scratch folder and create the "downloads" directory with "mkdir downloads", then try the Get-VMHostFirmware command again from PowerCLI and see if it succeeds.

[vlaxa]

Confirmed, same issue for upgraded host:

ESXi	6.0.0 U2 build: 3620759 => 6.5.0 U1 build: 5969303

Fresh install of 6.5.0 build: 5969303 worked OK.

Update:

mkdir downloads

in the /scratch/ fixed this issue

[EricJamesWL]

That will fix it temporarily, but this bug makes the scratch location non-persistent. So when you reboot the downloads dir will disappear.

You will need to either update or remove the Emulex vibs to fix it permanently.

[vlaxa]

Sure, it was updated with: VMW-ESXi-6.5.0-elxiscsi-11.4.1184.0-6410288

[AntonioMP]

Hi Eric,

         I'am sorry for my late reply.

Thank for pointing me in the right direction.

I'm using a DELL customized ISO and also I have the scratch partition problem after the upgrade (Emulex vibs included)

Creating the downloads folder under /scratch made me able to backup and restore my esxi configuration.

Thanks again, regards

[AntonioMP]

Hi, thanks for pointing to the build that fixes the problem. I temporarily made the workaround to create the download folder under the scratch partition (all is working ok now), waiting to apply the latest build to all of our ESXi servers.

Regards,

[sushilkm]

Hello Luc,

Did we ever find a solution to this . I tried all workarounds including downloads folders in scratch and none seems to be working. I am running exactly the same build version as Original Poster have and getting same error . Strangely when i check the downloads folder in scratch, it is there with no contents.

I am getting desired output when i do

Get-VMhost hostname |  Get-VMhostfirmware

It does outputs the Hostname and Uploadurl

but it never exports it to desired location . Very frustrating.

PowerCLI C:\> $PSVersionTable

Name Value

---- -----

PSVersion 4.0

WSManStackVersion 3.0

SerializationVersion 1.1.0.1

CLRVersion 4.0.30319.36399

BuildVersion 6.3.9600.18773

PSCompatibleVersions {1.0, 2.0, 3.0, 4.0}

PSRemotingProtocolVersion 2.2

PowerCLI C:\> Get-PowerCLIVersion

PowerCLI Version

----------------

VMware PowerCLI 6.5 Release 1 build 4624819

---------------

Component Versions

---------------

VMware Cis Core PowerCLI Component 6.5 build 4624453

VMware VimAutomation Core PowerCLI Component 6.5 build 4624450

VMWare ImageBuilder PowerCLI Component 6.5 build 4561891

VMWare AutoDeploy PowerCLI Component 6.5 build 4561891

VMware Vds PowerCLI Component 6.5 build 4624695

VMware Cloud PowerCLI Component 6.5 build 4624821

VMware HA PowerCLI Component 6.0 build 4525225

VMware HorizonView PowerCLI Component 7.0.2 build 4596620

VMware Licensing PowerCLI Component 6.5 build 4624822

VMware PCloud PowerCLI Component 6.5 build 4624825

VMware Storage PowerCLI Component 6.5 build 4624820

VMware vROps PowerCLI Component 6.5 build 4624824

  VMware vSphere Update Manager PowerCLI 6.5 build 4540462

PS C:\Users\username> Get-VMHostFirmware *************  -BackupConfiguration -DestinationPath C:\Backup

Get-VMHostFirmware : 1/12/2018 9:53:29 PM Get-VMHostFirmware        Unable to connect to the remote server

At line:1 char:1

+ Get-VMHostFirmware wprappl01-02.tceh.net -BackupConfiguration -DestinationPath C ...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo          : NotSpecified: (:) [Get-VMHostFirmware], ViError

+ FullyQualifiedErrorId : Client20_SystemManagementServiceImpl_BackupVmHostFirmware_DownloadError,VMware.VimAutoma

tion.ViCore.Cmdlets.Commands.Host.GetVMHostFirmware

[ShridharH]

Thank you. This worked for me.