VM backup fail : VBA EMC Networker 8 vsphere 5.5

[Hachem7]

Hello , im seeking help for the error below

Extracted 12 lines, suppressed 548 lines.

See file "/nsr/logs/policy/Gold/VMware/883859-Carte SN-2024-3-7-20-3-15" for complete logs. Extracted Lines:

2024-03-07 20:02:30 avvcbimage Error <0000>: [IMG0008] Starting with VDDK 6.0, it is required to set the SSL thumbprint flag for utilizing any backup/restore functionality. Please check the documentation for details. 2024-03-07 20:02:31 avvcbimage Warning <0000>: [IMG0008] VixDiskLib_PrepareForAccess returned fault(3), (3) One of the parameters was invalid

2024-03-07 20:02:31 avvcbimage Warning <0000>: [IMG0008] VixDiskLib_Cleanup() returned code(3),'(3) One of the parameters was invalid", 0 VMx require cleanup

2024-03-07 20:02:52 avvcbimage Error <0000>: [IMG0008] VixDiskLib_ConnectEx returned 3 (3) One of the parameters was invalid

2024-03-07 20:02:52 avvcbimage Error <0000>: [IMG0008] VixDiskLib_Connect(host:10.1.3.76 account:root dc:/&dsName: vmxpath: [VNX_datastore04] Carte_SN/Carte_SN.vmx port:443vmmoref:vm-287) detected a problem, returned (3) One of the parameters was invalid

2024-03-07 20:02:52 avvcbimage Warning <40632>: VixDiskLib vMotion reservation release requested for prior reservation

2024-03-07 20:02:52 avvcbimage Warning <0000>: [IMG0008] VixDiskLib vMotion reservation release returned fault(3) (3) One of the parameters was invalid - host:10.1.3.76 account:root dc:/DC-SNCFT&dsName: vmxpath:[VNX_datastore04] Carte_SN/Carte_SN.vmx port:443vmmoref:vm-287

2024-03-07T20:02:52.980-01:00 avnwcomm Warning <0000>: Plug-in returned non-zero exit code (205). Aborting NetWorker connection.

G2024-03-07T20:02:58.991-01:00 avnwcomm Warning <6690>: CTL workorder "VMware-1709838030326" non-zero exit status 'code 205: plugin error 05'

2024-03-07T20:02:58.991-01:00 avnwcomm Warning <0000>: The plug-in gave a bad return code: 205

[NateNateNAte]

Was it working previously?

And what changed in your environment prior to this failure?

It looks as if your root account either could not authenticate, or there was a permissions difference (parameter) that changed. 

More detail on your environment and the context of this failure would help. (The logs are also good, but only convey the machine perspective)

[Hachem7]

Hi thank you for your answer ,

as far i know it used to work in December 2023 then they had power outage.

what root account do you mean ? the ESX one ?

i keep suspecting this : from vmware official source  "As of VDDK 6.0 both SSL certificate verification and SSL thumbprint checking are mandatory and cannot be disabled. The Windows registry and Linux SSL setting are no longer checked, so neither has any effect." "The default SSL certificates of vCenter Server are valid for 10 years and that of ESX/ESXi 4.x/5.x are valid for a period of 11.5 years."

They are using the old java interface to manage vcenter not the web browser one

[NateNateNAte]

Ah, I see.  Yes in vSphere 6 and beyond it became mandatory to use SSL and we wend away from the java based client. 

Are you still working in vSphere 5.5?  If so, I'm assuming it's because of some type of security or approval restriction.  That's unfortunate, but I've been there.  

Yes, I meant the ESXi or VCS root account.  But with this new info, you can ignore that line of thought for now.

So are they able to use the web interface?  It also sounds like its an API call for this EMC backup interface. The web interface was OK in vSphere 5.x but much better from v6.x onward. If there was an upgrade to the java engine, it could have closed off an API port.  Similarly, after a power outage, if there was a recovery of an old config file, or even replacing with a new config file (on the network/firewall side) that could also explain why the API call is cut off.  

And also, based on your research, it's also VERY possible that the certificate for the java based vSphere client has finally expired and that is really hard to overcome.  You'll want to weigh the impacts of not having a current back-up service running against upgrading vSphere (to at least v6.7 if not all the way to v8) or implementing a different back-up solution. Or, if you cannot upgrade vSphere (just know more problems will begin to surface as 5.5 is almost out of it's support window) perhaps there is a different way to scrip the backup from the EMC side?  

I know, I'm kinda throwing out a lot of ideas, but not knowing your infrastructure in detail, I want to provide options based on the situation. 

[Hachem7]

thank you for everthing i really appreciate you

yes it's vsphere 5.5

im get back to you on monday for  specific details  i need more ideas , that's what im looking for :

i think the problem is this :

Certificate verification is mandatory in VMware VDDK 6.0. Commvault (Networker EMC in my case ) software handles the verification for the connection to vCenter; but certificate verification is also required when a connection is relayed to a specific ESX host.

If the vCenter SSL setting vCenter requires verified host SSL certificates is disabled, host certificate thumbprints are not automatically verified when the host is added to the inventory. As a result, the certificate thumbprint for the host is not relayed when establishing an NFC connection to that host.  Any host that does not have a verified SSL certificate thumbprint will fail to be accessed by the VDDK.

Please do you know how to do this " Center requires verified host SSL certificates option is displayed in any version of vCenter, manually verify the SSL certificate for each ESX host"

my plan is to first is to verify Validity Date for the SSL certificates of vcenter : download openssl in ssl vmware directory and execute

openssl x509 -in rui.crt -noout -text

[BarryGrowler]

I would suggest first of all to enable SSL certificate verification for ESXi hosts in vCenter. This forces vCenter to retrieve and store the SSL thumbprints required for VDDK-based backups like EMC Networker. From the vSphere Web Client, go to Administration > System Configuration > Nodes, select the vCenter instance, enable "vCenter requires verified host SSL certificates" under Settings. Once done, your backups should start working again by securely connecting through VDDK to the verified ESXi hosts.

[Hachem7]

Thank you, i just found out that vcenter SSL (the default auto signed) just expired after 10 years lately. 

It's possible to renew safely? . Can you please provide the steps to do so? 

[BarryGrowler]

I think this question is to VMware Support