Standalone Converter 6.3 converter.fault.SSLVerificationFault

[BTesdall71]

I am trying to use the new Standalone Converter 6.3 to V2V some Hyper-V VM's to a new VMWare install at a remote site and getting the error in the subject.

Here are the details:

At the remote site, I have a HyperV host which has has the Local Security Policy set to disable Admin Approval Mode.  I also have a brand-new vSphere 7.0.3 2-node vSAN cluster that I want to migrate the Hyper-V VM's to.  In my main datacenter, I have our vCenter 7 instance, and on a VM I have installed Standalone Converter as a local installation.

I am able to connect to my source host, select the turned-off VM that I want to migrate, connect to my vCenter, select the remote site datacenter, select the host I want to migrate the VM to, select my migration options (thin disks, CPU's, network card type, power on the VM when done, install VMWare Tools), and start the conversion job.  Almost immediately when the converter tries to clone the disk of the VM, the process fails at 1% with the error:  "FAILED: An error occurred during the conversion: 'converter.fault.SSLVerificationFault'

I located a KB article and some other articles on the internet on how to disable SSL in Standalone Converter by modifying the converter-worker.xml file to disable SSL.  I made this change but the error persists.  I've modified this file and the converter-agent.xml and converter-server.xml files to eliminate all traces of SSL from the XML and have restarted all of the Converter services.  The issue persists.

I initially ran into this problem with Standalone Converter 6.2 and when we were using vCenter and vSphere 6.7, but was unable to solve then, although I was able to work around the problem.

Can someone give me some sort of clue why Converter is continuing to use SSL when I have explicitly reconfigured it not to?  What is the source of this issue?  Our vCenter is using a self-signed SSL cert, is this the source of the problem?  Do we need to put a cert on vCenter with our internal PKI cert authority?

I appreciate some help and guidance on this!

[ivivanov]

Hi,

Thanks for posting to the community. It is not obvious where exactly the SSL fault is happening. Converter has multiple components (Client/Server/Agent) possibly at different machines, it also communicates with vCenter server and ESX server where the source/destination VM lives. It is not clear which of all connections is experincing issues. It also includes management traffic and data traffic. I am not sure what changes you have done to the XML files, but you cannot turn off the SSL communication on the management connections, just on the data connection for copying actual data (the reason for enabling turning it off is to mitigate some performance issues if SSL-encrypted traffic is much slower and it is clear that the network is secure).

Given that you are converting a Hyper-V VM and you have got to 1% into the progress indicates the error most likely is in the Converter Agent log file which is on the Hyper-V server. Alternatively you can try to export a log bundle from the UI by right-clicking on the failed task and it should contain all related log files for further investigation.

[BTesdall71]

Regarding the disabling of SSL, I first followed the instructions in this VMWare KB article to try to disable SSL:  https://kb.vmware.com/s/article/2020517

When that didn't work, I looked at all 3 of the configuration XML files (converter-agent.xml, converter-server.xml, and converter-worker.xml), found a place in the configuration files where there's a block marked SSL and a comment above the block saying "Remove the following node to disable SSL" and removed the node.

Neither of these seem to have had any effect.

I've put together a bundle of logs from one of the failed attempts.  Hopefully there is some sort of indicator in there.  I can look for the log files on the Hyper-V host and attach those if this log bundle doesn't provide any conclusive information about where the fault is.

[BTesdall71]

So I also checked for the log files on the Hyper-V host that I'm converting from, and found this in the logs (host names redacted):

2022-11-08T10:41:41.564-06:00 error vmware-converter-agent[05452] [Originator@6876 sub=HttpConnectionPool-000000] [ConnectComplete] Connect failed to <cs p:0a265010, TCP:xxx.xxx.xxx:443>; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters:
--> PeerThumbprint: DF:65:E8:9E:F9:95:C8:81:A7:2A:BC:66:63:F5:EB:19:75:44:60:54
--> ExpectedThumbprint:
--> ExpectedPeerName: xxx.xxx.xxx
--> The remote host certificate has these problems:
-->
--> * The host certificate chain is incomplete.
-->
--> * unable to get local issuer certificate)
2022-11-08T10:41:41.564-06:00 warning vmware-converter-agent[08940] [Originator@6876 sub=task-1] CreateVimConnection has SSL certificate error while trying to connect, Error: SSL Exception: Verification parameters:
--> PeerThumbprint: DF:65:E8:9E:F9:95:C8:81:A7:2A:BC:66:63:F5:EB:19:75:44:60:54
--> ExpectedThumbprint:
--> ExpectedPeerName: xxx.xxx.xxx
--> The remote host certificate has these problems:
-->
--> * The host certificate chain is incomplete.
-->
--> * unable to get local issuer certificate
-->
2022-11-08T10:41:41.564-06:00 error vmware-converter-agent[08940] [Originator@6876 sub=task-1] hosted to managed disk clone failed: converter.fault.SSLVerificationFault

Does this mean that the self-signed cert on the vCenter server is the source of the issue?  This was something I was suspecting, but hoping for some confirmation.  We have an Enterprise PKI that we could issue a cert from, if that will correct the problem.

[ivivanov]

Hi,

I believe your analysis is correct and this is indeed the cause for the failure. I have double-checked the log bundle and found the corresponding location for the error in the source code.

There is however something I do not understand. When setting up the conversion job in Converter Client, at the destination select step you need to connect to the vCenter sever and enter username and password. At this point Converter Client should display a prompt stating the server certificate is not trusted, display the thumbprint and ask for confirmation whether to proceed with the connection or not. If you select to ignore the error then the certificate thumbprint is saved in the job spec and then passed to the Converter Agent. The agent in turn connects to vCenter server and checks whether the saved certificate from the UI matches the actual server certificate. If this is the case it proceeds with the connection. In your case obviously it did not happen as confirmed by the error you have posted:

2022-11-08T10:41:41.564-06:00 warning vmware-converter-agent[08940] [Originator@6876 sub=task-1] CreateVimConnection has SSL certificate error while trying to connect, Error: SSL Exception: Verification parameters:
--> PeerThumbprint: DF:65:E8:9E:F9:95:C8:81:A7:2A:BC:66:63:F5:EB:19:75:44:60:54
--> ExpectedThumbprint:   <<<<<<<<<!!!!!!!!!!!!!!!!!!!!
--> ExpectedPeerName: xxx.xxx.xxx

Indeed the ExpectedThumbprint field is empty and it does not match the actual server thumbprint, which ultimately aborts the connection.

I have also checked the GUI log, where the expected thumbprint should be populated by the client and indeed we can see the following:

-->          vimConnect = (converter.VimConnectionSpec) {
-->             hostname = "vcenter.server.name", 
-->             port = <unset>, 
-->             credentials = (converter.VimConnectionSpec.LoginVimCredentials) {
-->                username = "<domain>\<user>", 
-->                password = (not shown)
-->             }, 
-->             verifyPeer = <unset>, 
-->             sslThumbprint = <unset>
-->          },

We can see the sslThumbprint property is unset in the client and that is the reason for passing an empty value to the agent. For a reference if we look at another conversion in the GUI log we see something different:

-->          vimConnect = (converter.VimConnectionSpec) {
-->             hostname = "10.x.y.z", 
-->             port = <unset>, 
-->             credentials = (converter.VimConnectionSpec.LoginVimCredentials) {
-->                username = "<user>", 
-->                password = (not shown)
-->             }, 
-->             verifyPeer = <unset>, 
-->             sslThumbprint = "2F:86:39:75:FB:8D:9F:03:45:1E:84:0A:9F:D3:08:50:FE:53:DE:8C"
-->          }, 

In this case the thumbprint is populated properly in the client and should be propagated to the agent.

I am not sure what could be the reason for this behavior, what comes to my mind is that the client machine (where Converter Client is running) trusts the vCenter certificate and the connection proceeds without an exception, which does not generate a prompt in the UI and as a result the vCenter certificate thumbprint does not get passed to the agent. On the other hand on the Hyper-V machine the vCenter certificate is not trusted, but since the thumbprint is not saved by the client (ExpectedThumbprint is empty), it cannot be confirmed and the connection to vCenter fails. I think the easiest workaround would be to configure the Hyper-V server to trust the vCenter certificate (note it should be configured for the local machine root, but not for the current user), however this is just a suggestion, I have not verified it locally since I have not reproduced the issue.

[plamen_d]

In case the thumbprint is empty because it has been previously saved by Converter, this is the location where these thumbprints are saved - [HKEY_CURRENT_USER\Software\VMware\Virtual Infrastructure Client\Preferences\UI\SSLIgnore]

If that's the issue, removing it from there could be a workaround.