Hi,
I have an issue with the migration to vcsa 6 fro 5.5
got a error message : "Error when trying Backup Authz"
When I check the appliance it crash at the step 122/400
What I can see in the console :
Overall Progress : (122/400)*100
Operation : Export Vmware Common logging Service data
Requesting data '\w+-FAILED_UPGRADE-Report_Messages' from storage '/tmp/system-data/cis_upgrade_data.sfe'
Acquiring lock /tmp/system-data/cis_upgrade_data.sfe.upgradeLock
Releasing lock /tmp/system-data/cis_upgrade_data.sfe.upgradeLock
Found results: --{}
Destination repository /tmp/cis-import-folder/system-data/cis_upgrade_data.sfe does not exist. Skip merging
2015-03-17T08:55:59:667Z: Upgrade EXPORT failed
in the vcsa0000.log I can see a strange thing, it calls ipv6 :
2015-03-17 09:48:36.451474 --net:Network 1=vim.Network:HaNetwork-d-admin
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.addr.family=ipv4
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.mode=static
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.pnid=172.16.21.210
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.addr=172.16.21.210
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.prefix=24
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.dns.servers=172.16.12.100,172.26.12.100
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.gateway=172.16.21.1
2015-03-17 09:48:36.451474 --prop:guestinfo.cis.clientlocale=fr
2015-03-17 09:48:36.465598 --targetSSLThumbprint=*
2015-03-17 09:53:41.452041 Ovf Service: [INFO] ovf Execute: VM IP- fe80::20c:29ff:fe3a:bc6d
2015-03-17 09:53:42.061323 Ovf Service: [INFO] ovf Execute: VMREF- vim.VirtualMachine:60
2015-03-17 09:53:42.061323 Progress Controller: Deployment Success
2015-03-17 09:53:42.061323 Progress Controller: Final Result: {"type":"result","statusCode":"PROCESS_STATUS","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3690","requestComponentId":"ovf","requestObjectId":"Ag5N-X38F-OXcw-1n6Q","result":"","isFinal":"false","item":["TARGET_ID","vim.VirtualMachine:60","RESULT","+ SUCCESS"],"complete":true,"vmRef":"60","ip":"fe80::20c:29ff:fe3a:bc6d"}
2015-03-17 09:53:42.061323 Progress Controller: [VMref] - VM ref: 60
2015-03-17 09:53:42.061323 Progress Controller: [ResultIP] - IP : fe80::20c:29ff:fe3a:bc6d
2015-03-17 09:53:52.086111 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3710","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}
2015-03-17 09:53:57.085338 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3712","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}
2015-03-17 09:54:02.087335 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3714","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}
2015-03-17 09:54:07.087895 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3716","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}
I have PTR and DNS registered and the source and destination are on a simple portgroup and on the same network. (and esx)
Thanks for your help !
Can you share all the export logs? It can be found under - /var/log/vmware/upgrade/
This is the entire log directory .
I really don't understand what happens here....
I thought it was a PTR problem but evrything is ok on the appliance, I can execute a Host -T IP without any problem (it's the command called by the vmware script to populate the hostname).
Thanks for your help
Seems like OutOfMemoryError -
2015-03-17T09:57:16.720Z INFO upgrade.states.component_states is:Export:
2015-03-17T09:57:16.720Z INFO upgrade.states.component_states is:Export: Mar 17, 2015 9:57:15 AM com.vmware.vim.dataservices.BackupAuthz main
2015-03-17T09:57:16.720Z INFO upgrade.states.component_states is:Export: SEVERE: Backup Authz failed
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: java.lang.OutOfMemoryError: Java heap space
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at java.util.Arrays.copyOf(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at java.lang.AbstractStringBuilder.append(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at java.lang.StringBuffer.append(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at java.io.StringWriter.write(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xml.internal.serializer.ToStream.characters(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xml.internal.serializer.ToUnknownStream.characters(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xalan.internal.xsltc.trax.StAXStream2SAX.handleCharacters(Unknown Source)
2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xalan.internal.xsltc.trax.StAXStream2SAX.bridge(Unknown Source)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xalan.internal.xsltc.trax.StAXStream2SAX.parse(Unknown Source)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(Unknown Source)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.BackupAuthz.getXML(BackupAuthz.java:252)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.BackupAuthz.backupAssignment(BackupAuthz.java:174)
2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export: at com.vmware.vim.dataservices.BackupAuthz.main(BackupAuthz.java:325)
o_O
This error looks like a bug in the migration tool.
I choose small deployment because I got 8 hosts ( 384GB RAM each) and 200 VM.
I don't think that's a Hard memory problem , perharps I should call support for that.
Thanks.
From error it looks like memory issue which could be because Inventory service export script itself.
BTW what is the vCSA configuration like memory, free disk, vCPU etc? When you ran into this issue how was the resource usage pattern.
Since the export itself failed; can you re-run the upgrade.Attach output of top command before you start upgrade and when you run into this issue.
How can i rerun the upgrade on the new vcsa ?
To re-run-
Once above steps are done you can re-run upgrade with same steps as you did earlier.
Oh !!
I didn't notice that the old vcsa go to 100% memory during the migration.
The old 5.5 vcsa is configured with 8GB RAM.
I only looked to the new vcsa performance but i forgot to monitore the old one
Ok I will try to increase it and re run migration.
no luck, still the same issue .
I increase RAM of the old vcsa and increase heap size (push a medium configuration) and no luck 😕
Thanks for trying this out. Another questions did you by chance re-started vCSA5.5 before upgrade.
You should contact GSS and meanwhile we will try to reproduce in-house.
Exact same thing for me. Frustrating.
Hi there,
I experienced similar error messages. My scenario is slightly different though, as I am testing the build in a nested environment (Workstation 11). Here are my observations:
2015-04-07 12:53:09.710365 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"NQdq-WlBc-hEzo-uEjM","requestId":"346","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}
2015-04-07 12:53:09.711365 Progress Controller: [RPMInstallProgress] - Installed oracle-instantclient11.2-odbc-11.2.0.2.0.x86_64.rpm
Starting Vmodl Based Transfer
Connecting to ESX host...
Starting Vmodl Based Transfer
Connecting to ESX host...
[2015-04-07 13:10:58] [INFO] Process created: "C:\Program Files (x86)\VMware\Client Integration Plug-in 6.0\fileTransfer.exe" "--logPath" "C:\Users\eadmin\AppData\Local\VMware\CIP\vcsaInstaller\sessions\session_0000\logs\filetransfer-6100.log" "--hostname" "192.168.213.128" "--port" "443" "--password" "VGVzdEAxMjMh" "--source" "/var/log/firstboot/rpmInstall.json" "--destination" "C:\Users\eadmin\AppData\Local\Temp\ft_vmgetfile_9497-699b-e5fb-6650.tmp" "--username" "root" "--thumbprint" "D9:A1:6C:DE:13:A8:C3:94:B3:F2:9C:6D:31:96:80:C7:03:6D:AD:D5" "--download" "--vmodl" "VMFileTransfer" "--moRef" "vim.VirtualMachine:1" "--vmUsername" "root" "--vmPassword" "VGVzdEAxMjMh" "--base64password"
[2015-04-07 13:10:58] [ERRO] There is already a process running with ID: 8088-jT99-8gEX-Xsia.
[2015-04-07 13:10:58] [ERRO] Request 696 - [fileTransfer: 8088-jT99-8gEX-Xsia].vmGetFile: Error Status PROC_FAIL_PROCESS_ALREAD_IN_MAP: Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP
[2015-04-07 13:10:58] [INFO] Request 699 - [session: session].ping: Received.
[2015-04-07 13:11:33] [WARN] The client has been inactive for 40 seconds...
[2015-04-07 13:12:03] [WARN] The client has been inactive for 70 seconds...
[2015-04-07 13:12:33] [WARN] The client has been inactive for 100 seconds...
[2015-04-07 13:12:53] [ERRO] The client went inactive for 120 seconds, shutting down client socket.
[2015-04-07 13:12:53] [INFO] Stop called on web socket: 50100
[2015-04-07 13:12:54] [INFO] Web Socket Server [50100] Terminated With no error code
[2015-04-07 13:15:05] [INFO] Process with ID: 8088-jT99-8gEX-Xsia has terminated , notifying owner
[2015-04-07 13:15:17] [ERRO] Request 533 - [fileTransfer: 8088-jT99-8gEX-Xsia].vmGetFile: Error Status FILE_NOT_FOUND: Error: No file downloaded from fileTransfer.exe
[2015-04-07 13:20:01] [INFO] Process exiting.
Let me know if anyone had luck with above/workaround. Many thanks in advance.
Same here.
Any help from VMWare?
Same here.
Looks like this issue is fixed in patch released on 16th April
please refer to KB
Seems so but there is no update for the upgrade ISO from 5.X to 6.
And also nothing about to apply the JRE fix.