VMware Cloud Community
Franck999
Enthusiast
Enthusiast

vcsa 5.5 to 6 migration error

Hi,

I have an issue with the migration to vcsa 6 fro 5.5

got a error message : "Error when trying  Backup Authz"

When I check the appliance it crash at the step 122/400

What I can see in the console :

Overall Progress : (122/400)*100

Operation : Export Vmware Common logging Service data

Requesting data '\w+-FAILED_UPGRADE-Report_Messages' from storage '/tmp/system-data/cis_upgrade_data.sfe'

Acquiring lock /tmp/system-data/cis_upgrade_data.sfe.upgradeLock

Releasing lock /tmp/system-data/cis_upgrade_data.sfe.upgradeLock

Found results: --{}

Destination repository /tmp/cis-import-folder/system-data/cis_upgrade_data.sfe does not exist. Skip merging

2015-03-17T08:55:59:667Z: Upgrade EXPORT failed

in the vcsa0000.log I can see a strange thing, it calls ipv6 :

2015-03-17 09:48:36.451474 --net:Network 1=vim.Network:HaNetwork-d-admin

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.addr.family=ipv4

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.mode=static

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.pnid=172.16.21.210

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.addr=172.16.21.210

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.prefix=24

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.dns.servers=172.16.12.100,172.26.12.100

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.appliance.net.gateway=172.16.21.1

2015-03-17 09:48:36.451474 --prop:guestinfo.cis.clientlocale=fr

2015-03-17 09:48:36.465598 --targetSSLThumbprint=*

2015-03-17 09:53:41.452041  Ovf Service: [INFO] ovf Execute: VM IP- fe80::20c:29ff:fe3a:bc6d

2015-03-17 09:53:42.061323  Ovf Service: [INFO] ovf Execute: VMREF- vim.VirtualMachine:60

2015-03-17 09:53:42.061323 Progress Controller: Deployment Success

2015-03-17 09:53:42.061323 Progress Controller: Final Result: {"type":"result","statusCode":"PROCESS_STATUS","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3690","requestComponentId":"ovf","requestObjectId":"Ag5N-X38F-OXcw-1n6Q","result":"","isFinal":"false","item":["TARGET_ID","vim.VirtualMachine:60","RESULT","+ SUCCESS"],"complete":true,"vmRef":"60","ip":"fe80::20c:29ff:fe3a:bc6d"}

2015-03-17 09:53:42.061323 Progress Controller: [VMref] - VM ref: 60

2015-03-17 09:53:42.061323 Progress Controller: [ResultIP] - IP : fe80::20c:29ff:fe3a:bc6d

2015-03-17 09:53:52.086111 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3710","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}

2015-03-17 09:53:57.085338 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3712","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}

2015-03-17 09:54:02.087335 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3714","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}

2015-03-17 09:54:07.087895 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"XLwp-HmNG-ewPO-CYpP","requestId":"3716","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}

I have PTR and DNS registered and the source and destination are on a simple portgroup and on the same network. (and esx)

Thanks for your help !

16 Replies
RAMESA
VMware Employee
VMware Employee

Can you share all the export logs? It can be found under - /var/log/vmware/upgrade/

Regards, Ramesh
Reply
0 Kudos
Franck999
Enthusiast
Enthusiast

This is the entire log directory .

I really don't understand what happens here....

I thought it was a PTR problem but evrything is ok on the appliance, I can execute a Host -T IP without any problem (it's the command called by the vmware script to populate the hostname).

Thanks for your help

Reply
0 Kudos
RAMESA
VMware Employee
VMware Employee

Seems like OutOfMemoryError - 

2015-03-17T09:57:16.720Z INFO upgrade.states.component_states is:Export:

2015-03-17T09:57:16.720Z INFO upgrade.states.component_states is:Export: Mar 17, 2015 9:57:15 AM com.vmware.vim.dataservices.BackupAuthz main

2015-03-17T09:57:16.720Z INFO upgrade.states.component_states is:Export: SEVERE: Backup Authz failed

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export: java.lang.OutOfMemoryError: Java heap space

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at java.util.Arrays.copyOf(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at java.lang.AbstractStringBuilder.ensureCapacityInternal(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at java.lang.AbstractStringBuilder.append(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at java.lang.StringBuffer.append(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at java.io.StringWriter.write(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xml.internal.serializer.ToStream.characters(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xml.internal.serializer.ToUnknownStream.characters(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xalan.internal.xsltc.trax.StAXStream2SAX.handleCharacters(Unknown Source)

2015-03-17T09:57:16.721Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xalan.internal.xsltc.trax.StAXStream2SAX.bridge(Unknown Source)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xalan.internal.xsltc.trax.StAXStream2SAX.parse(Unknown Source)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transformIdentity(Unknown Source)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.sun.org.apache.xalan.internal.xsltc.trax.TransformerImpl.transform(Unknown Source)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.vmware.vim.dataservices.BackupAuthz.getXML(BackupAuthz.java:252)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.vmware.vim.dataservices.BackupAuthz.backupAssignment(BackupAuthz.java:174)

2015-03-17T09:57:16.722Z INFO upgrade.states.component_states is:Export:     at com.vmware.vim.dataservices.BackupAuthz.main(BackupAuthz.java:325)

Regards, Ramesh
Reply
0 Kudos
Franck999
Enthusiast
Enthusiast

o_O

This error looks like a bug in the migration tool.

I choose small deployment because I got 8 hosts ( 384GB RAM each)  and 200 VM.

I don't think that's a Hard memory problem , perharps I should call support for that.

Thanks.

Reply
0 Kudos
RAMESA
VMware Employee
VMware Employee

From error it looks like memory issue which could be because Inventory service export script itself.

BTW what is the vCSA configuration like memory, free disk, vCPU etc? When you ran into this issue how was the resource usage pattern.

Since the export itself failed; can you re-run the upgrade.Attach output of top command before you start upgrade and when you run into this issue.

Regards, Ramesh
Reply
0 Kudos
Franck999
Enthusiast
Enthusiast

How can i rerun the upgrade on the new vcsa ?

Reply
0 Kudos
RAMESA
VMware Employee
VMware Employee

To re-run-

  1. Shutdown the new 6.0 vCSA.
  2. PowerON if PoweredOFF the old 5.5 vCSA.
  3. Make sure all services come up and VC is functional.

Once above steps are done you can re-run upgrade with same steps as you did earlier.

Regards, Ramesh
Reply
0 Kudos
Franck999
Enthusiast
Enthusiast

Oh !!

I didn't notice that the old vcsa go to 100% memory during the migration.

The old 5.5 vcsa is configured with 8GB RAM.

I only looked  to the new vcsa performance but i forgot to monitore the old one Smiley Happy

Ok I will try to increase it and re run migration.

Reply
0 Kudos
Franck999
Enthusiast
Enthusiast

no luck, still the same issue .

I increase RAM of the old vcsa and increase heap size (push a medium configuration) and no luck 😕

RAMESA
VMware Employee
VMware Employee

Thanks for trying this out. Another questions did you by chance re-started vCSA5.5 before upgrade.

You should contact GSS and meanwhile we will try to reproduce in-house.

Regards, Ramesh
Reply
0 Kudos
ppassin
Contributor
Contributor

Exact same thing for me. Frustrating.

alok858
Contributor
Contributor

Hi there,

I experienced similar error messages. My scenario is slightly different though, as I am testing the build in a nested environment (Workstation 11). Here are my observations:

  • Build 6.0.0-2562643
  • Installing built-in postgre db
  • vcsa_0000.log didn't record anything further, as soon as it experienced:

2015-04-07 12:53:09.710365 CIP Service: [VCSA ERROR] Error fetching File form VM:{"type":"error","statusCode":"PROC_FAIL_PROCESS_ALREAD_IN_MAP","message":"Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP","isFinal":"true","sessionId":"NQdq-WlBc-hEzo-uEjM","requestId":"346","requestComponentId":"fileTransfer","requestObjectId":"8088-jT99-8gEX-Xsia"}

2015-04-07 12:53:09.711365 Progress Controller: [RPMInstallProgress] - Installed oracle-instantclient11.2-odbc-11.2.0.2.0.x86_64.rpm

  • filetransfer-6100.log records this activity in loop since then:

Starting Vmodl Based Transfer

Connecting to ESX host...

Starting Vmodl Based Transfer

Connecting to ESX host...

  • output from csd:

[2015-04-07 13:10:58] [INFO] Process created: "C:\Program Files (x86)\VMware\Client Integration Plug-in 6.0\fileTransfer.exe" "--logPath" "C:\Users\eadmin\AppData\Local\VMware\CIP\vcsaInstaller\sessions\session_0000\logs\filetransfer-6100.log" "--hostname" "192.168.213.128" "--port" "443" "--password" "VGVzdEAxMjMh" "--source" "/var/log/firstboot/rpmInstall.json" "--destination" "C:\Users\eadmin\AppData\Local\Temp\ft_vmgetfile_9497-699b-e5fb-6650.tmp" "--username" "root" "--thumbprint" "D9:A1:6C:DE:13:A8:C3:94:B3:F2:9C:6D:31:96:80:C7:03:6D:AD:D5" "--download" "--vmodl" "VMFileTransfer" "--moRef" "vim.VirtualMachine:1" "--vmUsername" "root" "--vmPassword" "VGVzdEAxMjMh" "--base64password"

[2015-04-07 13:10:58] [ERRO] There is already a process running with ID: 8088-jT99-8gEX-Xsia.

[2015-04-07 13:10:58] [ERRO] Request 696 - [fileTransfer: 8088-jT99-8gEX-Xsia].vmGetFile: Error Status PROC_FAIL_PROCESS_ALREAD_IN_MAP: Failed to start file transfer; status: PROC_FAIL_PROCESS_ALREAD_IN_MAP

[2015-04-07 13:10:58] [INFO] Request 699 - [session: session].ping: Received.

[2015-04-07 13:11:33] [WARN] The client has been inactive for 40 seconds...

[2015-04-07 13:12:03] [WARN] The client has been inactive for 70 seconds...

[2015-04-07 13:12:33] [WARN] The client has been inactive for 100 seconds...

[2015-04-07 13:12:53] [ERRO] The client went inactive for 120 seconds, shutting down client socket.

[2015-04-07 13:12:53] [INFO] Stop called on web socket: 50100

[2015-04-07 13:12:54] [INFO] Web Socket Server [50100] Terminated With no error code

[2015-04-07 13:15:05] [INFO] Process with ID: 8088-jT99-8gEX-Xsia has terminated , notifying owner

[2015-04-07 13:15:17] [ERRO] Request 533 - [fileTransfer: 8088-jT99-8gEX-Xsia].vmGetFile: Error Status FILE_NOT_FOUND: Error: No file downloaded from fileTransfer.exe

[2015-04-07 13:20:01] [INFO] Process exiting.

Let me know if anyone had luck with above/workaround. Many thanks in advance.

Reply
0 Kudos
Buschfunker
Contributor
Contributor

Same here.

Any help from VMWare?

Reply
0 Kudos
Tentacle2011101
Contributor
Contributor

Same here.

Reply
0 Kudos
admin
Immortal
Immortal

Looks like this issue is fixed in patch released on 16th April

please refer to KB

http://kb.vmware.com/kb/2111640

Reply
0 Kudos
Buschfunker
Contributor
Contributor

Seems so but there is no update for the upgrade ISO from 5.X to 6.

And also nothing about to apply the JRE fix.

Reply
0 Kudos