3 Replies Latest reply on May 25, 2014 7:46 AM by virtualnobody

    Replace a VSA Cluster Member 5.1.3

    virtualnobody Enthusiast

      We have 2 hosts having VSA installed. One host is broken and replaced by a new one.

       

      I have followed this instructions to replace the failed host: Replace a VSA Cluster Member

       

      The "Replace appliance" assistant in "VSA Manager" tab does not show any progress. In the Recent Tasks area are some messages (attached screenshot) which shows following:

       

      1) The VSA manager updates network configuration and moves new host into cluster.

      2) OVF template is deployed to VSA-1 on new host and powered on.

      3) Next step should be "Syncing datastores" but VSA manager deletes VSA-1 and resets host network configuration.

       

      IP 192.168.10.152 is the new created VSA-1 management interface.

      IP 192.168.20.151 is the new created VSA-1 "VSA-Back End".

      Both hosts having greenfield configuration 2 NICs/2 Ports connected to 2 switches. Network configuration without VLANs.

      2x ESXi 5.1 and VSA 5.1.3

       

      Have found following entries in VSA logfiles:

       

      "VSA virtual machine () at IP address (192.168.20.151) cannot join the VSA cluster."

       

      ...but no reason given.

       

      Any help is appreciated.

       

      Thanks in advance,

      virtualnobody

       

       

      VSAManager.log:

       

      2014-05-19 18:09:52,147 525 [SVAVMConfigurator] [TaskExecutionService-4] INFO  - Successfully pinged VSA service: 192.168.10.152

      Ping Message: Ping from SVA Domain. @Mon May 19 02:00:00 CEST 2014

      Platform PING response @Mon May 19 02:00:00 CEST 2014

      2014-05-19 18:09:52,192 116 [KeystoreX509TrustManager] [TaskExecutionService-4] INFO  - Loading from trust store: C:/ProgramData/VMware/VSA Manager/security/keystore.jks

      2014-05-19 18:09:52,406 1050 [SessionService] [TaskExecutionService-4] INFO  - Posting VC Security events for list size: 1

      2014-05-19 18:09:52,456 74 [EventService] [TaskExecutionService-4] INFO  - Event SecurityCertificatesAccepted posted successfully

      2014-05-19 18:09:52,458 570 [SVAVMConfigurator] [TaskExecutionService-4] INFO  - Successfully login to SVA: 192.168.10.152 Certificate: [

      [...]

      2014-05-19 18:09:54,085 235 [Utils] [TaskExecutionService-4] INFO  - Retrieve dev.property: vm.config=true

      2014-05-19 18:09:54,086 805 [SVAVMConfigurator] [TaskExecutionService-4] INFO  - Disabled VM flag is true

      2014-05-19 18:09:54,429 1205 [PersistenceService] [Thread-2104] INFO  - Updating job: Task: REPLACE_NODE DC Ref ID: datacenter-2 Concurrency Allowed: false Type: 1 %: 100 Level: INFO

      2014-05-19 18:09:54,439 235 [Utils] [Thread-2104] INFO  - Retrieve dev.property: ha.config=true

      2014-05-19 18:09:56,524 436 [ClusterService] [Thread-2104] INFO  - Connect to VSA: 192.168.10.152

      2014-05-19 18:09:56,615 444 [ClusterService] [Thread-2104] INFO  - Successfully login to new VSA: 192.168.10.152 Certificate: [

      [...]

      2014-05-19 18:10:02,712 188 [SasMessagingService] [Thread-2104] INFO  - Calling SAS domain replaceStorageClusterMember()

      [...]

      2014-05-19 18:10:21,124 895 [ClusterService] [tomcat-http--39] INFO  - Calling SAS domain getStorageCluster()

      [...]

      2014-05-19 18:10:21,651 278 [DaoMapping] [tomcat-http--39] INFO  - VSA VM VSA-0 contains 2 storage entities

      2014-05-19 18:10:21,755 299 [DaoMapping] [tomcat-http--39] INFO  - VSA VM VSA-1 does not contain storage entities

      [...]

      2014-05-19 18:10:21,765 72 [MessagingService] [tomcat-http--39] INFO  - Using server endpoint for PSVA: https://192.168.10.151:4336/services/pseudosvaservice

      [...]

      2012014-05-19 18:10:26,412 77 [ClusterEventListener] [ActiveMQ Session Task] INFO  - Received message: com.vmware.sva.messaging.messages.BaseMessage@336ebb9e

      2014-05-19 18:10:26,414 158 [BaseEventListener] [ActiveMQ Transport: ssl:///192.168.10.154:4334] INFO  - TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:HBF-WBA-VC1-49455-1400422620842-0:0:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 195, responseRequired = true, messageId = ID:localhost-45411-1400421014565-2:0:39:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost-45411-1400421014565-2:0:39:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1400515600700, arrival = 0, brokerInTime = 1400515600700, brokerOutTime = 1400515600701, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@2adf9c73, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}

      2014-05-19 18:10:26,414 233 [EventUtils] [ActiveMQ Session Task] INFO  - Posting event to VC: ReplaceMemberTaskFailedEvent

      2014-05-19 18:10:26,418 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO  - Received Cluster message of type ReplaceMemberTaskFailedEvent

      2014-05-19 18:10:26,427 383 [ClusterEventHandlerImpl] [ActiveMQ Session Task] ERROR - VSA virtual machine (b4f3a473-35e2-4dfb-ae7e-aaef9b72bb21) at IP address (192.168.20.151) cannot join the VSA cluster.

      2014-05-19 18:10:26,428 1205 [PersistenceService] [ActiveMQ Session Task] INFO  - Updating job: Task: REPLACE_NODE DC Ref ID: datacenter-2 Concurrency Allowed: false Type: 1 %: 100 Level: INFO

      2014-05-19 18:10:26,433 409 [UserSession] [ActiveMQ Session Task] INFO  - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@336ebb9e

      2014-05-19 18:10:26,675 299 [ReplaceNodeThread] [Thread-2104] INFO  - Cleaning up configuration changes for replacing member node

        • 2. Re: Replace a VSA Cluster Member 5.1.3
          virtualnobody Enthusiast

          Ethan44,

           

          thanks for your info. I have discovered this from the logs of the running VSA-0-Node in /var/log/sva.log:

           

          "WrongSvaVersionException: VSA virtual machine () is running vSphere Storage Application version 5.1.1.0, but the required version is 5.1.3.0"

           

          Is it possible to update the running VSA-0 without having data lost?

          Sure I will do backups first, but it is on a remote site and maybe difficult to handle.

           

           

          /var/log/sva.log

           

          2014-05-19 16:06:40.618 SEVERE - com.vmware.sva.sasdomain.tasks.ReplaceMemberTask.handleZkStartupDelayTimerEvent params: [6f47b906-1f56-4321-a2b6-3aabd8f76386] - ReplaceMemberTask-6f47b906-1f56-4321-a2b6-3aabd8f76386.handleZkStartupDelayTimerEvent(): Unable to join replacement SVA to the storage cluster.

          com.vmware.sva.common.exceptions.JoinStorageClusterException: VSA virtual machine (b4f3a473-35e2-4dfb-ae7e-aaef9b72bb21) at IP address (192.168.20.151) cannot join the VSA cluster.

            at com.vmware.sva.sasdomain.tasks.ReplaceMemberTask.joinReplacementNode(ReplaceMemberTask.java:604)

            at com.vmware.sva.sasdomain.tasks.ReplaceMemberTask.handleZkStartupDelayTimerEvent(ReplaceMemberTask.java:504)

            at com.vmware.sva.sasdomain.tasks.ReplaceMemberTask.handleWaitZkStartDelay(ReplaceMemberTask.java:474)

            at com.vmware.sva.sasdomain.tasks.ReplaceMemberTask.handleEvent(ReplaceMemberTask.java:134)

            at com.vmware.sva.blc.AbstractBusinessLogicComponent.run(AbstractBusinessLogicComponent.java:534)

            at java.lang.Thread.run(Thread.java:619)

          Caused by: com.vmware.sva.common.exceptions.WrongSvaVersionException: VSA virtual machine (b4f3a473-35e2-4dfb-ae7e-aaef9b72bb21) is running vSphere Storage Application version 5.1.1.0, but the required version is 5.1.3.0.

            at com.vmware.sva.svadomain.DefaultBlc.handleJoinRequest(DefaultBlc.java:712)

            at com.vmware.sva.svadomain.DefaultBlc.handleEvent(DefaultBlc.java:1460)

            at com.vmware.sva.blc.AbstractBusinessLogicComponent.run(AbstractBusinessLogicComponent.java:534)

            at java.lang.Thread.run(Thread.java:619)

            at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(StreamRemoteCall.java:255)

            at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:233)

            at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:142)

            at com.vmware.sva.svadomain.SvaRmiService_Stub.joinStorageCluster(Unknown Source)

            at com.vmware.sva.sasdomain.tasks.ReplaceMemberTask.joinReplacementNode(ReplaceMemberTask.java:601)

            ... 5 more

          • 3. Re: Replace a VSA Cluster Member 5.1.3
            virtualnobody Enthusiast

            So I have to update the running VSA first before replacing the missing VSA-1-Node. I will try following steps:

             

            1) Make backups and verify backups

            2) Power down VSA-0-Node: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2040965

            3) Update existing VSA-0 using Update Manager hopefully without having to restore data

            4) Power up VSA-0 (VSA-1 still missing)

            5) Replace missing VSA-1-Node

             

            Edit: Updating the existing VSA did not work. Instead I have removed the VSA and HA Cluster and removed the VSA Manager and VSA Cluster Service also. Then reinstalled the new version 5.1.3 of VSA CLuster Service, VSA Manager and setup a new VSA Cluster using the wizard.