After numerous attempts we still are not able to install a 3 node VSA (brownfield) any help here is very much apreciated.
3 identical nodes, 3 raid 10 disks, setup each node with 4 NIC's devided over two switches as suggested in the documentation as in image below, we did not use VLAN tags and during install we avvoid DHCP and choose IPs in the same subnets as management network
install succesfully runs through the checks and everything seems to go fine untill half way cluster install where we get a timed-out error: Cannot create cluster: Datacenter : Datacenter 2 timed out.
Read all available articles, adjusted e.g. de time in the dev.properties to 5 minutes i.s.o. default 3, used VLANs just to try, etc.
VSA Manager log attached, any suggestion is more then welcome to help us out here !!! thanks.
Hi Burtiess.
I'm having the same issue, have you resolve yours? Kindly share the solution if any.?
Thanks
No, going for a greenfield install now, but again same issue pfff..
Losing faith in VSA all together
Thanks goodness mine was resolved after changing the timeout from 3 mins to 5. Maybe if could work for you if the increase a bit more to 10 mins maybe.?
Good luck.
Oke, last few days again a few tries, still with the same problem, must really missing a detail somewhere.
We could really use some help, starting to get frustrating
the log from VSa manager just before rollback, perhaps it helps? (please>>)
--------log VSAManager
2013-05-29 17:09:27,154 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: MemberOnlineEvent
2013-05-29 17:09:27,216 74 [EventService] [ActiveMQ Session Task] INFO - Event MemberOnlineEvent posted successfully
2013-05-29 17:09:27,216 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type MemberOnlineEvent
2013-05-29 17:09:27,216 235 [Utils] [ActiveMQ Session Task] INFO - Retrieve dev.property: com.vmware.sva.manager.security.ssl.factory=null
2013-05-29 17:09:28,230 193 [BaseEventListener] [ActiveMQ Session Task] INFO - Closing the JMS listener connection on 10.15.20.111
2013-05-29 17:09:28,277 193 [BaseEventListener] [ActiveMQ Session Task] INFO - Closing the JMS listener connection on 10.15.20.111
2013-05-29 17:09:28,667 126 [SVAManager] [tomcat-http--50] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:09:33,066 142 [BaseEventListener] [ActiveMQ Task] INFO - TransportListener: transportResumed
2013-05-29 17:09:33,081 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.111:4334] INFO - TransportListener: onCommand: WireFormatInfo { version=3, properties={CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
2013-05-29 17:09:33,081 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.111:4334] INFO - TransportListener: onCommand: BrokerInfo {commandId = 0, responseRequired = false, brokerId = ID:localhost.localdom-53265-1369840043378-0:0, brokerURL = ssl://localhost.localdom:4334, slaveBroker = false, masterBroker = false, faultTolerantConfiguration = false, networkConnection = false, duplexConnection = false, peerBrokerInfos = [], brokerName = sva1, connectionId = 0, brokerUploadUrl = null, networkProperties = null}
2013-05-29 17:09:33,097 210 [BaseEventListener] [ActiveMQ Session Task] INFO - Listening to topic: cluster.topic at URL failover:(ssl://10.15.20.111:4334?trace=true&daemon=true&keepAlive=true&soTimeout=60000)
2013-05-29 17:09:33,097 67 [MessageListenerService] [ActiveMQ Session Task] INFO - Initialized Cluster Event listener with 10.15.20.111
2013-05-29 17:09:33,097 76 [MessageListenerService] [ActiveMQ Session Task] INFO - Added default handler with listener @ 10.15.20.111
2013-05-29 17:09:33,097 210 [BaseEventListener] [ActiveMQ Session Task] INFO - Listening to topic: node.topic at URL failover:(ssl://10.15.20.111:4334?trace=true&daemon=true&keepAlive=true&soTimeout=60000)
2013-05-29 17:09:33,097 138 [MessageListenerService] [ActiveMQ Session Task] INFO - Initialized Node Event listener with 10.15.20.111
2013-05-29 17:09:33,097 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@692dbf7c
2013-05-29 17:09:33,097 77 [ClusterEventListener] [ActiveMQ Session Task] INFO - Received message: com.vmware.sva.messaging.messages.BaseMessage@33c4ea91
2013-05-29 17:09:33,097 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: CreateStorageClusterTaskProgressEvent
2013-05-29 17:09:33,112 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type CreateStorageClusterTaskProgressEvent
2013-05-29 17:09:33,112 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@33c4ea91
2013-05-29 17:09:38,447 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.105:4334] INFO - TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:vcenter-server-50502-1369818566332-0:12:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 70, responseRequired = true, messageId = ID:localhost.localdom-58949-1369840039786-2:0:14:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-58949-1369840039786-2:0:14:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1369840177814, arrival = 0, brokerInTime = 1369840177815, brokerOutTime = 1369840177815, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@272872a5, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
2013-05-29 17:09:38,447 77 [ClusterEventListener] [ActiveMQ Session Task] INFO - Received message: com.vmware.sva.messaging.messages.BaseMessage@3221e751
2013-05-29 17:09:38,556 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: MemberOnlineEvent
2013-05-29 17:09:38,588 74 [EventService] [ActiveMQ Session Task] INFO - Event MemberOnlineEvent posted successfully
2013-05-29 17:09:38,588 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type MemberOnlineEvent
2013-05-29 17:09:38,588 235 [Utils] [ActiveMQ Session Task] INFO - Retrieve dev.property: com.vmware.sva.manager.security.ssl.factory=null
2013-05-29 17:09:39,336 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.105:4334] INFO - TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:vcenter-server-50502-1369818566332-0:12:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 75, responseRequired = true, messageId = ID:localhost.localdom-58949-1369840039786-2:0:15:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-58949-1369840039786-2:0:15:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1369840178697, arrival = 0, brokerInTime = 1369840178697, brokerOutTime = 1369840178697, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@133ff5a9, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
2013-05-29 17:09:39,617 193 [BaseEventListener] [ActiveMQ Session Task] INFO - Closing the JMS listener connection on 10.15.20.108
2013-05-29 17:09:39,617 193 [BaseEventListener] [ActiveMQ Session Task] INFO - Closing the JMS listener connection on 10.15.20.108
2013-05-29 17:09:44,406 142 [BaseEventListener] [ActiveMQ Task] INFO - TransportListener: transportResumed
2013-05-29 17:09:44,421 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.108:4334] INFO - TransportListener: onCommand: WireFormatInfo { version=3, properties={CacheSize=1024, CacheEnabled=true, SizePrefixDisabled=false, MaxInactivityDurationInitalDelay=10000, TcpNoDelayEnabled=true, MaxInactivityDuration=30000, TightEncodingEnabled=true, StackTraceEnabled=true}, magic=[A,c,t,i,v,e,M,Q]}
2013-05-29 17:09:44,421 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.108:4334] INFO - TransportListener: onCommand: BrokerInfo {commandId = 0, responseRequired = false, brokerId = ID:localhost.localdom-55572-1369840041237-0:0, brokerURL = ssl://localhost.localdom:4334, slaveBroker = false, masterBroker = false, faultTolerantConfiguration = false, networkConnection = false, duplexConnection = false, peerBrokerInfos = [], brokerName = sva1, connectionId = 0, brokerUploadUrl = null, networkProperties = null}
2013-05-29 17:09:44,468 210 [BaseEventListener] [ActiveMQ Session Task] INFO - Listening to topic: cluster.topic at URL failover:(ssl://10.15.20.108:4334?trace=true&daemon=true&keepAlive=true&soTimeout=60000)
2013-05-29 17:09:44,468 67 [MessageListenerService] [ActiveMQ Session Task] INFO - Initialized Cluster Event listener with 10.15.20.108
2013-05-29 17:09:44,468 76 [MessageListenerService] [ActiveMQ Session Task] INFO - Added default handler with listener @ 10.15.20.108
2013-05-29 17:09:44,468 210 [BaseEventListener] [ActiveMQ Session Task] INFO - Listening to topic: node.topic at URL failover:(ssl://10.15.20.108:4334?trace=true&daemon=true&keepAlive=true&soTimeout=60000)
2013-05-29 17:09:44,468 138 [MessageListenerService] [ActiveMQ Session Task] INFO - Initialized Node Event listener with 10.15.20.108
2013-05-29 17:09:44,468 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@3221e751
2013-05-29 17:09:44,468 77 [ClusterEventListener] [ActiveMQ Session Task] INFO - Received message: com.vmware.sva.messaging.messages.BaseMessage@44480328
2013-05-29 17:09:44,468 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: CreateStorageClusterTaskProgressEvent
2013-05-29 17:09:44,468 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type CreateStorageClusterTaskProgressEvent
2013-05-29 17:09:44,484 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@44480328
2013-05-29 17:09:58,741 126 [SVAManager] [tomcat-http--4] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:10:17,631 1479 [ClusterService] [SanityService-1] INFO - No Hosts have been added/removed since: 13-05-29 16:45:17
2013-05-29 17:10:28,737 126 [SVAManager] [tomcat-http--4] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:10:58,717 126 [SVAManager] [tomcat-http--4] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:11:28,713 126 [SVAManager] [tomcat-http--4] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:11:58,709 126 [SVAManager] [tomcat-http--50] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:12:28,736 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:12:58,717 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:13:28,716 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:13:58,714 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:14:27,761 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.105:4334] INFO - TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:vcenter-server-50502-1369818566332-0:12:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 80, responseRequired = true, messageId = ID:localhost.localdom-58949-1369840039786-2:0:16:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-58949-1369840039786-2:0:16:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1369840467167, arrival = 0, brokerInTime = 1369840467167, brokerOutTime = 1369840467167, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@5f00a3fb, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
2013-05-29 17:14:27,777 77 [ClusterEventListener] [ActiveMQ Session Task] INFO - Received message: com.vmware.sva.messaging.messages.BaseMessage@230de477
2013-05-29 17:14:27,777 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.105:4334] INFO - TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:vcenter-server-50502-1369818566332-0:12:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 85, responseRequired = true, messageId = ID:localhost.localdom-58949-1369840039786-2:0:17:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-58949-1369840039786-2:0:17:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1369840467202, arrival = 0, brokerInTime = 1369840467202, brokerOutTime = 1369840467202, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@17d4556b, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
2013-05-29 17:14:27,824 158 [BaseEventListener] [ActiveMQ Transport: ssl:///10.15.20.105:4334] INFO - TransportListener: onCommand: MessageDispatch {commandId = 0, responseRequired = false, consumerId = ID:vcenter-server-50502-1369818566332-0:12:1:1, destination = topic://cluster.topic, message = ActiveMQObjectMessage {commandId = 90, responseRequired = true, messageId = ID:localhost.localdom-58949-1369840039786-2:0:18:1:1, originalDestination = null, originalTransactionId = null, producerId = ID:localhost.localdom-58949-1369840039786-2:0:18:1, destination = topic://cluster.topic, transactionId = null, expiration = 0, timestamp = 1369840467246, arrival = 0, brokerInTime = 1369840467246, brokerOutTime = 1369840467246, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@24d31460, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = true, readOnlyBody = true, droppable = false}, redeliveryCounter = 0}
2013-05-29 17:14:27,839 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: StorageEntityCreatedEvent
2013-05-29 17:14:27,870 74 [EventService] [ActiveMQ Session Task] INFO - Event StorageEntityCreatedEvent posted successfully
2013-05-29 17:14:27,870 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type StorageEntityCreatedEvent
2013-05-29 17:14:27,870 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@230de477
2013-05-29 17:14:27,870 77 [ClusterEventListener] [ActiveMQ Session Task] INFO - Received message: com.vmware.sva.messaging.messages.BaseMessage@691d04b
2013-05-29 17:14:27,964 1693 [VCUtils] [ActiveMQ Session Task] ERROR - Host does not contain a matched NFS datastore.
2013-05-29 17:14:27,964 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: StorageEntityOnlineEvent
2013-05-29 17:14:27,980 74 [EventService] [ActiveMQ Session Task] INFO - Event StorageEntityOnlineEvent posted successfully
2013-05-29 17:14:27,995 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type StorageEntityOnlineEvent
2013-05-29 17:14:27,995 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@691d04b
2013-05-29 17:14:27,995 77 [ClusterEventListener] [ActiveMQ Session Task] INFO - Received message: com.vmware.sva.messaging.messages.BaseMessage@14f4cda6
2013-05-29 17:14:27,995 233 [EventUtils] [ActiveMQ Session Task] INFO - Posting event to VC: CreateStorageClusterTaskProgressEvent
2013-05-29 17:14:27,995 111 [ClusterEventHandlerImpl] [ActiveMQ Session Task] INFO - Received Cluster message of type CreateStorageClusterTaskProgressEvent
2013-05-29 17:14:27,995 409 [UserSession] [ActiveMQ Session Task] INFO - Sending event to UI: com.vmware.sva.messaging.messages.BaseMessage@14f4cda6
2013-05-29 17:14:28,760 126 [SVAManager] [tomcat-http--4] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:14:58,711 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:15:17,696 1479 [ClusterService] [SanityService-1] INFO - No Hosts have been added/removed since: 13-05-29 16:45:17
2013-05-29 17:15:28,710 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:15:58,724 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:16:28,722 126 [SVAManager] [tomcat-http--21] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:16:58,752 126 [SVAManager] [tomcat-http--26] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:17:28,719 126 [SVAManager] [tomcat-http--26] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:17:58,718 126 [SVAManager] [tomcat-http--26] INFO - VSA Manager dummyPing invoked.
2013-05-29 17:18:21,650 106 [VSAEventForwarder] [pool-60-thread-1] INFO - Removed listener CreateStorageCluster-datacenter-2 for class com.vmware.sva.messaging.ws.model.mgmtevents.CreateStorageClusterTaskCompletedEvent
2013-05-29 17:18:21,650 106 [VSAEventForwarder] [pool-60-thread-1] INFO - Removed listener CreateStorageCluster-datacenter-2 for class com.vmware.sva.messaging.ws.model.mgmtevents.CreateStorageClusterTaskFailedEvent
2013-05-29 17:18:21,650 89 [DefaultTaskRunner] [pool-60-thread-1] INFO - Notifying listeners of task exception. TaskID:CreateCluster
2013-05-29 17:18:21,650 80 [DefaultJobRunner] [pool-60-thread-1] ERROR - Exception caught. Starting rollback for job if rollback support . JobName:ClusterCreation-datacenter-2
com.vmware.sva.manager.jobservices.JobErrorException: Creating cluster for datacenter: datacenter-2 timed out.
at com.vmware.sva.manager.jobservices.tasks.TaskCreateCluster.execute(TaskCreateCluster.java:115)
at com.vmware.sva.manager.jobservices.impl.DefaultTaskRunner.handleTaskStart(DefaultTaskRunner.java:70)
at com.vmware.sva.manager.jobservices.impl.DefaultTaskRunner.call(DefaultTaskRunner.java:33)
at com.vmware.sva.manager.jobservices.impl.SyncTaskGroup.execute(SyncTaskGroup.java:46)
at com.vmware.sva.manager.jobservices.impl.DefaultJobRunner.run(DefaultJobRunner.java:60)
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source)
at java.util.concurrent.FutureTask.run(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2013-05-29 17:18:21,650 109 [DefaultJobRunner] [pool-60-thread-1] INFO - Rollback started for job. JobName:ClusterCreation-datacenter-2