VMware Networking Community
nagesh_u
Enthusiast
Enthusiast

NSX controller deploy error vcoperationfailedexception: core-services:1500:operation failed on vc for more

Hi Team,

i am trying to deploy NSX controller but i am getting error 70% , its says as vcoperationfailedexception: core-services:1500:operation failed on vc for more.

my NSX version is:6.4.5

deployed in Vmware workstation 15.0.2 build-10952284

my DVS port configured with 1600MTU

NSX and Vcenter sitting in same VLAN so create the IP POOL with same VLAN, All device is sitting same VLAN.

NTP is working fine

please find the log

15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271942

2019-11-24 02:13:50.816 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

2019-11-24 02:13:50.862 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

2019-11-24 02:13:50.863 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

2019-11-24 02:13:50.864 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

2019-11-24 02:13:50.883 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 48

2019-11-24 02:13:50.884 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

2019-11-24 02:13:50.884 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48257 num vc objs:48

2019-11-24 02:13:50.894 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271943

2019-11-24 02:13:52.444 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:13:52.446 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271944

2019-11-24 02:13:52.510 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

2019-11-24 02:13:52.528 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

2019-11-24 02:13:52.529 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

2019-11-24 02:13:52.530 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

2019-11-24 02:13:52.532 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 18

2019-11-24 02:13:52.533 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

2019-11-24 02:13:52.533 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48258 num vc objs:48

2019-11-24 02:13:52.539 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271945

2019-11-24 02:13:54.487 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:13:54.489 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271946

2019-11-24 02:13:56.546 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:13:56.548 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271947

2019-11-24 02:13:57.452 UTC  INFO taskScheduler-6 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58337 to EXECUTING

2019-11-24 02:13:57.482 UTC  INFO taskScheduler-6 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module policy and total number o                               f buckets 1

2019-11-24 02:13:57.482 UTC  INFO taskScheduler-6 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task com.vmware.vshield.blueprint.policy.task.policyStatus                               NotificationTask [id:task-100] is added to the SchedulerQueue

2019-11-24 02:13:57.483 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-100 and running executor threads 2

2019-11-24 02:13:57.563 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271948

2019-11-24 02:13:57.754 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271949

2019-11-24 02:13:57.814 UTC  INFO TaskFrameworkExecutor-20 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58337 to COMPLETED

2019-11-24 02:13:57.822 UTC  INFO TaskFrameworkExecutor-20 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-96

2019-11-24 02:13:57.855 UTC  INFO TaskFrameworkExecutor-20 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool acti                               ve count 1

2019-11-24 02:13:58.615 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:13:58.616 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271950

2019-11-24 02:14:00.748 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:00.750 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271951

2019-11-24 02:14:02.790 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:02.792 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271952

2019-11-24 02:14:04.653 UTC  INFO EamConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on property                               Collector, operationID=opId-c8639-271953

2019-11-24 02:14:04.934 UTC  INFO EamConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionManager,                                operationID=opId-c8639-271954

2019-11-24 02:14:04.987 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:04.988 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271955

2019-11-24 02:14:07.122 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:07.124 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271956

2019-11-24 02:14:09.168 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:09.169 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271957

2019-11-24 02:14:11.267 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:11.269 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271958

2019-11-24 02:14:12.821 UTC  INFO messagingTaskExecutor-7 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-63

2019-11-24 02:14:12.964 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271959

2019-11-24 02:14:13.022 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271960

2019-11-24 02:14:13.301 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:13.302 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271961

2019-11-24 02:14:13.356 UTC  INFO messagingTaskExecutor-6 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-51

2019-11-24 02:14:13.381 UTC  INFO messagingTaskExecutor-8 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-41

2019-11-24 02:14:15.340 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:15.342 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271962

2019-11-24 02:14:16.169 UTC  INFO taskScheduler-19 AppVisibilityCollectionThread:43 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] App Visibility Collection task invoked, collectionInProgr                               ess - false

2019-11-24 02:14:17.382 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:17.383 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271963

2019-11-24 02:14:19.438 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:19.439 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271964

2019-11-24 02:14:21.557 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:21.558 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271965

2019-11-24 02:14:23.678 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:23.679 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271966

2019-11-24 02:14:24.125 UTC  INFO edgeVseMonitoringThread EdgeVseHealthMonitoringThread:298 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Finished Health check for 0 edge vms in 0 millise                               c

2019-11-24 02:14:25.789 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:25.792 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271967

2019-11-24 02:14:26.073 UTC ERROR taskScheduler-15 ResourcePoolVcOperationsImpl:443 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed to install OVF.

java.util.concurrent.TimeoutException: null

        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) [vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [s                               pring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE                               ]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-a                               spects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) [vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [s                               pring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE                               ]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-a                               spects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) [vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [s                               pring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE                               ]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-a                               spects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

2019-11-24 02:14:26.080 UTC ERROR taskScheduler-15 AbstractSvmMgmtTask:206 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed, jobId 'jobdata-4521' during svm mgmt task. Attempting ret                               ry: '1'

com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

Caused by: java.util.concurrent.TimeoutException

        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) ~[vsphere-1.0.jar:?]

        ... 36 more

2019-11-24 02:14:26.085 UTC ERROR taskScheduler-15 Worker:229 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] BaseException thrown while executing task instance taskinstance-59201

com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

Caused by: java.util.concurrent.TimeoutException

        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) ~[vsphere-1.0.jar:?]

        ... 36 more

2019-11-24 02:14:26.162 UTC  INFO taskScheduler-15 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58306 to FAILED

2019-11-24 02:14:26.214 UTC  INFO TaskFrameworkExecutor-10 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool acti                               ve count 0

2019-11-24 02:14:26.854 UTC  INFO DeploymentMonitor DeploymentMonitor:131 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Purge work for finished job jobdata-4521 in status: FAILED

2019-11-24 02:14:26.855 UTC  INFO DeploymentMonitor DeploymentMonitor:135 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Job jobdata-4521 for controller controller-5 failed!

2019-11-24 02:14:26.887 UTC  INFO DeploymentMonitor DeploymentMonitor:175 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] about to remove controller from database: controller-5 VM id null i                               p 10.10.10.136 uuid null version 12633898 VSM id 564D66D5-CABD-D4B2-9198-08B2277F333C

2019-11-24 02:14:26.888 UTC  INFO DeploymentMonitor RelationshipManagerImpl:840 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Recursively removing domain object controller-5 updateParent                                false

2019-11-24 02:14:27.020 UTC  INFO DeploymentMonitor AuditingServiceImpl:174 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [AuditLog] UserName:'', ModuleName:'VdnNvpController', Operation:                               'DELETE', Resource:'nalabcnt01', Time:'Sun Nov 24 02:14:26.918 UTC 2019', Status:'SUCCESS', Universal Object:'false'

2019-11-24 02:14:27.034 UTC  INFO DeploymentMonitor IPAMServiceImpl:420 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Released IP address 10.10.10.136 from pool ipaddresspool-1, status =                                true

2019-11-24 02:14:27.035 UTC  INFO DeploymentMonitor DeploymentMonitor:182 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] release ip to ip pool: ipaddresspool-1 10.10.10.136

2019-11-24 02:14:27.043 UTC  INFO DCNPool-10 VirtualWireInFirewallRuleNotificationHandler:59 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Recieved VDN DELETE notification for context con                               troller-5:Controller

2019-11-24 02:14:27.512 UTC  INFO taskScheduler-10 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58338 to EXECUTING

2019-11-24 02:14:27.546 UTC  INFO taskScheduler-10 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module policy and total number                                of buckets 1

2019-11-24 02:14:27.548 UTC  INFO taskScheduler-10 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task com.vmware.vshield.blueprint.policy.task.policyStatu                               sNotificationTask [id:task-100] is added to the SchedulerQueue

2019-11-24 02:14:27.549 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-100 and running executor threads 1

2019-11-24 02:14:27.669 UTC  INFO TaskFrameworkExecutor-9 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58338 to COMPLETED

2019-11-24 02:14:27.679 UTC  INFO TaskFrameworkExecutor-9 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-96

2019-11-24 02:14:27.701 UTC  INFO TaskFrameworkExecutor-9 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

2019-11-24 02:14:27.965 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:27.966 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271968

2019-11-24 02:14:28.122 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271969

2019-11-24 02:14:28.254 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271970

2019-11-24 02:14:30.119 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:30.120 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271971

2019-11-24 02:14:31.134 UTC  INFO taskScheduler-13 FlowCollectionThread:44 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Flow Collection task invoked, collectionInProgress - false

2019-11-24 02:14:32.210 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:32.212 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271972

2019-11-24 02:14:34.257 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:34.258 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271973

2019-11-24 02:14:36.300 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:36.302 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271974

2019-11-24 02:14:36.903 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

2019-11-24 02:14:36.946 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

2019-11-24 02:14:36.948 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

2019-11-24 02:14:36.950 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

2019-11-24 02:14:36.959 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 47

2019-11-24 02:14:36.960 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

2019-11-24 02:14:36.960 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48259 num vc objs:48

2019-11-24 02:14:36.968 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271975

2019-11-24 02:14:38.353 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:38.354 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271976

2019-11-24 02:14:40.417 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:40.419 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271977

2019-11-24 02:14:42.572 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:42.573 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271978

2019-11-24 02:14:42.850 UTC  INFO messagingTaskExecutor-1 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-63

2019-11-24 02:14:43.363 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271979

2019-11-24 02:14:43.402 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271980

2019-11-24 02:14:43.404 UTC  INFO messagingTaskExecutor-8 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-51

2019-11-24 02:14:43.514 UTC  INFO messagingTaskExecutor-7 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-41

2019-11-24 02:14:44.632 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:44.633 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271981

2019-11-24 02:14:46.672 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:46.673 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271982

2019-11-24 02:14:48.913 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:48.915 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271983

2019-11-24 02:14:50.966 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

2019-11-24 02:14:50.976 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:50.977 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271984

2019-11-24 02:14:51.082 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

2019-11-24 02:14:51.084 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

2019-11-24 02:14:51.086 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

2019-11-24 02:14:51.090 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 83

2019-11-24 02:14:51.091 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

2019-11-24 02:14:51.092 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48260 num vc objs:48

2019-11-24 02:14:51.102 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271985

2019-11-24 02:14:52.734 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

2019-11-24 02:14:52.768 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

2019-11-24 02:14:52.770 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

2019-11-24 02:14:52.771 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

2019-11-24 02:14:52.779 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 30

2019-11-24 02:14:52.780 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

2019-11-24 02:14:52.780 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48261 num vc objs:48

2019-11-24 02:14:52.797 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271986

2019-11-24 02:14:53.051 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:53.052 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271987

2019-11-24 02:14:55.113 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:55.114 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271988

2019-11-24 02:14:57.383 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:57.384 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271989

2019-11-24 02:14:58.277 UTC  INFO taskScheduler-8 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58339 to EXECUTING

2019-11-24 02:14:58.330 UTC  INFO taskScheduler-8 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module policy and total number o                               f buckets 1

2019-11-24 02:14:58.331 UTC  INFO taskScheduler-8 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task com.vmware.vshield.blueprint.policy.task.policyStatus                               NotificationTask [id:task-100] is added to the SchedulerQueue

2019-11-24 02:14:58.332 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-100 and running executor threads 1

2019-11-24 02:14:58.472 UTC  INFO TaskFrameworkExecutor-6 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58339 to COMPLETED

2019-11-24 02:14:58.480 UTC  INFO TaskFrameworkExecutor-6 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-96

2019-11-24 02:14:58.484 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271990

2019-11-24 02:14:58.539 UTC  INFO TaskFrameworkExecutor-6 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

2019-11-24 02:14:58.627 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271991

2019-11-24 02:14:59.878 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:14:59.879 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271992

2019-11-24 02:15:00.100 UTC  INFO taskScheduler-9 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58340 to EXECUTING

2019-11-24 02:15:00.142 UTC  INFO taskScheduler-9 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module fabric_state_updater and                                total number of buckets 1

2019-11-24 02:15:00.143 UTC  INFO taskScheduler-9 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task core.services.fabric.stateUpdaterTaskName [id:task-98                               ] is added to the SchedulerQueue

2019-11-24 02:15:00.143 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-98 and running executor threads 1

2019-11-24 02:15:00.331 UTC  INFO TaskFrameworkExecutor-3 TaskServiceImpl:101 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:Created Job with ID jobdata-4525

2019-11-24 02:15:00.334 UTC  INFO TaskFrameworkExecutor-3 TaskServiceImpl:425 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:Scheduling Job jobdata-4525

2019-11-24 02:15:00.351 UTC  INFO TaskFrameworkExecutor-3 SchedulerService:198 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Job jobdata-4525 is scheduled and put data in to scheduleInfoT                               able

2019-11-24 02:15:00.379 UTC  INFO UnitOfWorkPool-1 SchedulerService:193 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received job creation DCN for job id jobdata-4525

2019-11-24 02:15:00.397 UTC  INFO TaskFrameworkExecutor-3 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58340 to COMPLETED

2019-11-24 02:15:00.412 UTC  INFO TaskFrameworkExecutor-3 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-94

2019-11-24 02:15:00.424 UTC  INFO TaskFrameworkExecutor-3 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

2019-11-24 02:15:00.877 UTC  INFO pool-17-thread-1 ImmediateScheduler:35 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:Schedule Now Job ID jobdata-4525

2019-11-24 02:15:00.919 UTC  INFO taskScheduler-30 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58341 to EXECUTING

2019-11-24 02:15:00.972 UTC  INFO taskScheduler-30 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module fabric and total number                                of buckets 1

2019-11-24 02:15:00.974 UTC  INFO taskScheduler-30 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task core.services.fabric.stateUpdaterPerDUTaskName;deplo                               ymentunit-1 [id:task-5458] is added to the SchedulerQueue

2019-11-24 02:15:00.975 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-5458 and running executor threads 1

2019-11-24 02:15:01.919 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:15:01.921 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271993

2019-11-24 02:15:03.382 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:415 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current eamAgentRuntimeStatus is green, and eamAgentGoalStat                               e is enabled for 902be1ac-3a0d-4268-90cc-ce9ec886a24f

2019-11-24 02:15:03.601 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:415 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current eamAgentRuntimeStatus is green, and eamAgentGoalStat                               e is enabled for 40014283-52ae-425d-a076-2195db0c50d0

2019-11-24 02:15:03.968 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:415 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current eamAgentRuntimeStatus is green, and eamAgentGoalStat                               e is enabled for 2a46d6dd-13b1-4d0a-889d-751bc233faf2

2019-11-24 02:15:04.022 UTC  INFO taskScheduler-12 RuleStatsCollector:40 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Stats Collection task invoked, collectionInProgress - false

2019-11-24 02:15:04.025 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:15:04.032 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271994

2019-11-24 02:15:04.061 UTC  INFO taskScheduler-12 RuleStatsDataService:157 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Rule stats collected for 1 rules

2019-11-24 02:15:04.142 UTC  INFO taskScheduler-12 RuleStatsDataService:134 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Send request to host for stats collection

2019-11-24 02:15:04.154 UTC  INFO taskScheduler-12 FirewallProtobufConverter:1560 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Setting client id - 49b1da11-7494-40f1-99f4-747e3f517ffe

2019-11-24 02:15:04.155 UTC  INFO taskScheduler-12 FirewallMessagingManager:844 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Sending StatsReq to 49b1da11-7494-40f1-99f4-747e3f517ffe

2019-11-24 02:15:04.159 UTC  INFO taskScheduler-12 FirewallProtobufConverter:1560 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Setting client id - a2ceab2c-4f44-4d0e-8bdc-715f368e642f

2019-11-24 02:15:04.161 UTC  INFO taskScheduler-12 FirewallMessagingManager:844 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Sending StatsReq to a2ceab2c-4f44-4d0e-8bdc-715f368e642f

2019-11-24 02:15:04.164 UTC  INFO taskScheduler-12 FirewallProtobufConverter:1560 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Setting client id - ca61ca3f-e207-4410-99b7-0368d29bc002

2019-11-24 02:15:04.165 UTC  INFO taskScheduler-12 FirewallMessagingManager:844 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Sending StatsReq to ca61ca3f-e207-4410-99b7-0368d29bc002

2019-11-24 02:15:04.195 UTC  INFO messagingTaskExecutor-4 FlowRpcEndpoint$RpcMessageHandler:128 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing stats response - RULESTATSRESPONSE

2019-11-24 02:15:04.196 UTC  INFO messagingTaskExecutor-7 FlowRpcEndpoint$RpcMessageHandler:128 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing stats response - RULESTATSRESPONSE

2019-11-24 02:15:04.218 UTC  INFO messagingTaskExecutor-7 RuleStatsDataProcessor:58 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Starting enqueue with max queue limit 500, startEnqueuTim                               e - 1574561704218 hostId host-41

2019-11-24 02:15:04.219 UTC  INFO messagingTaskExecutor-4 RuleStatsDataProcessor:58 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Starting enqueue with max queue limit 500, startEnqueuTim                               e - 1574561704219 hostId host-51

2019-11-24 02:15:04.221 UTC  INFO Thread-15 RuleStatsDataService:92 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received rule stats response from host: host-41

2019-11-24 02:15:04.223 UTC  INFO Thread-15 RuleStatsDataService:92 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received rule stats response from host: host-51

2019-11-24 02:15:04.351 UTC  INFO messagingTaskExecutor-1 FlowRpcEndpoint$RpcMessageHandler:128 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing stats response - RULESTATSRESPONSE

2019-11-24 02:15:04.372 UTC  INFO messagingTaskExecutor-1 RuleStatsDataProcessor:58 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Starting enqueue with max queue limit 500, startEnqueuTim                               e - 1574561704372 hostId host-63

2019-11-24 02:15:04.375 UTC  INFO Thread-15 RuleStatsDataService:92 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received rule stats response from host: host-63

2019-11-24 02:15:04.394 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:186 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] using deployment unit DeploymentUnit [serviceId=service-2, s                               erviceInstanceId=null, agencyId=c27de4f2-bcf5-46c3-9628-6dd6381cd327, fabricStatus=ENABLED, healthStatus=SUCCEEDED, installedServiceVersion=6.4.5.13282012, fabricOperationStatus=null, objectId=deploymentunit                               -1, objectType=DeploymentUnit] for com.vmware.vim.eam.agency.statusChanged event on agency c27de4f2-bcf5-46c3-9628-6dd6381cd327

2019-11-24 02:15:04.480 UTC  INFO TaskFrameworkExecutor-28 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module fabric and total                                number of buckets 1

2019-11-24 02:15:04.482 UTC  INFO TaskFrameworkExecutor-28 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task core.services.fabric.stateUpdaterPurgeAgents                               TaskName [id:task-5459] is added to the SchedulerQueue

2019-11-24 02:15:04.483 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-5459 and running executor threads 2

2019-11-24 02:15:04.544 UTC  INFO TaskFrameworkExecutor-28 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool acti                               ve count 1

2019-11-24 02:15:04.665 UTC  INFO TaskFrameworkExecutor-1 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58341 to COMPLETED

2019-11-24 02:15:04.674 UTC  INFO TaskFrameworkExecutor-1 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-4525

2019-11-24 02:15:04.694 UTC  INFO TaskFrameworkExecutor-1 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

2019-11-24 02:15:06.262 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-11-24 02:15:06.263 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271995

Reply
0 Kudos
15 Replies
nagesh_u
Enthusiast
Enthusiast

Hi team,

can some one please help me on this ?

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

Moderator: Moved to NSX


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

anyone ?????????????????? please

Reply
0 Kudos
KocPawel
Hot Shot
Hot Shot

Maybe problem is connected with Workstation

Can you go though this video:

NSX in VMware Workstation - YouTube

And step by step check if are missing something.

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

Hi KocPawel,

Thank you so much for your replay and i will check and let you know

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

pastedImage_0.png

pastedImage_1.png

no luck same issue

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

pastedImage_0.png

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

any other sugition please

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

Hi Any help on those error ?

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

anyone ?????????????????????????

Reply
0 Kudos
scott28tt
VMware Employee
VMware Employee

This thread may be useful: NSX controller deployment. Operation failed on VC. For more details, refer to the rootCauseString or...


-------------------------------------------------------------------------------------------------------------------------------------------------------------

Although I am a VMware employee I contribute to VMware Communities voluntarily (ie. not in any official capacity)
VMware Training & Certification blog
Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

i can ping my VC host from the NSX server:, sometime conttroller is fail in 60% sometime 90%

^Cnalabnsx01.nalab.local> ping nalabvh01.nalab.local

PING nalabvh01.nalab.local (10.10.10.7): 56 data bytes

64 bytes from 10.10.10.7: icmp_seq=0 ttl=64 time=19.987 ms

64 bytes from 10.10.10.7: icmp_seq=1 ttl=64 time=30.484 ms

64 bytes from 10.10.10.7: icmp_seq=2 ttl=64 time=4.380 ms

64 bytes from 10.10.10.7: icmp_seq=3 ttl=64 time=7.937 ms

^C--- nalabvh01.nalab.local ping statistics ---

4 packets transmitted, 4 packets received, 0% packet loss

round-trip min/avg/max/stddev = 4.380/15.697/30.484/10.312 ms

nalabnsx01.nalab.local> ping nalabvh02.nalab.local

PING nalabvh02.nalab.local (10.10.10.5): 56 data bytes

64 bytes from 10.10.10.5: icmp_seq=0 ttl=64 time=6.824 ms

64 bytes from 10.10.10.5: icmp_seq=1 ttl=64 time=1.837 ms

64 bytes from 10.10.10.5: icmp_seq=2 ttl=64 time=11.345 ms

64 bytes from 10.10.10.5: icmp_seq=3 ttl=64 time=17.446 ms

^C--- nalabvh02.nalab.local ping statistics ---

4 packets transmitted, 4 packets received, 0% packet loss

round-trip min/avg/max/stddev = 1.837/9.363/17.446/5.752 ms

nalabnsx01.nalab.local> ping nalabvh03.nalab.local

PING nalabvh03.nalab.local (10.10.10.6): 56 data bytes

64 bytes from 10.10.10.6: icmp_seq=0 ttl=64 time=3.831 ms

64 bytes from 10.10.10.6: icmp_seq=1 ttl=64 time=14.988 ms

64 bytes from 10.10.10.6: icmp_seq=2 ttl=64 time=2.440 ms

64 bytes from 10.10.10.6: icmp_seq=3 ttl=64 time=7.123 ms

64 bytes from 10.10.10.6: icmp_seq=4 ttl=64 time=2.439 ms

^C--- nalabvh03.nalab.local ping statistics ---

5 packets transmitted, 5 packets received, 0% packet loss

round-trip min/avg/max/stddev = 2.439/6.164/14.988/4.732 ms

nalabnsx01.nalab.local>

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

got this error log:

2019-12-31 18:16:33.780 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a15836538b8c]52c5520f-031f-ac2b-72bc-860092df1814, operationID=opId-09723-950311

2019-12-31 18:16:34.156 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on propertyCollector, operationID=opId-09723-950312

2019-12-31 18:16:34.193 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionManager, operationID=opId-09723-950313

2019-12-31 18:16:35.820 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-12-31 18:16:35.821 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a15836538b8c]52c5520f-031f-ac2b-72bc-860092df1814, operationID=opId-09723-950314

2019-12-31 18:16:36.540 UTC  INFO taskScheduler-11 AppVisibilityCollectionThread:43 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] App Visibility Collection task invoked, collectionInProgress - false

2019-12-31 18:16:37.875 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

2019-12-31 18:16:37.876 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a15836538b8c]52c5520f-031f-ac2b-72bc-860092df1814, operationID=opId-09723-950315

2019-12-31 18:16:39.636 UTC ERROR taskScheduler-4 ResourcePoolVcOperationsImpl:443 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed to install OVF.

java.util.concurrent.TimeoutException: null

        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) [vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) [vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) [vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

2019-12-31 18:16:39.638 UTC ERROR taskScheduler-4 AbstractSvmMgmtTask:206 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed, jobId 'jobdata-15523' during svm mgmt task. Attempting retry: '1'

com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) [vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

Caused by: java.util.concurrent.TimeoutException

        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) ~[vsphere-1.0.jar:?]

        ... 36 more

2019-12-31 18:16:39.642 UTC ERROR taskScheduler-4 Worker:229 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] BaseException thrown while executing task instance taskinstance-215650

com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

        at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

        at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

Caused by: java.util.concurrent.TimeoutException

        at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) ~[vsphere-1.0.jar:?]

        ... 36 more

2019-12-31 18:16:39.676 UTC  INFO taskScheduler-4 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-203842 to FAILED

2019-12-31 18:16:39.708 UTC  INFO TaskFrameworkExecutor-17 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool active count 0

2019-12-31 18:16:39.918 UTC  INFO DeploymentMonitor DeploymentMonitor:131 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Purge work for finished job jobdata-15523 in status: FAILED

2019-12-31 18:16:39.919 UTC  INFO DeploymentMonitor DeploymentMonitor:135 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Job jobdata-15523 for controller controller-9 failed!

2019-12-31 18:16:39.963 UTC  INFO DeploymentMonitor DeploymentMonitor:175 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] about to remove controller from database: controller-9 VM id null ip 10.10.10.136 uuid null version 12633898 VSM id 564D66D5-CABD-D4B2-9198-08B2277F333C

2019-12-31 18:16:39.965 UTC  INFO DeploymentMonitor RelationshipManagerImpl:840 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Recursively removing domain object controller-9 updateParent false

2019-12-31 18:16:40.009 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

is this time out issue ?

2019-12-31 18:16:39.636 UTC ERROR taskScheduler-4 ResourcePoolVcOperationsImpl:443 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed to install OVF.

java.util.concurrent.TimeoutException: null

Reply
0 Kudos
nagesh_u
Enthusiast
Enthusiast

what about this ?????

019-12-31 18:16:39.642 UTC ERROR taskScheduler-4 Worker:229 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] BaseException thrown while executing task instance taskinstance-215650

com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

        at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

        at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

Reply
0 Kudos