7 Replies Latest reply on Dec 6, 2019 3:33 PM by nagesh_u

    NSX controller deploy error vcoperationfailedexception: core-services:1500:operation failed on vc for more

    nagesh_u Lurker

      Hi Team,

       

      i am trying to deploy NSX controller but i am getting error 70% , its says as vcoperationfailedexception: core-services:1500:operation failed on vc for more.

       

      my NSX version is:6.4.5

      deployed in Vmware workstation 15.0.2 build-10952284

      my DVS port configured with 1600MTU

      NSX and Vcenter sitting in same VLAN so create the IP POOL with same VLAN, All device is sitting same VLAN.

      NTP is working fine

       

      please find the log

       

      15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271942

      2019-11-24 02:13:50.816 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

      2019-11-24 02:13:50.862 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

      2019-11-24 02:13:50.863 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

      2019-11-24 02:13:50.864 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

      2019-11-24 02:13:50.883 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 48

      2019-11-24 02:13:50.884 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

      2019-11-24 02:13:50.884 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48257 num vc objs:48

      2019-11-24 02:13:50.894 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271943

      2019-11-24 02:13:52.444 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:13:52.446 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271944

      2019-11-24 02:13:52.510 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

      2019-11-24 02:13:52.528 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

      2019-11-24 02:13:52.529 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

      2019-11-24 02:13:52.530 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

      2019-11-24 02:13:52.532 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 18

      2019-11-24 02:13:52.533 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

      2019-11-24 02:13:52.533 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48258 num vc objs:48

      2019-11-24 02:13:52.539 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271945

      2019-11-24 02:13:54.487 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:13:54.489 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271946

      2019-11-24 02:13:56.546 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:13:56.548 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271947

      2019-11-24 02:13:57.452 UTC  INFO taskScheduler-6 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58337 to EXECUTING

      2019-11-24 02:13:57.482 UTC  INFO taskScheduler-6 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module policy and total number o                               f buckets 1

      2019-11-24 02:13:57.482 UTC  INFO taskScheduler-6 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task com.vmware.vshield.blueprint.policy.task.policyStatus                               NotificationTask [id:task-100] is added to the SchedulerQueue

      2019-11-24 02:13:57.483 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-100 and running executor threads 2

      2019-11-24 02:13:57.563 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271948

      2019-11-24 02:13:57.754 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271949

      2019-11-24 02:13:57.814 UTC  INFO TaskFrameworkExecutor-20 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58337 to COMPLETED

      2019-11-24 02:13:57.822 UTC  INFO TaskFrameworkExecutor-20 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-96

      2019-11-24 02:13:57.855 UTC  INFO TaskFrameworkExecutor-20 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool acti                               ve count 1

      2019-11-24 02:13:58.615 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:13:58.616 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271950

      2019-11-24 02:14:00.748 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:00.750 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271951

      2019-11-24 02:14:02.790 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:02.792 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271952

      2019-11-24 02:14:04.653 UTC  INFO EamConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on property                               Collector, operationID=opId-c8639-271953

      2019-11-24 02:14:04.934 UTC  INFO EamConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionManager,                                operationID=opId-c8639-271954

      2019-11-24 02:14:04.987 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:04.988 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271955

      2019-11-24 02:14:07.122 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:07.124 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271956

      2019-11-24 02:14:09.168 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:09.169 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271957

      2019-11-24 02:14:11.267 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:11.269 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271958

      2019-11-24 02:14:12.821 UTC  INFO messagingTaskExecutor-7 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-63

      2019-11-24 02:14:12.964 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271959

      2019-11-24 02:14:13.022 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271960

      2019-11-24 02:14:13.301 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:13.302 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271961

      2019-11-24 02:14:13.356 UTC  INFO messagingTaskExecutor-6 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-51

      2019-11-24 02:14:13.381 UTC  INFO messagingTaskExecutor-8 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-41

      2019-11-24 02:14:15.340 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:15.342 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271962

      2019-11-24 02:14:16.169 UTC  INFO taskScheduler-19 AppVisibilityCollectionThread:43 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] App Visibility Collection task invoked, collectionInProgr                               ess - false

      2019-11-24 02:14:17.382 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:17.383 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271963

      2019-11-24 02:14:19.438 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:19.439 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271964

      2019-11-24 02:14:21.557 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:21.558 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271965

      2019-11-24 02:14:23.678 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:23.679 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271966

      2019-11-24 02:14:24.125 UTC  INFO edgeVseMonitoringThread EdgeVseHealthMonitoringThread:298 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Finished Health check for 0 edge vms in 0 millise                               c

      2019-11-24 02:14:25.789 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:25.792 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271967

      2019-11-24 02:14:26.073 UTC ERROR taskScheduler-15 ResourcePoolVcOperationsImpl:443 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed to install OVF.

      java.util.concurrent.TimeoutException: null

              at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) [vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [s                               pring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE                               ]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-a                               spects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) [vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [s                               pring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE                               ]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-a                               spects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) [vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) [s                               pring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) [spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE                               ]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) [spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) [spring-a                               spects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

              at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

      2019-11-24 02:14:26.080 UTC ERROR taskScheduler-15 AbstractSvmMgmtTask:206 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] Failed, jobId 'jobdata-4521' during svm mgmt task. Attempting ret                               ry: '1'

      com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) [vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

              at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

      Caused by: java.util.concurrent.TimeoutException

              at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) ~[vsphere-1.0.jar:?]

              ... 36 more

      2019-11-24 02:14:26.085 UTC ERROR taskScheduler-15 Worker:229 - - [nsxv@6876 comp="nsx-manager" level="ERROR" subcomp="manager"] BaseException thrown while executing task instance taskinstance-59201

      com.vmware.vshield.vsm.inventory.vcoperations.VcOperationFailedException: nested exception is java.util.concurrent.TimeoutException

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:455) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl$AjcClosure9.run(ResourcePoolVcOperationsImpl.java:1) ~[vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf(ResourcePoolVcOperationsImpl.java:402) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm_aroundBody18(OvfInstaller.java:446) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure19.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvfAndManageStrandedVm(OvfInstaller.java:313) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf_aroundBody16(OvfInstaller.java:287) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.OvfInstaller$AjcClosure17.run(OvfInstaller.java:1) ~[vsphere-1.0.jar:?]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96cproceed(AbstractTransactionAspect.aj:66) ~[                               spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect$AbstractTransactionAspect$1.proceedWithInvocation(AbstractTransactionAspect.aj:72) ~[spring-aspects-5.1.4.RELEASE.jar:5.1.4.RELEAS                               E]

              at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:294) ~[spring-tx-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at org.springframework.transaction.aspectj.AbstractTransactionAspect.ajc$around$org_springframework_transaction_aspectj_AbstractTransactionAspect$1$2a73e96c(AbstractTransactionAspect.aj:70) ~[spring-                               aspects-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at com.vmware.vshield.vsm.ovf.OvfInstaller.installOvf(OvfInstaller.java:280) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfHelper.installOvf(DeployOvfHelper.java:86) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deployVm(DeployOvfTask.java:185) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.deploy(DeployOvfTask.java:148) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.DeployOvfTask.runInternal(DeployOvfTask.java:130) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.ovf.task.AbstractSvmMgmtTask.run(AbstractSvmMgmtTask.java:189) ~[vsphere-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker.runtask(Worker.java:194) [vsm-core-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker.access$0(Worker.java:167) [vsm-core-1.0.jar:?]

              at com.vmware.vshield.vsm.task.service.Worker$1.run(Worker.java:137) [vsm-core-1.0.jar:?]

              at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) [spring-context-5.1.4.RELEASE.jar:5.1.4.RELEASE]

              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_202]

              at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_202]

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_202]

              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_202]

              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_202]

              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_202]

              at java.lang.Thread.run(Thread.java:748) [?:1.8.0_202]

      Caused by: java.util.concurrent.TimeoutException

              at java.util.concurrent.FutureTask.get(FutureTask.java:205) ~[?:1.8.0_202]

              at com.vmware.vshield.vsm.inventory.vcoperations.impl.ResourcePoolVcOperationsImpl.installOvf_aroundBody8(ResourcePoolVcOperationsImpl.java:425) ~[vsphere-1.0.jar:?]

              ... 36 more

      2019-11-24 02:14:26.162 UTC  INFO taskScheduler-15 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58306 to FAILED

      2019-11-24 02:14:26.214 UTC  INFO TaskFrameworkExecutor-10 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool acti                               ve count 0

      2019-11-24 02:14:26.854 UTC  INFO DeploymentMonitor DeploymentMonitor:131 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Purge work for finished job jobdata-4521 in status: FAILED

      2019-11-24 02:14:26.855 UTC  INFO DeploymentMonitor DeploymentMonitor:135 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Job jobdata-4521 for controller controller-5 failed!

      2019-11-24 02:14:26.887 UTC  INFO DeploymentMonitor DeploymentMonitor:175 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] about to remove controller from database: controller-5 VM id null i                               p 10.10.10.136 uuid null version 12633898 VSM id 564D66D5-CABD-D4B2-9198-08B2277F333C

      2019-11-24 02:14:26.888 UTC  INFO DeploymentMonitor RelationshipManagerImpl:840 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Recursively removing domain object controller-5 updateParent                                false

      2019-11-24 02:14:27.020 UTC  INFO DeploymentMonitor AuditingServiceImpl:174 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [AuditLog] UserName:'', ModuleName:'VdnNvpController', Operation:                               'DELETE', Resource:'nalabcnt01', Time:'Sun Nov 24 02:14:26.918 UTC 2019', Status:'SUCCESS', Universal Object:'false'

      2019-11-24 02:14:27.034 UTC  INFO DeploymentMonitor IPAMServiceImpl:420 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Released IP address 10.10.10.136 from pool ipaddresspool-1, status =                                true

      2019-11-24 02:14:27.035 UTC  INFO DeploymentMonitor DeploymentMonitor:182 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] release ip to ip pool: ipaddresspool-1 10.10.10.136

      2019-11-24 02:14:27.043 UTC  INFO DCNPool-10 VirtualWireInFirewallRuleNotificationHandler:59 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Recieved VDN DELETE notification for context con                               troller-5:Controller

      2019-11-24 02:14:27.512 UTC  INFO taskScheduler-10 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58338 to EXECUTING

      2019-11-24 02:14:27.546 UTC  INFO taskScheduler-10 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module policy and total number                                of buckets 1

      2019-11-24 02:14:27.548 UTC  INFO taskScheduler-10 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task com.vmware.vshield.blueprint.policy.task.policyStatu                               sNotificationTask [id:task-100] is added to the SchedulerQueue

      2019-11-24 02:14:27.549 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-100 and running executor threads 1

      2019-11-24 02:14:27.669 UTC  INFO TaskFrameworkExecutor-9 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58338 to COMPLETED

      2019-11-24 02:14:27.679 UTC  INFO TaskFrameworkExecutor-9 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-96

      2019-11-24 02:14:27.701 UTC  INFO TaskFrameworkExecutor-9 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

      2019-11-24 02:14:27.965 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:27.966 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271968

      2019-11-24 02:14:28.122 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271969

      2019-11-24 02:14:28.254 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271970

      2019-11-24 02:14:30.119 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:30.120 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271971

      2019-11-24 02:14:31.134 UTC  INFO taskScheduler-13 FlowCollectionThread:44 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Flow Collection task invoked, collectionInProgress - false

      2019-11-24 02:14:32.210 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:32.212 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271972

      2019-11-24 02:14:34.257 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:34.258 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271973

      2019-11-24 02:14:36.300 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:36.302 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271974

      2019-11-24 02:14:36.903 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

      2019-11-24 02:14:36.946 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

      2019-11-24 02:14:36.948 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

      2019-11-24 02:14:36.950 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

      2019-11-24 02:14:36.959 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 47

      2019-11-24 02:14:36.960 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

      2019-11-24 02:14:36.960 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48259 num vc objs:48

      2019-11-24 02:14:36.968 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271975

      2019-11-24 02:14:38.353 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:38.354 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271976

      2019-11-24 02:14:40.417 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:40.419 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271977

      2019-11-24 02:14:42.572 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:42.573 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271978

      2019-11-24 02:14:42.850 UTC  INFO messagingTaskExecutor-1 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-63

      2019-11-24 02:14:43.363 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271979

      2019-11-24 02:14:43.402 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271980

      2019-11-24 02:14:43.404 UTC  INFO messagingTaskExecutor-8 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-51

      2019-11-24 02:14:43.514 UTC  INFO messagingTaskExecutor-7 ControllerInfoHandler:48 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] handle controller info request: host-41

      2019-11-24 02:14:44.632 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:44.633 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271981

      2019-11-24 02:14:46.672 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:46.673 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271982

      2019-11-24 02:14:48.913 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:48.915 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271983

      2019-11-24 02:14:50.966 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

      2019-11-24 02:14:50.976 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:50.977 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271984

      2019-11-24 02:14:51.082 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

      2019-11-24 02:14:51.084 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

      2019-11-24 02:14:51.086 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

      2019-11-24 02:14:51.090 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 83

      2019-11-24 02:14:51.091 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

      2019-11-24 02:14:51.092 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48260 num vc objs:48

      2019-11-24 02:14:51.102 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271985

      2019-11-24 02:14:52.734 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Virtual Center: Updating Inventory. new:0 modified:1 removed:0

      2019-11-24 02:14:52.768 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] 48/48 objects published.

      2019-11-24 02:14:52.770 UTC  INFO ViInventoryThread VimObjectBridge:891 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Ending inventory update

      2019-11-24 02:14:52.771 UTC  INFO ViInventoryThread VimObjectBridge:224 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing 1 updates and 0 deletions for this transaction

      2019-11-24 02:14:52.779 UTC  INFO ViInventoryThread VimObjectBridge:229 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] VimObjectBridge: Time taken to process transaction : 30

      2019-11-24 02:14:52.780 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] cleared the deleted objects set

      2019-11-24 02:14:52.780 UTC  INFO ViInventoryThread Tracer:269 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Resolved, last version:48261 num vc objs:48

      2019-11-24 02:14:52.797 UTC  INFO ViInventoryThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.waitForUpdatesEx on session[529199af-b05d-8                               861-f42d-a15836538b8c]529723d7-30ea-650a-c137-6e871f307225, operationID=opId-09723-271986

      2019-11-24 02:14:53.051 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:53.052 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271987

      2019-11-24 02:14:55.113 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:55.114 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271988

      2019-11-24 02:14:57.383 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:57.384 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271989

      2019-11-24 02:14:58.277 UTC  INFO taskScheduler-8 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58339 to EXECUTING

      2019-11-24 02:14:58.330 UTC  INFO taskScheduler-8 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module policy and total number o                               f buckets 1

      2019-11-24 02:14:58.331 UTC  INFO taskScheduler-8 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task com.vmware.vshield.blueprint.policy.task.policyStatus                               NotificationTask [id:task-100] is added to the SchedulerQueue

      2019-11-24 02:14:58.332 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-100 and running executor threads 1

      2019-11-24 02:14:58.472 UTC  INFO TaskFrameworkExecutor-6 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58339 to COMPLETED

      2019-11-24 02:14:58.480 UTC  INFO TaskFrameworkExecutor-6 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-96

      2019-11-24 02:14:58.484 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking PropertyCollector.retrievePropertiesEx on pr                               opertyCollector, operationID=opId-09723-271990

      2019-11-24 02:14:58.539 UTC  INFO TaskFrameworkExecutor-6 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

      2019-11-24 02:14:58.627 UTC  INFO DefaultVcConnectionKeepaliveThread VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking SessionManager.sessionIsActive on SessionMan                               ager, operationID=opId-09723-271991

      2019-11-24 02:14:59.878 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:14:59.879 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271992

      2019-11-24 02:15:00.100 UTC  INFO taskScheduler-9 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58340 to EXECUTING

      2019-11-24 02:15:00.142 UTC  INFO taskScheduler-9 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module fabric_state_updater and                                total number of buckets 1

      2019-11-24 02:15:00.143 UTC  INFO taskScheduler-9 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task core.services.fabric.stateUpdaterTaskName [id:task-98                               ] is added to the SchedulerQueue

      2019-11-24 02:15:00.143 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-98 and running executor threads 1

      2019-11-24 02:15:00.331 UTC  INFO TaskFrameworkExecutor-3 TaskServiceImpl:101 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:Created Job with ID jobdata-4525

      2019-11-24 02:15:00.334 UTC  INFO TaskFrameworkExecutor-3 TaskServiceImpl:425 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:Scheduling Job jobdata-4525

      2019-11-24 02:15:00.351 UTC  INFO TaskFrameworkExecutor-3 SchedulerService:198 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Job jobdata-4525 is scheduled and put data in to scheduleInfoT                               able

      2019-11-24 02:15:00.379 UTC  INFO UnitOfWorkPool-1 SchedulerService:193 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received job creation DCN for job id jobdata-4525

      2019-11-24 02:15:00.397 UTC  INFO TaskFrameworkExecutor-3 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58340 to COMPLETED

      2019-11-24 02:15:00.412 UTC  INFO TaskFrameworkExecutor-3 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-94

      2019-11-24 02:15:00.424 UTC  INFO TaskFrameworkExecutor-3 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

      2019-11-24 02:15:00.877 UTC  INFO pool-17-thread-1 ImmediateScheduler:35 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:Schedule Now Job ID jobdata-4525

      2019-11-24 02:15:00.919 UTC  INFO taskScheduler-30 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58341 to EXECUTING

      2019-11-24 02:15:00.972 UTC  INFO taskScheduler-30 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module fabric and total number                                of buckets 1

      2019-11-24 02:15:00.974 UTC  INFO taskScheduler-30 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task core.services.fabric.stateUpdaterPerDUTaskName;deplo                               ymentunit-1 [id:task-5458] is added to the SchedulerQueue

      2019-11-24 02:15:00.975 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-5458 and running executor threads 1

      2019-11-24 02:15:01.919 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:15:01.921 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271993

      2019-11-24 02:15:03.382 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:415 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current eamAgentRuntimeStatus is green, and eamAgentGoalStat                               e is enabled for 902be1ac-3a0d-4268-90cc-ce9ec886a24f

      2019-11-24 02:15:03.601 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:415 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current eamAgentRuntimeStatus is green, and eamAgentGoalStat                               e is enabled for 40014283-52ae-425d-a076-2195db0c50d0

      2019-11-24 02:15:03.968 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:415 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Current eamAgentRuntimeStatus is green, and eamAgentGoalStat                               e is enabled for 2a46d6dd-13b1-4d0a-889d-751bc233faf2

      2019-11-24 02:15:04.022 UTC  INFO taskScheduler-12 RuleStatsCollector:40 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Stats Collection task invoked, collectionInProgress - false

      2019-11-24 02:15:04.025 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:15:04.032 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271994

      2019-11-24 02:15:04.061 UTC  INFO taskScheduler-12 RuleStatsDataService:157 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Rule stats collected for 1 rules

      2019-11-24 02:15:04.142 UTC  INFO taskScheduler-12 RuleStatsDataService:134 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Send request to host for stats collection

      2019-11-24 02:15:04.154 UTC  INFO taskScheduler-12 FirewallProtobufConverter:1560 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Setting client id - 49b1da11-7494-40f1-99f4-747e3f517ffe

      2019-11-24 02:15:04.155 UTC  INFO taskScheduler-12 FirewallMessagingManager:844 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Sending StatsReq to 49b1da11-7494-40f1-99f4-747e3f517ffe

      2019-11-24 02:15:04.159 UTC  INFO taskScheduler-12 FirewallProtobufConverter:1560 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Setting client id - a2ceab2c-4f44-4d0e-8bdc-715f368e642f

      2019-11-24 02:15:04.161 UTC  INFO taskScheduler-12 FirewallMessagingManager:844 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Sending StatsReq to a2ceab2c-4f44-4d0e-8bdc-715f368e642f

      2019-11-24 02:15:04.164 UTC  INFO taskScheduler-12 FirewallProtobufConverter:1560 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Setting client id - ca61ca3f-e207-4410-99b7-0368d29bc002

      2019-11-24 02:15:04.165 UTC  INFO taskScheduler-12 FirewallMessagingManager:844 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Sending StatsReq to ca61ca3f-e207-4410-99b7-0368d29bc002

      2019-11-24 02:15:04.195 UTC  INFO messagingTaskExecutor-4 FlowRpcEndpoint$RpcMessageHandler:128 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing stats response - RULESTATSRESPONSE

      2019-11-24 02:15:04.196 UTC  INFO messagingTaskExecutor-7 FlowRpcEndpoint$RpcMessageHandler:128 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing stats response - RULESTATSRESPONSE

      2019-11-24 02:15:04.218 UTC  INFO messagingTaskExecutor-7 RuleStatsDataProcessor:58 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Starting enqueue with max queue limit 500, startEnqueuTim                               e - 1574561704218 hostId host-41

      2019-11-24 02:15:04.219 UTC  INFO messagingTaskExecutor-4 RuleStatsDataProcessor:58 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Starting enqueue with max queue limit 500, startEnqueuTim                               e - 1574561704219 hostId host-51

      2019-11-24 02:15:04.221 UTC  INFO Thread-15 RuleStatsDataService:92 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received rule stats response from host: host-41

      2019-11-24 02:15:04.223 UTC  INFO Thread-15 RuleStatsDataService:92 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received rule stats response from host: host-51

      2019-11-24 02:15:04.351 UTC  INFO messagingTaskExecutor-1 FlowRpcEndpoint$RpcMessageHandler:128 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Processing stats response - RULESTATSRESPONSE

      2019-11-24 02:15:04.372 UTC  INFO messagingTaskExecutor-1 RuleStatsDataProcessor:58 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Starting enqueue with max queue limit 500, startEnqueuTim                               e - 1574561704372 hostId host-63

      2019-11-24 02:15:04.375 UTC  INFO Thread-15 RuleStatsDataService:92 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Received rule stats response from host: host-63

      2019-11-24 02:15:04.394 UTC  INFO TaskFrameworkExecutor-28 EamListenerHelper:186 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] using deployment unit DeploymentUnit [serviceId=service-2, s                               erviceInstanceId=null, agencyId=c27de4f2-bcf5-46c3-9628-6dd6381cd327, fabricStatus=ENABLED, healthStatus=SUCCEEDED, installedServiceVersion=6.4.5.13282012, fabricOperationStatus=null, objectId=deploymentunit                               -1, objectType=DeploymentUnit] for com.vmware.vim.eam.agency.statusChanged event on agency c27de4f2-bcf5-46c3-9628-6dd6381cd327

      2019-11-24 02:15:04.480 UTC  INFO TaskFrameworkExecutor-28 SchedulerQueueServiceImpl:71 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] [TF] Created a new bucket for module fabric and total                                number of buckets 1

      2019-11-24 02:15:04.482 UTC  INFO TaskFrameworkExecutor-28 SchedulerQueueServiceImpl:90 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] The task core.services.fabric.stateUpdaterPurgeAgents                               TaskName [id:task-5459] is added to the SchedulerQueue

      2019-11-24 02:15:04.483 UTC  INFO pool-16-thread-1 ScheduleSynchronizer:62 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Start executing task: task-5459 and running executor threads 2

      2019-11-24 02:15:04.544 UTC  INFO TaskFrameworkExecutor-28 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool acti                               ve count 1

      2019-11-24 02:15:04.665 UTC  INFO TaskFrameworkExecutor-1 JobWorker:243 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Updating the status for jobinstance-58341 to COMPLETED

      2019-11-24 02:15:04.674 UTC  INFO TaskFrameworkExecutor-1 JobWorker:129 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] TF:All tasks in the job completed for jobdata-4525

      2019-11-24 02:15:04.694 UTC  INFO TaskFrameworkExecutor-1 ScheduleSynchronizer:74 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Releasing a thread to executor pool and executor pool activ                               e count 0

      2019-11-24 02:15:06.262 UTC  INFO HttpNfcLeaseExtender HttpNfcLeaseExtender:46 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Extending HttpNfcLease..

      2019-11-24 02:15:06.263 UTC  INFO HttpNfcLeaseExtender VcConnection$1:227 - - [nsxv@6876 comp="nsx-manager" level="INFO" subcomp="manager"] Invoking HttpNfcLease.progress on session[529199af-b05d-8861-f42d-a                               15836538b8c]52c7128a-5d10-42bd-0475-f749b8046b25, operationID=opId-09723-271995