Hi all,
I have a Horizon 7.7 environment. I am running a Instant Clone pool and appvolume of Windows 10 VMs . I log to AppVolumes Manager and provision/configure the appstack successfully. I then successfully assign the appstack to the user and system in my instant clone pool and attached Appstack and Writable Volume all user and everything normal except running fifth esxi host. when i login vdi on 5.esx cannot attached any vmdk to vm.
Server Side error log
C:\Program Files (x86)\CloudVolumes\Manager\log\production0000x.log
[2019-03-10 11:26:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 125220 secs old
[2019-03-10 11:26:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 131744 secs old
[2019-03-10 11:26:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 125220 secs old
[2019-03-10 11:26:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#42074280 root@hldmrk-esx-05.xyz.local>" - 125220 secs old
[2019-03-10 11:26:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#38242480 root@hldmrk-esx-05.xyz.local>" - 125220 secs old
[2019-03-10 11:26:16 UTC] P3984DJ14691 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 10061 secs old
[2019-03-10 11:26:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 128540 secs old
[2019-03-10 11:26:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 128540 secs old
[2019-03-10 11:26:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 132328 secs old
[2019-03-10 11:26:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#168393740 root@hldmrk-esx-05.xyz.local>" - 128540 secs old
[2019-03-10 11:26:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#114259320 root@hldmrk-esx-05.xyz.local>" - 128540 secs old
[2019-03-10 11:26:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 131778 secs old
[2019-03-10 11:26:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 129944 secs old
[2019-03-10 11:26:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 129944 secs old
[2019-03-10 11:26:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#34309560 root@hldmrk-esx-05.xyz.local>" - 129944 secs old
[2019-03-10 11:26:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#22644900 root@hldmrk-esx-05.xyz.local>" - 129944 secs old
[2019-03-10 11:26:23 UTC] P3932 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 132335 secs old
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 1981 secs old
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: Validating SSL certificate for "hldmrk-vcn-10.xyz.local": Trusted by an administrator
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: Validated SSL certificate for "hldmrk-vcn-10.xyz.local". Validation Result = true, Validation Status = 2.
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: Size of AVM SSL Fingerprint cache is 1
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: Validating SSL certificate for "hldmrk-vcn-10.xyz.local": Trusted by an administrator
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: Validated SSL certificate for "hldmrk-vcn-10.xyz.local". Validation Result = true, Validation Status = 2.
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: Size of AVM SSL Fingerprint cache is 1
[2019-03-10 11:26:29 UTC] P6864DJ14743 INFO Cvo: [AdapterState] Active - "Hypervisor <VMware vCenter#26203460 administrator@vsphere.local@hldmrk-vcn-10.xyz.local>" - 443 secs old - Ping took 75ms
[2019-03-10 11:27:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 125280 secs old
[2019-03-10 11:27:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 131804 secs old
[2019-03-10 11:27:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 125280 secs old
[2019-03-10 11:27:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#42074280 root@hldmrk-esx-05.xyz.local>" - 125280 secs old
[2019-03-10 11:27:10 UTC] P3728 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#38242480 root@hldmrk-esx-05.xyz.local>" - 125280 secs old
[2019-03-10 11:27:16 UTC] P3984DJ14691 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 10121 secs old
[2019-03-10 11:27:17 UTC] P3984DJ14749 INFO 2019-03-10T14:27:17+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job UpdateTimeseriesJob#perform (id=14766) RUNNING
[2019-03-10 11:27:17 UTC] P3984DJ14766 INFO Cvo: Starting job: Update Graphs #<Thread:0x000000000134ff20>
[2019-03-10 11:27:17 UTC] P3984DJ14766 INFO Cvo: Background worker updating snapvol timeseries
[2019-03-10 11:27:17 UTC] P3984DJ14766 INFO Cvo: Background worker updated snapvol_timeseries successfully
[2019-03-10 11:27:17 UTC] P3984DJ14766 INFO Cvo: Finished job: Update Graphs #<Thread:0x000000000134ff20>
[2019-03-10 11:27:17 UTC] P3984DJ14766 INFO 2019-03-10T14:27:17+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job UpdateTimeseriesJob#perform (id=14766) COMPLETED after 0.2557
[2019-03-10 11:27:17 UTC] P3984DJ14766 INFO 2019-03-10T14:27:17+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job ExpireSessionsJob#perform (id=14767) RUNNING
[2019-03-10 11:27:17 UTC] P3984DJ14767 INFO Cvo: Starting job: Expire Sessions #<Thread:0x000000000134ff20>
[2019-03-10 11:27:17 UTC] P3984DJ14767 INFO Cvo: Background worker expiring old sessions
[2019-03-10 11:27:17 UTC] P3984DJ14767 INFO Cvo: Finished job: Expire Sessions #<Thread:0x000000000134ff20>
[2019-03-10 11:27:17 UTC] P3984DJ14767 INFO 2019-03-10T14:27:17+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job ExpireSessionsJob#perform (id=14767) COMPLETED after 0.0152
[2019-03-10 11:27:17 UTC] P3984DJ14767 INFO 2019-03-10T14:27:17+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job RemoveStaleStorageLocationsJob#perform (id=14768) RUNNING
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Starting job: Remove Stale Storages #<Thread:0x000000000134ff20>
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 10123 secs old
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Hypervisor Pool: Acquired 'vcenters' adapter 1 of 25 for 'vcenters-1982216792' in 0.0 seconds
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Starting cleanup of stale storage locations
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Hypervisor Pool: Acquired 'vcenter' adapter 1 of 25 for 'vcenter93436986' in 0.0 seconds
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO RvSphere: Connecting to vSphere "administrator@vsphere.local@hldmrk-vcn-10.xyz.local" on "#<Thread:0x0000000003159480>"
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO RvSphere: Establishing verified SSL connection to vSphere at "https://administrator@vsphere.local@hldmrk-vcn-10.xyz.local" without certificate authorities due to missing file: "C:/Program Files (x86)/CloudVolumes/Manager/config/cacert.pem"
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Validating SSL certificate for "hldmrk-vcn-10.xyz.local": Trusted by an administrator
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Validated SSL certificate for "hldmrk-vcn-10.xyz.local". Validation Result = true, Validation Status = 2.
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Size of AVM SSL Fingerprint cache is 1
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Validating SSL certificate for "hldmrk-vcn-10.xyz.local": Trusted by an administrator
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Validated SSL certificate for "hldmrk-vcn-10.xyz.local". Validation Result = true, Validation Status = 2.
[2019-03-10 11:27:17 UTC] P3984DJ14768 INFO Cvo: Size of AVM SSL Fingerprint cache is 1
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO RvSphere: Connection to vSphere at "administrator@vsphere.local@hldmrk-vcn-10.xyz.local" on "#<Thread:0x0000000003159480>" was successful and took 112.07ms
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO RvSphere: vSphere Server: "VMware vCenter Server 6.7.0 build-11727113" Instance: "d47a4f88-462e-4cd8-98f1-88d5be1586cf" License: "VMware VirtualCenter Server"
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO Cvo: Size of the AVM SSL Fingerprint cache is 0
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO Cvo: Processing 0 stale storage locations
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO Cvo: Finished job: Remove Stale Storages #<Thread:0x000000000134ff20>
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO 2019-03-10T14:27:18+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job RemoveStaleStorageLocationsJob#perform (id=14768) COMPLETED after 0.5011
[2019-03-10 11:27:18 UTC] P3984DJ14768 INFO 2019-03-10T14:27:18+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job ImportStorageGroupsJob#perform (id=14769) RUNNING
[2019-03-10 11:27:18 UTC] P3984DJ14769 INFO Cvo: Starting job: Import Storage Groups #<Thread:0x000000000134ff20>
[2019-03-10 11:27:18 UTC] P3984DJ14769 INFO Cvo: Importing volumes from storage groups
[2019-03-10 11:27:18 UTC] P3984DJ14769 INFO Cvo: Finished job: Import Storage Groups #<Thread:0x000000000134ff20>
[2019-03-10 11:27:18 UTC] P3984DJ14769 INFO 2019-03-10T14:27:18+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job ImportStorageGroupsJob#perform (id=14769) COMPLETED after 0.0067
[2019-03-10 11:27:18 UTC] P3984DJ14769 INFO 2019-03-10T14:27:18+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job MarshalWritablesJob#perform (id=14770) RUNNING
[2019-03-10 11:27:18 UTC] P3984DJ14770 INFO Cvo: Starting job: Marshal Writables #<Thread:0x000000000134ff20>
[2019-03-10 11:27:18 UTC] P3984DJ14770 INFO Cvo: "Queued 0 writable volume 0 operations (0 move, 0 restore, 0 backup)"
[2019-03-10 11:27:18 UTC] P3984DJ14770 INFO Cvo: Finished job: Marshal Writables #<Thread:0x000000000134ff20>
[2019-03-10 11:27:18 UTC] P3984DJ14770 INFO 2019-03-10T14:27:18+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] Job MarshalWritablesJob#perform (id=14770) COMPLETED after 0.0153
[2019-03-10 11:27:18 UTC] P3984DJ14770 INFO 2019-03-10T14:27:18+0300: [Worker(host:HLDMRK-APP-01 pid:3984)] 5 jobs processed at 5.6051 j/s, 0 failed
[2019-03-10 11:27:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 128600 secs old
[2019-03-10 11:27:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 128600 secs old
[2019-03-10 11:27:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 132388 secs old
[2019-03-10 11:27:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#168393740 root@hldmrk-esx-05.xyz.local>" - 128600 secs old
[2019-03-10 11:27:19 UTC] P3580 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#114259320 root@hldmrk-esx-05.xyz.local>" - 128600 secs old
[2019-03-10 11:27:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 131838 secs old
[2019-03-10 11:27:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 130004 secs old
[2019-03-10 11:27:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 130004 secs old
[2019-03-10 11:27:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#34309560 root@hldmrk-esx-05.xyz.local>" - 130004 secs old
[2019-03-10 11:27:20 UTC] P3292 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <VMware ESX#22644900 root@hldmrk-esx-05.xyz.local>" - 130004 secs old
[2019-03-10 11:27:23 UTC] P3932 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 132395 secs old
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: [AdapterState] Not Connected - "Hypervisor <Multiple vCenters>" - 2041 secs old
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: Validating SSL certificate for "hldmrk-vcn-10.xyz.local": Trusted by an administrator
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: Validated SSL certificate for "hldmrk-vcn-10.xyz.local". Validation Result = true, Validation Status = 2.
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: Size of AVM SSL Fingerprint cache is 1
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: Validating SSL certificate for "hldmrk-vcn-10.xyz.local": Trusted by an administrator
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: Validated SSL certificate for "hldmrk-vcn-10.xyz.local". Validation Result = true, Validation Status = 2.
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: Size of AVM SSL Fingerprint cache is 1
[2019-03-10 11:27:29 UTC] P6864DJ14743 INFO Cvo: [AdapterState] Active - "Hypervisor <VMware vCenter#26203460 administrator@vsphere.local@hldmrk-vcn-10.xyz.local>" - 503 secs old - Ping took 64ms
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Started GET "/cv_api/jobs/schedule?jobs=update_timeseries" for 127.0.0.1 at 2019-03-10 14:27:31 +0300
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Processing by CvApi::JobsController#schedule as */*
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Parameters: {"jobs"=>"update_timeseries"}
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Referred by nothing using Ruby
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Cvo: Scheduling job "Update Graphs"
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Cvo: Unable to run job "UpdateTimeseriesJob" because it ran too recently, it can run in 30 minutes
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Rendered text template (0.0ms)
[2019-03-10 11:27:31 UTC] P3728R1124 INFO Completed 200 OK in 0ms (Views: 0.4ms | ActiveRecord: 0.0ms)
[2019-03-10 11:27:31 UTC] P3728R1124 INFO
