proactive tests failed on VSAN 6.2

[linjianxi]

Hi,

I am building a 3 nodes homelab vsan cluster, each node with 1 disk group consisting of a 800GB Cache PCIe flash SSD and a 6.4TB Capacity PCIe flash SSD, and trying to run vsan builtin's proactive tests.

The proactive tests with heavy load, such as Stress Test and Performance Characterization tests, can be passed on thin provision, but all failed on thick provision.

Here is the log I grabbed from ESXi hosts:

2017-01-16T03:54:25.801Z cpu5:512111 opID=383bdec)World: 15544: VC opID 78df09b8-db9f-11e6-b0b4 maps to vmkernel opID 383bdec

2017-01-16T03:54:25.801Z cpu5:512111 opID=383bdec)FS3: 226: <START fbb>

2017-01-16T03:54:25.801Z cpu5:512111 opID=383bdec)261120 resources, each of size 0

2017-01-16T03:54:25.801Z cpu5:512111 opID=383bdec)Organized as 21 CGs, 64 C/CG and 200 R/C

2017-01-16T03:54:25.801Z cpu5:512111 opID=383bdec)CGsize 65536. 0th CG at 65536

2017-01-16T03:54:25.801Z cpu5:512111 opID=383bdec)FS3: 228: <END fbb>

2017-01-16T03:54:26.050Z cpu7:512111 opID=383bdec)FS3: 226: <START fdc>

2017-01-16T03:54:26.050Z cpu7:512111 opID=383bdec)130000 resources, each of size 2048

2017-01-16T03:54:26.050Z cpu7:512111 opID=383bdec)Organized as 11 CGs, 64 C/CG and 200 R/C

2017-01-16T03:54:26.050Z cpu7:512111 opID=383bdec)CGsize 26279936. 0th CG at 65536

2017-01-16T03:54:26.050Z cpu7:512111 opID=383bdec)FS3: 228: <END fdc>

2017-01-16T03:54:26.237Z cpu6:512111 opID=383bdec)FS3: 226: <START pbc>

2017-01-16T03:54:26.237Z cpu6:512111 opID=383bdec)64512 resources, each of size 4096

2017-01-16T03:54:26.237Z cpu6:512111 opID=383bdec)Organized as 63 CGs, 64 C/CG and 16 R/C

2017-01-16T03:54:26.237Z cpu6:512111 opID=383bdec)CGsize 4259840. 0th CG at 65536

2017-01-16T03:54:26.237Z cpu6:512111 opID=383bdec)FS3: 228: <END pbc>

2017-01-16T03:54:26.475Z cpu4:512111 opID=383bdec)FS3: 226: <START sbc>

2017-01-16T03:54:26.475Z cpu4:512111 opID=383bdec)32000 resources, each of size 8192

2017-01-16T03:54:26.475Z cpu4:512111 opID=383bdec)Organized as 8 CGs, 64 C/CG and 64 R/C

2017-01-16T03:54:26.475Z cpu4:512111 opID=383bdec)CGsize 33619968. 0th CG at 65536

2017-01-16T03:54:26.475Z cpu4:512111 opID=383bdec)FS3: 228: <END sbc>

2017-01-16T03:54:26.624Z cpu4:512111 opID=383bdec)FS3: 226: <START pb2>

2017-01-16T03:54:26.624Z cpu4:512111 opID=383bdec)256 resources, each of size 4096

2017-01-16T03:54:26.624Z cpu4:512111 opID=383bdec)Organized as 1 CGs, 64 C/CG and 16 R/C

2017-01-16T03:54:26.624Z cpu4:512111 opID=383bdec)CGsize 4259840. 0th CG at 65536

2017-01-16T03:54:26.624Z cpu4:512111 opID=383bdec)FS3: 228: <END pb2>

2017-01-16T03:54:26.825Z cpu5:512111 opID=383bdec)Res3: 6312: SDDir: type: 0x5, fileLength: 0x118, numBlocks: 1

2017-01-16T03:54:26.849Z cpu5:512111 opID=383bdec)Vol3: 3999: Created VMFS-5.61 with config 0x16 on vol 'f1437c58-9397-445b-c5a1-f8bc1234eb5e'

2017-01-16T03:54:26.896Z cpu6:42863)LSOM: LSOMCheckCapacity:575: Failed to create component on disk 5209b30e0493897244b98c5394763f7e. Insufficient capacity: Required 219907358720, Available 79657696011.

2017-01-16T03:54:26.896Z cpu6:42863)WARNING: LSOM: LSOMCreatePrepareDispatch:724: Component admission check failed for uuid f7437c58-6987-7417-894a-f8bc124881ee : Out of resources

2017-01-16T03:54:26.896Z cpu6:42863)LSOM: LSOMCheckCapacity:575: Failed to create component on disk 5209b30e0493897244b98c5394763f7e. Insufficient capacity: Required 219907358720, Available 79657696011.

2017-01-16T03:54:26.896Z cpu6:42863)WARNING: LSOM: LSOMCreatePrepareDispatch:724: Component admission check failed for uuid f7437c58-8df2-7617-d88d-f8bc124881ee : Out of resources

2017-01-16T03:54:26.896Z cpu6:42863)LSOM: LSOMCheckCapacity:575: Failed to create component on disk 5209b30e0493897244b98c5394763f7e. Insufficient capacity: Required 219907358720, Available 79657696011.

2017-01-16T03:54:26.896Z cpu6:42863)WARNING: LSOM: LSOMCreatePrepareDispatch:724: Component admission check failed for uuid f7437c58-b396-7817-7f65-f8bc124881ee : Out of resources

2017-01-16T03:54:27.232Z cpu2:490077 opID=a4d0f2cb)World: 15544: VC opID 78df09b8-db9f-11e6-b0b8 maps to vmkernel opID a4d0f2cb

2017-01-16T03:54:27.232Z cpu2:490077 opID=a4d0f2cb)Vol3: 956: Unable to register file system f1437c58-9397-445b-c5a1-f8bc1234eb5e for APD timeout notifications: Inappropriate ioctl for device

2017-01-16T03:54:27.344Z cpu0:490075 opID=a4d0f2cb)World: 15544: VC opID 78df09b8-db9f-11e6-b0b8 maps to vmkernel opID a4d0f2cb

2017-01-16T03:54:27.344Z cpu0:490075 opID=a4d0f2cb)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-6843-1455-4d53-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.378Z cpu2:35383 opID=2cfe505b)World: 15544: VC opID 78df09b8-db9f-11e6-b0c7 maps to vmkernel opID 2cfe505b

2017-01-16T03:54:27.378Z cpu2:35383 opID=2cfe505b)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-a57b-9255-8394-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.408Z cpu4:34761 opID=e61e88c4)World: 15544: VC opID 78df09b8-db9f-11e6-b0dd maps to vmkernel opID e61e88c4

2017-01-16T03:54:27.408Z cpu4:34761 opID=e61e88c4)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-47f1-1b56-d0d3-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.437Z cpu2:490079 opID=e3d422af)World: 15544: VC opID 78df09b8-db9f-11e6-b0e0 maps to vmkernel opID e3d422af

2017-01-16T03:54:27.437Z cpu2:490079 opID=e3d422af)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-138f-1c56-fe1e-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.467Z cpu1:36491 opID=7796b56c)World: 15544: VC opID 78df09b8-db9f-11e6-b0d1 maps to vmkernel opID 7796b56c

2017-01-16T03:54:27.467Z cpu1:36491 opID=7796b56c)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-9519-1d56-50d3-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.503Z cpu7:490080 opID=fef51cd2)World: 15544: VC opID 78df09b8-db9f-11e6-b0c1 maps to vmkernel opID fef51cd2

2017-01-16T03:54:27.503Z cpu7:490080 opID=fef51cd2)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-67b5-1d56-c964-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.555Z cpu2:35393 opID=ef88d42d)World: 15544: VC opID 78df09b8-db9f-11e6-b0be maps to vmkernel opID ef88d42d

2017-01-16T03:54:27.555Z cpu2:35393 opID=ef88d42d)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-f5b9-1e56-35e3-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.592Z cpu0:490077 opID=a02cf2e1)World: 15544: VC opID 78df09b8-db9f-11e6-b0d7 maps to vmkernel opID a02cf2e1

2017-01-16T03:54:27.592Z cpu0:490077 opID=a02cf2e1)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-c452-2156-e0ba-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.629Z cpu2:490076 opID=2baa3262)World: 15544: VC opID 78df09b8-db9f-11e6-b0da maps to vmkernel opID 2baa3262

2017-01-16T03:54:27.629Z cpu2:490076 opID=2baa3262)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-87c2-4c56-cacd-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.668Z cpu4:36489 opID=8fcd32a1)World: 15544: VC opID 78df09b8-db9f-11e6-b0ca maps to vmkernel opID 8fcd32a1

2017-01-16T03:54:27.668Z cpu4:36489 opID=8fcd32a1)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-842d-6356-eea0-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.707Z cpu4:35384 opID=257a3545)World: 15544: VC opID 78df09b8-db9f-11e6-b0cd maps to vmkernel opID 257a3545

2017-01-16T03:54:27.707Z cpu4:35384 opID=257a3545)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-1f2a-2357-4fe7-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.741Z cpu7:35390 opID=4038b883)World: 15544: VC opID 78df09b8-db9f-11e6-b0c4 maps to vmkernel opID 4038b883

2017-01-16T03:54:27.741Z cpu7:35390 opID=4038b883)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-7c71-2257-5a8b-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.778Z cpu7:490073 opID=918908a6)World: 15544: VC opID 78df09b8-db9f-11e6-b0bb maps to vmkernel opID 918908a6

2017-01-16T03:54:27.778Z cpu7:490073 opID=918908a6)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-724d-f058-82c4-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.815Z cpu7:490074 opID=72e5ea37)World: 15544: VC opID 78df09b8-db9f-11e6-b0d4 maps to vmkernel opID 72e5ea37

2017-01-16T03:54:27.815Z cpu7:490074 opID=72e5ea37)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-5bac-5d5a-61b1-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.850Z cpu0:36492 opID=808457ee)World: 15544: VC opID 78df09b8-db9f-11e6-b0e4 maps to vmkernel opID 808457ee

2017-01-16T03:54:27.850Z cpu0:36492 opID=808457ee)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-8176-e25d-950d-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.886Z cpu7:36490 opID=7048f035)World: 15544: VC opID 78df09b8-db9f-11e6-b0e8 maps to vmkernel opID 7048f035

2017-01-16T03:54:27.886Z cpu7:36490 opID=7048f035)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-736d-1264-77bc-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.922Z cpu1:34990 opID=5dcd8924)World: 15544: VC opID 78df09b8-db9f-11e6-b0ec maps to vmkernel opID 5dcd8924

2017-01-16T03:54:27.922Z cpu1:34990 opID=5dcd8924)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-fb94-a66a-4be8-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.960Z cpu5:490072 opID=ae54611b)World: 15544: VC opID 78df09b8-db9f-11e6-b0f4 maps to vmkernel opID ae54611b

2017-01-16T03:54:27.960Z cpu5:490072 opID=ae54611b)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-6676-2775-e23d-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:27.996Z cpu5:34761 opID=c3885ddc)World: 15544: VC opID 78df09b8-db9f-11e6-b0f0 maps to vmkernel opID c3885ddc

2017-01-16T03:54:27.996Z cpu5:34761 opID=c3885ddc)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-fb08-9678-8aa3-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:28.034Z cpu2:35383 opID=18895211)World: 15544: VC opID 78df09b8-db9f-11e6-b0f8 maps to vmkernel opID 18895211

2017-01-16T03:54:28.034Z cpu2:35383 opID=18895211)WARNING: VSAN: VsanIoctlCtrlNode:1979: f3437c58-8aa1-af7f-cb3c-f8bc1234eb5e: RPC to DOM returned: Underlying device has no free space

2017-01-16T03:54:28.862Z cpu2:33312)DOM: DOMCmmdsUpdateObjectDeletedBuildEntryCB:2918: composite 81437c58-bc01-decc-9b45-b083fede6ce7 -- component 81437c58-6c02-45cd-dbc0-b083fede6ce7 deleted 112 sec ago. OK.

2017-01-16T03:54:28.935Z cpu2:32901)DOM: DOMCmmdsUpdateObjectDeletedBuildEntryCB:2918: composite 81437c58-bc01-decc-9b45-b083fede6ce7 -- component 81437c58-6c02-45cd-dbc0-b083fede6ce7 deleted 112 sec ago. OK.

2017-01-16T03:54:37.706Z cpu1:33190)NMP: nmp_ResetDeviceLogThrottling:3349: last error status from device naa.6c81f660eabe9d001f925aa71a7e65e1 repeated 2 times

My question is:

Does this log literally mean that it was running out of space when creating vmdk for testing?

Does increasing the cache or capacity help in order to pass all the stress tests with thick provision?

When, for example, "Stress test" specifies that "Uses 1TB of space per host", how can I calculate the overall free space requirement for thick provision?

Or Am I missing something about vsan configuration that proactive tests do require?

below is my raid1-thick policy:

        Number of failures to tolerate      1

        Flash read cache reservation      0

        StripeWidth                               1

        Object space reservation           100

        Disable object checksum           No

        IOPS limit for object                  0

I am new to virtual san. Hope someone can guide me through. Thanks.

[elerium]

Based on the log output, it does look like the test failure is from insufficient disk space. In terms of how much is required to run on thick provisioning during testing, I'm not sure. When it says 1TB required per host, i take that to mean 2TB of raw data (because of FTT=1 and having 2 copies) so 6TB total on the datastore.

How much free space does your datastore report? I think it's quite possible that the proactive test wasn't built in a way to consider thick provisioning.

[linjianxi]

Hi, elerium

The cluster reports that the capacity is 21.27TB, free space left is 20.98TB, which should be big enough to hold on disks consumption based on the requirement.

[elerium]

Hrm unfortunately I don't know, it may be a bug with the proactive test