Dear all,
The problem is cloning or exporting a VM from a data store (NVMe SSD) for example when we clone a VM from this datastore to another one it stops when it reaches 33% of the process and throws the following error:
2023-06-20T08:30:26.271Z Wa(180) vmkwarning: cpu13:2099204)WARNING: NVMEIO:2505 command 0x45b9039cc9c0 failed: ctlr 257, queue 1, psaCmd 0x45b8c2a663c0, status 0x281, opc 0x2, cid 1041, nsid 1
2023-06-20T08:30:26.271Z Wa(180) vmkwarning: cpu13:2097690)WARNING: NVMEPSA:213 Complete vmkNvmeCmd: 0x45b9039cc9c0, vmkPsaCmd: 0x45b8c2a663c0, cmdId.initiator=0x4305f4facfc0, CmdSN: 0x19858, status: 0x281
2023-06-20T08:30:26.271Z Wa(180) vmkwarning: cpu13:2097690)WARNING: NvmeScsi: 196: SCSI opcode 0x28 (0x45b8c2a663c0) on path vmhba2:C0:T0:L0 to namespace t10.NVMe____SAMSUNG_MZVL22T0HBLB2D00B00______________C86DB411B6382500 failed with NVMe error status:
2023-06-20T08:30:26.271Z Wa(180) vmkwarning: cpu13:2097690)WARNING: 0x281 translating to SCSI error H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0 Medium Error, LBA: 111960704
2023-06-20T08:30:26.271Z Wa(180) vmkwarning: cpu13:2097690)WARNING: HPP: HppThrottleLogForDevice:1133: Cmd 0x28 (0x45b8c2a663c0, 2553369) to dev "t10.NVMe____SAMSUNG_MZVL22T0HBLB2D00B00______________C86DB411B6382500" on path "vmhba2:C0:T0:L0" Failed:
2023-06-20T08:30:26.271Z Wa(180) vmkwarning: cpu13:2097690)WARNING: HPP: HppThrottleLogForDevice:1141: Error status H:0x0 D:0x2 P:0x0 Valid sense data: 0x3 0x11 0x0. hppAction = 1
2023-06-20T08:30:26.432Z Wa(180) vmkwarning: cpu13:2097309)WARNING: NVMEIO:2505 command 0x45b9038151c0 failed: ctlr 257, queue 1, psaCmd 0x45b8c270f640, status 0x281, opc 0x2, cid 1302, nsid 1
2023-06-20T08:30:26.432Z Wa(180) vmkwarning: cpu13:2097690)WARNING: NVMEPSA:213 Complete vmkNvmeCmd: 0x45b9038151c0, vmkPsaCmd: 0x45b8c270f640, cmdId.initiator=0x4305f4facfc0, CmdSN: 0x198b3, status: 0x281
2023-06-20T08:30:33.751Z Wa(180) vmkwarning: cpu13:2097757)WARNING: NVMEIO:2505 command 0x45b903987fc0 failed: ctlr 257, queue 1, psaCmd 0x45b8c28ac600, status 0x281, opc 0x2, cid 1381, nsid 1
2023-06-20T08:30:33.751Z Wa(180) vmkwarning: cpu13:2097690)WARNING: NVMEPSA:213 Complete vmkNvmeCmd: 0x45b903987fc0, vmkPsaCmd: 0x45b8c28ac600, cmdId.initiator=0x4305f4facfc0, CmdSN: 0x264f3, status: 0x281.
It was working fine, but the problem came when our SSD for that datastore ran out of capacity, we deleted some VMs to reduce the capacity to 90% and cloned more VMs to save some space on it, while we clone or export the VMs. I got the above logs and failed 33% of the cloning process.
FYI, we have another NVMe SSD that works well. And the VMware ESXi is v8.
We need help to solve this issue ASAP.
I really appreciate any help you can provide.