Hi,
I'm facing a strange problem with HP BL460c G6 server. There is fibre channel adapters and I have no problems with them at all.
But I also have local storage (2x136Go SAS disks) which is problematic.
When I make a SCP copy file all is fine, transfer rate is quite good, and delays are fine in esxtop.
If I make a template deploy from FC SAN to local disks, problems happens.
For one template, if I choose the "Thin disk", all go fine, arround 36MB/s, for 2 SAS drives on RAID 0 and write cache enabled.
If I use thick or default, all go slow, arround 2MB/sec, and KAVG is like 1000 when DAVG is like 20.
For another template, same thing occurs, whatever I choose between thick or thin or default.
I tried storage vmotion one deployed template from local storage to FC san, and same thing happens, very high KAVG, low MBREAD/s.
(pic taken when storage vmotion from local to san)
All the guides I read about queue depth talk about FC SAN queu depth issues, not local storage.
I find very poor performances 2MB/s for 2 SAS disks, less than a CDROM ...
Also the local disk looks over saturated when doing this, because it lags a lot when trying to do something else with local disk (like browse files via ssh)
So the esxtop tells me it's an issue with vmkernel delay taking to process the commands...
I have no ideas on this one.. Any help?
edit : local disk controller is hp p410i
normally write is very bad on local disks if there is no raid card with cache .. you might want to check if you have that ..
There is write cache, SAS disks are on a controller board HP p410i that have enable / disable write cache.
Anyway, with or without cache, 2xSAS disks in RAID0 should perform more than 2MB/s.
And btw, if you did read my posts, you would notice that I said some disk operations works very well at good speeds.
p410i does not have any cache and battery/flash backed write cache comes as optional .. http://h18004.www1.hp.com/products/servers/proliantstorage/arraycontrollers/index.html
Please, read
Anyway, with or without cache, 2xSAS disks in RAID0 should perform more than 2MB/s.
And btw, if you did read my posts, you would notice that I said some disk operations works very well at good speeds.
36MB/s is hardly good speed. Taking into account that ESX zeros space as it writes, you should be seeing 200+MB/s in these operations since the zeros will never make it to the disk surface.
As said above it sounds very much that write cache policy is write-through for this volume. The actual transfer rate in that scenario can be roughly calculated by write size * spindle speed / 60.
I'd attach a new 8GB thick provisioned VMDK on the local storage to an existing Windows VM and use IOMeter to test 32K sequential read then 32K sequential write with 16 outstanding IOs. If it is caching related clearly we would expect to see much higher speeds in read than write.
Yes it's not a good speed, but way better than 2MB and the KAVG is close to 0.
another exemple, doing a cp of a flat vmdk from the local datastore to the same local datastore goes like this :
No KAVG.
Poor perfs, but this would be related to the lack of cache u mentioned maybe. But it's "acceptable" and don't make the disk completly stressed.
Why storage vmotion and deploy template goes like this it's weird.
Could it be because local storage is super slow compared to FC san so the SAN is like "spaming" the local storage so much that the speed drops significantly ?
Looking at the 2 screenshots what's weird is there is no QUEUE when doing local to local cp, but high queue when doing FC -> local or local -> FC
edit :
Here is the local -> FC SAN cp of vmdk screen :
And here FC SAN cp of vmdk -> local storage :
Has an alternate RAID level been tested (I recall an instance in the past where RAID-0 gave very poor throughput).
Going back to your first post, what looks interesting there is that it is the READ performance of the local storage that is rubbish - have I read that correctly?
I updated my previous post with pictures.
This p410i controller offers only RAID 0 or RAID1+0 (don't ask me how HP makes a 1+0 raid with 2 disks ...)
I tested with both, got 5MB improvment in RAID0 so I stick with RAID0 for the tests.
No I don't think it's read related as my original problem is deploying a template from FC SAN (read) to local storage (write)
OK, I'm a bit confused here as in the opening post "pic taken when storage vmotion from local to san" indicates read speed of under 2MB/s.
Can you test with IO Meter and post some results?
My english don't help maybe I guess
Yes I was planing on runing intensive disk i/o tests using various software, in fact I noticed this slow down accidentaly.
We never use local storage, and I wanted to make a full chart with local storage vs old fc san vs new fc san using various paramters as queuedepth etc .
Anyway, it's insane that thing like cp works as it should, and storage vmotion / template deploy, etc are so buggy.
Way better performances when powering on a VM ^^
And when creating a snapshot (without quiesce without memory) :
But performs ok when deleting the snapshot (commit) :
hdparm on test linux show good performances :
Ok I figured it out, maybe ...
The log is full of these things :
May 25 12:35:54 vmkernel: 1:00:40:12.476 cpu2:4471)WARNING: NMP: nmpDeviceAttemptFailover: Retry world failover device "naa.600508b1001cdef4deaedc3bbeb43eb3" - issuing command 0x4102bf20e040
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmpCompleteRetryForPath: Retry world recovered device "naa.600508b1001cdef4deaedc3bbeb43eb3"
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)scsi_cmd_alloc returned NULL!
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027eb36440) to NMP device "naa.600508b1001cdef4deaedc3bbeb43eb3" failed on physical path "vmhba2:C0:T0:L0" H:0x1 D:0x0 P:0x0 Possible sense data:
May 25 12:36:02 0 0x0 0x0.
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)WARNING: NMP: nmp_DeviceRetryCommand: Device "naa.600508b1001cdef4deaedc3bbeb43eb3": awaiting fast path state update for failover with I/O blocked. No prior reservation exists on the device.
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027efa8840) to NMP device "naa.600508b1001cdef4deaedc3bbeb43eb3" failed on physical path "vmhba2:C0:T0:L0" H:0x1 D:0x0 P:0x0 Possible sense data:
May 25 12:36:02 0 0x0 0x0.
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4102bf9ea240) to NMP device "naa.600508b1001cdef4deaedc3bbeb43eb3" failed on physical path "vmhba2:C0:T0:L0" H:0x1 D:0x0 P:0x0 Possible sense data:
May 25 12:36:02 0 0x0 0x0.
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027eaee740) to NMP device "naa.600508b1001cdef4deaedc3bbeb43eb3" failed on physical path "vmhba2:C0:T0:L0" H:0x1 D:0x0 P:0x0 Possible sense data:
May 25 12:36:02 0 0x0 0x0.
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x4102be9d6240) to NMP device "naa.600508b1001cdef4deaedc3bbeb43eb3" failed on physical path "vmhba2:C0:T0:L0" H:0x1 D:0x0 P:0x0 Possible sense data:
May 25 12:36:02 0 0x0 0x0.
May 25 12:35:54 vmkernel: 1:00:40:12.486 cpu3:4099)NMP: nmp_CompleteCommandForPath: Command 0x2a (0x41027efaa540) to NMP device "naa.600508b1001cdef4deaedc3bbeb43eb3" failed on physical path "vmhba2:C0:T0:L0" H:0x1 D:0x0 P:0x0 Possible sense data:
May 25 12:36:02 0 0x0 0x0.
Interresting here is the Host status :
H:0x1 D:0x0 P:0x0
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=289902
http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&externalId=1003433
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=102903...
->host status :
SG_ERR_DID_NO_CONNECT [0x01] Couldn't connect before timeout period
0/1 0x0 0x0 0x0 - DID_NO_CONNECT
When the fabric returns DID_NO_CONNECT status, the ESX host detected that a target is no longer present. The DID_NO_CONNECT status is caused by a fabric switch failure, a disconnected physical cable, or a zoning change that no longer allows the ESX host to see the array.
Note: This is the only sense code that affects both Active/Active and Active/Passive arrays. The other sense codes only trigger fail over on Active/Passive arrays
Ok so I'm not on an array, but still. Only 1 path, local storage, so wth is this status here ?
The issue is not present with a older dell 2950 server. Not present on HP server when deploying from SAN to SAN. Only issue is when deploying from SAN to local disks of HP servers.
But bandwhich is not a problem, I mean when doing a cp command, things go fine, not very fast, but at least the KAVG is ok.
Only doing operations that require SCSI reservations looks like to be problematic.
Anyway I found this thread http://communities.vmware.com/thread/307924 that points to http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c01832427&lang=en&cc=us&taskI...
So I think the HP missing flash cache is causing this ultra poor scsi reservation bug... Nice work HP !
Will see if I can buy a pack, or not...