System Error 51 in VM?

[FAL-Moray]

hi all,

Not sure if this is corect forum but I am sure that someone will put me right.

I have a Windows Server 2003 SP2 running on a SAN attached to my very small ESX3.5 server supporting six virtual servers. Two of the servers have been moved onto the SAN while the the other remain on local storage of the ESX host.

One of these servers is configured as follows: Server 2003 SP2 32bit, 2048mg RAM with 2 50GB SCSI HDDs confgiured as basic as opposed to dynamic disks inside Windows.The server host several small SourceSafe databases as well as some new software being used by our QA department. This software contains a large number of small files stored in its own local database format. I am unsure if this softwar is the culprit in my scenario.

What I am seeing is a small number of System error 51s being generated in the system event log on this server at random times during the day. Googling this error against VMs comes back with questions and answers from 2005-2008 about virtualising SQL servers with a high amount of disk I/O. I am wondering if I am seeing the same thing, if this is really an issue or if I can make some further configuration changes to help solve the problem?

regards,

Gordon

[wila]

Hi,

The forum location should be fine.

As for your windows system event log ID 51 error, have a look here and try to decode the data block.

http://support.microsoft.com/kb/244780

I would probably try to increase the memory of the server, depending on some more research. Is it possible the VM is swapping at that moment?

You might also want to inspect the vmware.log file for some hints at that same time as when the error 51 gets reported.

--

Wil

[FAL-Moray]

hi Wil,

Nothing inthe VM logs since 25/2 and the error last occured this morning at 0250. At first I thought the error had to do with the required services for this application as it occurred immediately after starting the Windows services but there is no corresponding entry for the night before or the night before that.

Here is the details for the system error:

0000: 04 00 68 00 01 00 ba 00   ..h...º.

0008: 00 00 00 00 33 00 04 80   ....3..€

0010: 2d 01 00 00 0e 00 00 c0   -......À

0018: 00 00 00 00 00 00 00 00   ........

0020: 00 ae 1e 4b 08 00 00 00   .®.K....

0028: 32 ff bf 08 00 00 00 00   2ÿ¿.....

0030: ff ff ff ff 01 00 00 00   ÿÿÿÿ....

0038: 40 00 00 84 02 00 02 00   @..„....

0040: 31 20 0a 12 82 07 60 40   1 ..‚.`@

0048: 00 00 01 00 3c 00 00 00   ....<...

0050: 00 00 00 00 80 16 a2 89   ....&#128;.¢&#137;

0058: 00 00 00 00 f8 31 74 89   ....ø1t&#137;

0060: 18 61 d4 89 57 8f 25 04   .aÔ&#137;W&#143;%.

0068: 2a 00 04 25 8f 57 00 00   *..%&#143;W..

0070: 80 00 00 00 00 00 00 00   &#128;.......

0078: 70 00 05 00 00 00 00 40   p......@

0080: 00 00 00 00 25 83 10 80   ....%&#131;.&#128;

0088: 00 00 00 00 00 00 00 00   ........

All I can make it is that it is a write (04). The rest is greek to me....

regards,

Gordon

I am wondering about any issues with moving the local databases to our VM SQL2000 server. Is I/O on a SQL VM still an issue?

[wila]

Hi,

IO on a VM should not be an issue, not sure what you mean by "is it still an issue".

But MS SQL server 2000 ? Wow.. that's some old SQL server.. can't you update that to something more recent?

Even SQL 2005 is a bit old, but at least it is from this decade. :smileysilly:

As far as decoding goes.

NTSTATUS error is 33 00 04 80 > Error 855639168, not telling me anything

0x20

8

Byte offset to bad sector, if any

0020: 00 ae 1e 4b 08 00 00 00

Hmm... that's certainly any

I've tried decoding the scsi block, but it starts with code 40 and that's not defined in the scsi.h file.. :s

Unless it is reported here as decimal.. because if I change it into hexadecimal it would be  0x28 which is

SCSISTAT_QUEUE_FULL but that would be guessing I suppose hmm...

Microsoft logging always helpful.
--
Wil

[FAL-Moray]

hi Wil,

Back in the dim dark days of 2005 and earlier when VM was still sort of coming online there was a call from (I think M$ but don't quote me on this...) about virualising SQL servers. I know that around this time M$ was not supporting SQL on a VM.

The argument on some sites and papers I read was along the line of any SQL db that was highly utilised (and thus had lots of disk I/O) was not a good candidate for virtualisation. Virutalising these servers could lead to performance problems etc. About this time we virtualised our SQL server and have not seen any error or had any issues with it. Granted it is fairly lightly utilised but it does host our SharePoint dbs so at time there has to be some I/O going on there.

I was wondering if you had heard of any other issues with SQL on VM. I did pull down the VMware PDF on SQL Server and am going through that. This paper seems to imply that you should not have any issues.

Yes we are using SQL2000 SP4 - legacy reasons there I'm afraid - stuck with it for awhile. The app on this server does support SQL and I am wondering about the possibility of installing SQL2005 and using this as it db rather than its native format. I am afraid if I do that I willjust be transferring the problem somewhere else (from the app to SQL2005) and still see the same issues.

G

[wila · Accepted solution]

Hi,

In regards to the remark from MS and VM's not performant enough, it is interesting to note that MS only has that type of comments until their own product (Hyper-V et al.) are powerful enough to support MS SQL properly in a virtual environment. I would chalk it up to MS FUD.

Anyways, I guess that's mud slinging and not very relevant for the discussion.

For  a more relevant counter argument, have a look at this article from VMware performance group:

http://blogs.vmware.com/performance/2008/05/100000-io-opera.html

Or for a non vmware report have a look at this study from Brocade:

http://www.brocade.com/downloads/documents/white_papers/WP_VMWareSQLServerBenchmark-00.pdf

By using virtualisation and external storage arrays you can scale out better as by just assigning the whole machine to the SQL Server and assuming that you get the same performance by scaling up. So by applying the right configuration you can actually get more performance from your hardware in virtual machines as by using the physical hardware.

Regarding SQL2000 vs SQL2005. I don't know your application, but SQL 2005 is if I remember correctly -it's a while ago now- backwards compatible with SQL 2000 for pretty much everything. I'm a database developer myself, but use pretty much any SQL server, not just MS's products.

The reason I mentioned it is that I don't think that even Microsoft still supports SQL 2000, so if you can upgrade it, it would be nice. As for resolving the system error 51 report, I also don't think it will resolve that, but don't know really. I can't test/verify it here as I don't have SQL 2000 setups anymore.

Now to get back to your event syslog error. Personally I doubt it is a serious error and just a report of a cache miss, but at the moment I'm having trouble to back up that statement. If it is an error that happens daily then you might be able to setup some performance counters to verify if the host is swapping at the moment of such a report.

If I look on my current VM's system log, I also see an error 51 error, but it's for my floppy.

An error was detected on device \Device\Floppy1 during a paging operation.

I take it that your report is for the physical harddisk? Does it also have the "paging operation" part?

You only have one harddisk setup with this VM?

--

Wil

[FAL-Moray]

hi Wil,

Thank you for the comments and links. We have been using a VM SQL Server for several years and have seen no issues. I haven't even had to tweak it.

Yes this error is occurring on one of the HDDs - An error was detected on device \Device\Harddisk1 during a paging operation.

The VM has 2 SCSI HDDs configured in Windows as basic disks. The reason I mention this is that when the machine was virtualsed HDD1 went in as a dynamic disk. This in its own caused me no issues until the user of this specific app tried to delete 5000+ records from the native DB. According to him the "app hung and then refused to complete the operation". When I investigated I found the dynamic HDD marked as suspect by Windows and seversl error 51s in the system log.

We recovered the system by ghosting onto a new basic disk (lost no data as far as I can tell). This app no seems happy from what the users are telling me but I am still seeing this occasional error 51 cropping up.

I can through another 1GB against the VM (32bit version of Server 2003) and then adjust the page file and see if this makes any difference. I guess what I am concerened about is having the users really start winding up this tracking app with a lots of data and then have the entire VM fail.

regards,

Gord

[FAL-Moray]

hi Wil,

I am going to mark this as corrct and move on. I cannot reliably get the error to reproduce. It appears as if we are going to try and move the VM to 64bit Server 2008 and SQL 2008 for the apps db.

Thank you for the input and support info.

regards,

Gordon

[wila]

Oh that's a big improvement. I've found that Windows 2008 R2 x64 runs very smoothly as a guest and is normally more performant as its 2003 alternative.

For the record I've not updated to SP1 yet. Usually I wait a bit before going into a possibly troublesome update and wait and see here and run it for a while in a testlab.

SQL2008 is a big step up from SQL2000. There are many new features there too.

Sounds like a good plan and with virtualisation you can easily make a testrun of the platform before switching over to production with the new system.

--

Wil