We've hit a problem along our virtualization journey with a virtual printer server, which we've lovingly dubbed "the print server from Hell" . We are looking at moving from physical printer servers, each holding around 400 printers queues and running Windows 2003 server to virtual printer servers.
We therefore commissioned a new printer server VM with 2 processors and 4Gb of RAM.. There currently just over 200 printer queues on this VM - all corresponding to various Xerox printer models. We have loaded the latest Xerox printer drivers.
We are having problems with the stability of the printer spooler. While most of the jobs do print (the server averages about 10 to 12 thousand prints per day during the working week) a significant number of jobs (about 50-60 per day) just vanish into thin air after spooling and are never printed. These vanishing jobs are accompanied by 6161 print errors in the system log. The printer spooler service also tends to terminate abnormally once per day during the working week with a 7034 error in the system log. If the spooler service is restarted then things hum along merrily for another 24 hours or so. This behavior (vanishing print jobs and daily spooler crashes) does not occur during weekends when the print volumes drop to around 2000 prints per day.
I thought initially this was a performance issue. I've run performance monitor logs against the virtual printer server and also checked the VC's own performance figures for the VM, and the processor load averages around 50-55 % over the two processors with no sustained peaks. In addition the memory usage rarely goes above 1/4 of its 4Gb allocation and there are no obvious processor or disk queues. Curiously the problems tend to occur in the afternoon (after close to 24 hours of continuous running) and not during the period of peak usage in the morning.
I am planning to raise a support call with Microsoft, but I fear that I may be kicked into touch once i disclose the fact that the printer server is virtual. We do get the odd 7034 spooler failure on the physical printer servers but only about once ever 3 months rather than daily. Similarly there are about 3 or 4 instances of 6161 print errors on the physical servers daily rather than 50 to 60 on the virtual server, which currently has half the number of queues of the physical servers.
Does anyone run virtual Windows printer servers with around 200 (or more) printer queues? If so have you come across similar instability with the spooler service?
I can try upping the processors to 4, but the performance stats do not seem to indicate processor bottlenecks.
Our environment is VMware 3.5 U4 with 16 core 2.0 Ghz servers. Our storage layer is based on a Netapp Filers presenting NFS volumes. The physical printer servers are dual processor Xeon 2.6Ghz servers with 2Gb of RAM each.
Any advice on this much appreciated.
In the company I work for, we have some W2K3 VMs with Windows Spools, over ESX U3.
The largest one has over 1000 printers, and it's got 1 vCPU and 1 GB RAM. No performance issues. We migrated the printers with PRINTMIG (new VM, no P2V's for this).
So, I think this is not an issue for VMware Forums. Like you say, raise a support call to Microsoft, BUT DO IT WITHOUT FEAR.
They HAVE TO give you support, since they added VMware ESX 3.5 U2 or higher to their Server Virtualization Validation Program (SVVP). Check this links:
The printer spooler is embedded in Windows, so the first link is enough for you if Microsoft doesn't help you.
If you have problems in the future with Microsoft Software, here you have specific Support Policies for their applications:
Thanks for the very helpful reply. I was seeking to ascertain whether other VMware community members had encountered similar problems when running virtual printer servers with large amounts of printers and/or printing throughput.
We did the same thing that you did - i.e. create a new VM and Migrate printers over using PRINTMIG. It is interesting that you are able to run on one virtual CPU with the number of printers you have - we tried this initially and quickly had to move to 2 vcpus as the cpu maxxed at 100 percent continuously!
My concern wrt to Microsoft is that i'd read a policy statement (admittedly a while ago) that said that they reserved the option to require the customer to reproduce the problem on physical hardware. My problem is that we haven't seen the problem on our physical printer servers
So given your reply, my take is that the root cause could be
1) A problem with Microsoft printing subsystem
2) A problem with the Xerox print drivers (I'm gravitating towards this)
3) A problem with our Vmware infrastructure (storage layer, network layer etc)
Our Netapp team have checked their logs and have assured me that no issues are present around the times that the printer server loses jobs. My perfmon stats for the VM appears to support this - no disk queues reported at the time.
My plan of action is to
1) temporarily move the VM off shared storage to the local drive of one of the Vmware physical servers. if the problem goes away then the root cause is our storage layer.
2) Raise faults with Microsoft and Xerox (now done).
3) Get stats from our Networking people team on the network cards (gigabit and bonded) to the storage layer and the internal network.
Thanks again for the reply - I'll post an update once (hopefully) this is resolved.
OK, I see you have lots of ways for troubleshooting )
Regarding the "reproduce the issue in HW" statement, yes, you read it a while ago:D
Before SVVP, Microsoft could ask you for reproducing the issue in HW, before taking any action (except for Premier contracts, in that case it was "reassonable effort" first, not knowing what is that exactly)
Now, they have to investigate FIRST, even talking with VMware, and if they can't fix it, they still may ask for reproducing it in HW. from the first link:
This support will include coordinating with the vendor to jointly investigate support issues. As part of the investigation, Microsoft may still require the issue to be reproduced independently from the non-Microsoft hardware virtualization software. Where issues are confirmed to be unrelated to the non-Microsoft hardware virtualization software, Microsoft will support its software in a manner that is consistent with support provided when that software is not running together with non-Microsoft hardware virtualization software.