WAN Performance - PCoIP/Zero Client - Any Ideas?

[TheDeepRough]

We have started a rollout of View 4.0.1. Mostly we have financial analysts using this tool. They are working in Excel, Word, Outlook and PDF files. Occasional web browsing. Very little, if any video.

SETUP

HQ (ESXi 4 servers hosted here, view 4.0.1, several clients)

- 15 PCOIP Clients (Mix of Samsung NC240 and Thick Clients)

- 3 Windows 7, 12 Windows XP SP3

- 20 Mbps connection to MPLS

Branch 1

- 2 PCOIP Clients (Wyse P20, Dual 19" Monitors - 1280x1024)

- 5 Mbps connection to MPLS

- 25 ms round trip latency, very little packet loss/jitter

Branch 2

- 3 PCOIP Clients (Samsung NC240)

- 5 Mbps connection to MPLS

- 70 ms round trip latency, very little packet loss/jitter

We have followed the optimization guidelines laid out in the PCoIP Zero Client to VMware View 4 documentation from Teradici. All Zero clients are using the July 30th release of PCoIP firmware from Teradici. All bandwidth/latency numbers have been confirmed to be valid.

PERFORMANCE

HQ

- NC240 Performance is great, we even have a user with an external monitor attached, still good performance

- Only issue might be slightly degraded experience while scrolling through a PDF file quickly

Branch 1

- Acceptable performance, more noticeable degradation in large PDF files. Overall experience is acceptable.

- Active Bandwidth Limit seems to land around 2000 or 3000 kbps.

Branch 2

- Unacceptable performance. Scrolling a PDF is painfully slow. Scrolling in Outlook is jumpy. The time it takes for images to resolve (go from being highly pixelated to clear) is far too long.

- We have a Wyse P20 at this location as well, hooked up to a 1024x768 projector. This is more tolerable, but still jumpy on PDF files.

- Looking at the session stats through the PCoIP status page for the device, the units seem to end up at an Active Bandwidth Limit of 1000 kbps.

QUESTIONS

Does anyone have any experience in this 70ms latency range? How has your end user experience been? Was bandwidth an issue? Are you using a WAN Accelerator?

Prior to making this jump, we had done some trials internally, attempted to simulate the latency and bandwidth requirements for this second branch using tools like WanEMU. Performance under those conditions seemed acceptable. But in the real world, it seems to be different.

I am just curious to see if anyone else has implemented a similar infrastructure, and if it is performing well/meets user expectations. I am hoping it is just something in our configuration!

Thanks for any feedback!

[Wonderdog]

Firstly, I would recommend trying to connect via RDP rather than PCOIP and see if the user experience at your problematic site is improved any.

Also, check with your MPLS provider and/or whomever manages your MPLS/WAN routers to see if they are applying QoS to the link - as this may also be causeing the issue.

It sounds like packet loss/mangling rather than latency are your issue (70ms is a-okay, I've had useable if laggy full screen 3d graphics tests running at 600ms from Japan>UK before) - remember than PCOIP is UDP based, which should be taken into consideration if your usng any kind of prioritisation/wan optimisation tools or appliances (UDP tends to be handled at a lower priority than TCP traffic by default by many WAN optimisation tools).

Best of luck resolving your issue!

#Phil

[TheDeepRough]

We had one of the users at Branch 2 switch over to RDP from PCoIP. He says the experience within Excel/Outlook is improved as he does not need to 'wait' for the screen to become less blurry all the time. However, PDF files and Internet browsing are worse (as expected).

There are no WAN optimization pieces in place at this point. There is no QoS on this particular line.

Do you think 70ms / 5mbps (for 5 users) should provide a user experience that the end user would enjoy? I am worried I will have a mutiny on my hands if anyone else gets moved over to this solution

[Wonderdog]

70ms roundtrip + 1mbps per user should be more than adequete for a good PCOIP experience. It is interesting that RDP improves the situation, that would certainly point to packet mangling. due to PCOIP being UDP based, it tolerates a bit of packet loss and jitter, but go above a certain amount and it really starts to drop off fast. RDP on the other hand will still appear slightly unresponsive on big pages updates etc, but should give better perceptible screen quality due to the error correction it gains from being TCP based.

A sensible step to eliminate the line as the problem would be to to some packet loss and throughput tests - Is there any other activity on the circuit with the exception of your View traffic? Do you have the capability to monitor traffic on the connection somehow (perhaps the free version of Netflow analyser?). As a really simple test, start a continuous or high count ping from your remote site to head office or visa-versa to get an idea of how many lost or seriously delayed (200ms+ spikes) packets show up.

If your users are accessing the same desktop pools/configs etc as your other remote site and head office, it is unlikely your server side config is at fault (though they could be a factor, so dont totally discount it).

Also, you may want to try connecting from your remote site using the View software client from a regular PC, and compare the performance to that of your thin clients at the site.

Best of luck.

#Phil

EDIT - apologies, I notice you have tested packetloss/jitter, so please ignore that suggestion. Definately try the software View client if possible however...

[TheDeepRough]

Interesting.

Yes, within the HQ location, performance is great. At the 25ms latency location, performance is good as well, not perfect, but I would be comfortable working there day to day.

When I was last at the higher latency site, I was connected via the view client (off a windows 7 laptop), and the performance seemed to be improved over the Samsung NC240. I simply attributed that to the much lower resolution of my 13" laptop (1280x800) vs the 24" NC240 (1920x1080). But could it point to some other problem?

There is no other activity on this circuit, it is just the view sessions. There are no servers/workstations in that office, just the router/switches and zero clients right now. So not much one can do to run any sort of analysis on the connection. I can try a long running ping test from the router -> hq to see if there are any spikes. Though, when I have done it short term in the past, it has generally stayed pretty rock solid at 70ms.

Thanks for your help!

[Wonderdog]

It seems too much of a co-incidence (with my cynical nature!) that your problem site only has the NC240's, while the other site has the WYSE units and is not showing the same issues (the difference in latency is not sigificant for your described software packages). As the NC240's in use at your HQ are local to the View servers we cannot assume they will not exhibit the same symptoms.

Its highly unlikely the jump from 1280x800 to 1920x1080 resolution would be the sole reason for such a significant drop in performance (PCOIP is designed to scale extremely well - multi monitors etc, its one of its strengths). It will use more bandwidth and need a little bit more processing grunt to render the higher resolution at the client end, but for largely stationary stuff it shouldnt be any issue for your NC240's at all - your not running high motion video or 3d apps after all.

I would compare a software client at 1920x1080 to the thin client and see how that fares - alternatively ship a spare wyse client to Site2 for comparison.

In the meantime, try rolling back the Firmware on one of the NC240's by a release (or two) and see if it alters the user experience any. If you can de-activate the teradici hardware acceleration feature on the NC240 you might want to try that as well for testing purposes.

EDIT:- I notice you are using View 4.0.1 - are you running the latest version of VMware tools and the View Agent on your Terminal server / VDI's? You may wish to uninstall/reinstall the latest releases of both to be on the safe side.

[TheDeepRough]

We had the December 2009 firmware (3.0.?) on there initially, and in an attempt to improve the performance, Teradici suggested we upgrade to the most recent one (3.2).

We do actually have two Wyse P20s in that office, and they are attached to projectors in conference rooms. They are displaying at 1024x768 and do offer up a better user experience. I would not say it is perfect or as good as it is over a 25ms connection, but it's something I could live with. Again, here, I just made the assumption that it was the lower resolution. What I could try to do is have one of the P20's hooked up to the NC240 (use it as an external monitor) and see how that goes. That would answer questions about the resolution.

We should have the latest versions of the view connection server and agent on vms installed.

Attached is a copy of a pcoip_server log from a user session from this morning. It does not make a whole lot of sense to me, other than seeing that it seems to keep bumping the bandwidth limit down. There were only two users in the office today, so there should not have been much, if any contention on the link. Anyone know much about these files?

[TheDeepRough]

Just as an update here. I received some interesting feedback from the Branch 2 office (70ms latency).

1) Some of them have switched over to RDP on the Samsung monitors and are claiming the performance is pretty good for word/excel type work. They would rather use RDP than PCoIP. PCoIP is just too slow to 'clear up' the screen, that the progressive loads of the screen are taking too long, ie. it is very 'blurry' then gets slightly better, then slightly better. They find this more annoying than RDP that just blocks for a little bit then updates the entire screen.

2) Had a chance to do some UDP testing on the MPLS using iperf:

UDP, 2000 kbps, 120 seconds:

29/20410 packets lost (0.14%)

7.520 ms of jitter

UDP, 4000 kbps, 120 seconds:

134/40818 packets lost (0.33%)

4.745 ms of jitter

UDP, 250kbps, 120 seconds:

11/2553 packets lost (0.43%)

0.705 ms of jitter

That test was done end to end from a client machine attached to the network at the branch office, direct to a vm running xp sp3 and the view agent, along with the iperf server.

Seems like the network is okay? Is that level of jitter/packet loss acceptable?

3) Internet vs MPLS - We had a user with a laptop connect with the open view client through our view security server on the internet (internet connection in branch location -> public security server in HQ). Then connect with the same client over the MPLS to the HQ. Both cases, the connection is RDP. Performance was noticeably better over the internet connection. Which makes little sense to me, why would it be better over the Internet tunnelled through the view security server? Granted this is all 'subjective' testing, but user experience is king here.

4) Samsung as a Monitor - had someone attach a Wyse P20 to a Samsung NC240 (use the samsung as a monitor for the P20), so the resolution was higher. Performance was no different than the Samsung NC240 by itself. So it seems the Samsung may not be at fault here?

[PCoIPinsider]

Folks,

Teradici has published a remote access configuration and optimization

guide for connecting VIEW4 virtual desktops to zero clients using PCoIP

protocol.

This publication describes optimal GPO settings for image quality

(including min. initial quality and max quality) in addition to

bandwidth limits (floor, ceiling and frame rate) for those relatively

few use cases in which the default GPO settings are not providing

optimum user experience. I'd expect this particular case in which a

progressive build is taking too long can be improved through GPO

optimizations.

Please visit the Teradici web site (or contact our support team

support@teradici.com) to get the document. Also, please be reminded that

our Systems Engineers are available to help you resolve these

configuration issues.

Thanks

Ian

[TheDeepRough]

This issue has been resolved.

For anyone who may be experiencing similar issues, on a similar link, our problem turned out to be the packet loss. I did not think the packet loss looked too bad, but turns out 0.5% packet loss causes some major issues with PCoIP. It turned out to be a bad switch. It was dropping packets on our HQ side. We resolved that, and ended up with 0 packet loss from the branch locations to HQ. As soon as that was resolved, users reported the experience being improved tremendously.

At Branch 1 (25 ms latency - 5mbps bandwidth), The experience is not much different than being in the HQ office itself

At Branch 2 (70ms latency - 5mbps bandwidth), while not as good as Branch 1, the experience is very good (subjective here, but people are happy to work with it on a daily basis).

I would like to thank Teradici's support team for helping out on this issue. I emailed them with the problem and send them the pcoip logs, and they quickly identified the problem as packet loss. We do have a support contract with VMware, and I had an open ticket with them on this, but they were unable to resolve the issue without having us create some controlled tests that involved logs at every stage of the session. VMware needs to pickup their game here and have their support team understand this PCoIP protocol. Given that I have no contract with Teradici, I am impressed that they'll go out of their way to help a user of their protocol!

[AdamG53]

PCoIP is certainly a great display protocol. However in some scenarios of slow remote connections (like over certain WANs) there may be issues where PCoIP doesn't function quite as well. In those cases, you can complement the VMware View deployment with Ericom Blaze, a software-based RDP acceleration and compression product that provides improved performance over WANs. Besides delivering higher frame rates and reducing screen freezes and choppiness, Blaze accelerates RDP performance by up to 10-25 times, while significantly reducing network bandwidth consumption over low-bandwidth/high latency connections.

You can use VMware View with PCoIP for your LAN and fast WAN users, and at the same time use VMware View with Blaze over RDP for your slow WAN users. This combined solution can provide enhanced performance in both types of environments, letting you get the best out of VMware View for your users.

Read more about Blaze and download a free evaluation at:

http://www.ericom.com/ericom_blaze.asp

Adam