ESX 3.5U4 Upgrade and Issues Thread

[ufo8mydog]

Hi all,

Now that U4 has come out, has anyone had a chance to be a guinea pig and test it - have you had any issues, bugs, or weird stuff happen? (well, a lot of weird stuff happens with ESX. It is just a matter of degree ).

[Flicker]

ESX 3.5 u4 running fine on HP Bl2x220 blades (yes i know they are technically unsupported);

Useing HP N532m 10Gb flex10 cards at full 10Gbs speeds.

U4 fixed the problem (after hours of phone support with Vmware) with the kernal, where the n532m's were running out of heap and eventual posd. This would happen even at 1Gbs speeds under high utilization conditions.

Updated using update manager on a few test machines; stress tested using the exchange load generator for 10h. So far so good.

[John_Hoover]

VMware support just called me back, I'm on the phone with them now. They are saying that the logs show timeouts from reservation conflicts. Multipathing also seems to be messed up, as I'm only showing 1 path to the datastores that are working vs 4 paths (2 hbas, each connected to a switch that is connected to each controller in the array).

IE: Node1HBA1>FCSW1>ArrayController0port0, ArrayController1port0; Node1HBA2>FCSW2>ArrayController0port1, ArrayController1port1 = 4 paths to each LUN if they're presented to all four controller ports.

I'm going to try re-presenting the LUNs and redo the Zoning in the FC switches and see if this works any better.

[Neilly]

Well I got ESX 3.5 update 4 to install. After trying a bunch of stuff (details too boring and not really useful) it turns out it was an error with the DVD burner in the machine I used to create the install CD. But now I have a different problem - an error mounting the root filesystem on reboot. I see that there are a number of comments about this with prior versions of ESX and I will try out the various suggested solutions. Also, oddly to me, is that there is no default route setup during the install of ESX. So the server boots up but cannot be reached from my workstation which is in a different subnet. That should be easy to fix, of course being a noob I'll have to look up the correct syntax and find the correct file to modify to make the route permanent.

Cheers,

Ron

[Neilly]

Okay, so now the server is rebooting properly. The fix for this was to boot to the Troubleshooting mode and run esxcfg-boot -b.

As for the routing issue it was not the ESX server - I was misinterpreting the long delay in completing the output from the route command and in my impatience I was Ctrl-C'ing before the default route could be displayed. So it had a proper default route; and my firewall admin missed updating the subnet mask in a couple of places for the subnet that this server is in, which explains why I could not ping it from outside hosts. Thus we have two stupid user errors, one hardware error and one reasonably definate ESX error (viz the boot error). Not bad...

[Neilly]

Looks like I spoke to soon. The server is NOT rebooting properly. It still comes up with errors leading to a kernel panic. The messages on the service console are:

RAMDISK: compressed image found at block 0

crc error(6)freeing initrd memory: 9627k freed

VFS: cannot open root device "UUID=bloody long string" or 00:00

Please append a correct "root=" boot option

Kernel panic: VFS: unable to mount root fs on 00:00

Any ideas what is wrong with this puppy?

Cheers,

Ron

[Neilly]

(Takes washcloth and removes egg from face...) I spoke too soon again. The boot issue this time around was because I installed a new device driver, but didn't execute the esxcfg-boot -b command afterwards. Did that and the server boots fine again. Note to self, for any boot issues run esxcfg-boot -b first...sigh. Such a noob.

Cheers,

Ron

[MKguy]

Installed U4 on 5 ESX and an ESXi host (HP DL 380 G4 and G5) during the last couple of days with UM. I haven't experienced any real issues apart from the following glitches:

On a 2 node HA Cluster, when patching one node, the HA agent (namely the ftPerl process) on the remaining host started to eat all available processing power fof the CPU in the service console. I had to manually reconfigure HA for the host in order to get rid of it. I'm not sure whether this was related with the actual update or with HA itself. If I put a host manually in maintenance mode, I don't have this issue.

Since I don't need the vmfs2 and nfsclient modules, I disabled them with esxcfg-module long before the update (and unloaded them with vmkload_mod back then). Now these modules get loaded again, even though they are supposed to be disabled.

Module Enabled Loaded

nfsclient false true

vmfs2 false true

esx.conf:

/vmkernel/module/nfsclient/enabled = "false"

/vmkernel/module/vmfs2/enabled = "false"

Another thing I noticed was that before the update, I would occasionally see 24/0 0x0 0x0 0x0 SCSI reservation conflicts in the vmkernel log, especially during higher I/O workloads. These appear much less frequent, if any, now.

Also, I haven't had any problems with previously installed HP Agents. Update to U4 -> Uninstall HP 8.11 or 8.0 agent -> Install 8.2 agent worked fine for all servers.

I'd also highly recommend setting the service console memory to about 512MB, since I saw that the COS was swapping before the update with the old HP agent and 384MB memory (apart from the fact that top and free indicated swapping on the COS, I didn't notice any actual performance impact).

I wish VMware would have given a clearer and detailed answer on the reason or benefits why we would need to Update the VMware tools. Just addressing it with one line in the release notes that states "This version of ESX Server requires a VMware Tools upgrade." is not convincing or reassuring at all.

[nikosm]

Upgraded my lab servers (2xhp dl 385, 3xhp dl 585) to ESX3.5 U4. No issues so far!

Didn't upgrade HP Agents because 8.1 seems to work fine with U4.

[FJF]

With Tyan motherboard (Toledo S5180) which incidentally works fine with XenSource, iESX doesn't recongise any of the hard drives for installation.

This Tyan motherboard only has SATA interfaces no IDE.

Very frustrating.

We are testing our products with various virtualisation platforms to ensure compatiability.

In fact our products were developed entirely on VMWare software but can't get iESX to install at all.

Any help would be much appreciated.

Regards

[Balmer]

I've just tried a fresh install of Update 4, onto BL460c. All looked fine until on checking it appears to be unable to load the LPe1105 drivers. (Firmware 2.72a2)

I'm getting a wonderful error when loading lputil "Driver lpfcdfc is incompatable. The driver must be at DFC Rev 1.19"

I'm deploying lputil 7.4.0.13-1, as part of the automated build. I tried deploying everything as it was but with ESX 3.5 Update 3 distribution files and it works fine, tried Update 4 again - zap, no banana. Strange - steering well clear of Update 4 for now.

Anyone else managed to deploy Update 4 onto LPe1105 HBA's??

[tmancini]

Balmer, I take it you are using Emulex HBA's?

[Balmer]

hi tmancini,

yep, the LPe1105 are Emulex cards. These are lab machines so crash and burn... I've just attached it into VC - and the storage is all hooked up and working. Its just lputil that isn't happy...

noticed in the release notes:

Emulex Fibre Channel Adapter Driver Update - The driver for Emulex Fibre Channel Adapters has been upgraded to version 7.4.0.40. This release provides support for the HBAnyware 4.0 Emulex management suite.

and

The ESX Server 3.5 Update 3 Emulex driver is, in some cases, incompatible with the Emulex fibre channel adapters

When Emulex LPe1150 fibre channel adapter with firmware version 1.00a5 or older is used with the Emulex driver shipped in ESX Server 3.5 Update 3, the driver fails to claim the Emulex fibre channel adapter.

This issue is resolved in this release. The VMware driver for this issue has been updated in ESX Server 3.5 Update 4. The new driver is now also compatible with Emulex fibre channel adapters that use firmware versions 1.00a5 and older. In short, the updated driver works irrespective of the Emulex fibre channel adapter firmware version.

I'm guess we'll be needing an updated lputil then.... I'll keep digging

[John_Hoover]

After completely redoing the zoning and presentation of the LUNs, I'm able to get one node up, but not both. A few minutes after the second node comes up, I start getting SCSI reservation conflicts and eventually lose access to one or more datastores on at least one node. Have not had any issues with this before now from 3.5GA to 3.5 U3. I guess I'm lucky that my VMs can run on one node without overtaxing the resources at this point. I'll continue to update the post here as I work with VMware support on the issue.

[NYSDHCR]

Has anyone upgraded their ESX host to 3.5 U4 with their VC at 2.5 U2? I need to apply a patch that is in ESX3.5 U4. it will be easier to apply ESX3.5 U4 than to apply the individual patch. The few enhancements in VC 2.5 U4 isn't really necessary at this time, since we will be going to v4 when it is released.

[RobMokkink]

I have installed U4 on all the lab servers. I don believe in upgrades, so i did fresh installs, it takes 10 minutes from bare-metal to managing them in VC.

I didn have any problems, update 3 was a disaster release. I will have to wait 2 more months before test, acceptance and production will be installed.

[nkrishnan]

Hi,

Please look in to http://www.vmware.com/pdf/vi3_35/esx_3/r35/vi3_35_25_compat_matrix.pdf , VC 25U2 is support esx35u4.

[bitsorbytes]

With Tyan motherboard (Toledo S5180) which incidentally works fine with XenSource, iESX doesn't recongise any of the hard drives for installation.

This Tyan motherboard only has SATA interfaces no IDE.

Very frustrating.

We are testing our products with various virtualisation platforms to ensure compatiability.

In fact our products were developed entirely on VMWare software but can't get iESX to install at all.

Any help would be much appreciated.

Regards

@ FJF - You do know that VMware only supports you on hardware that is on the HCL. There is no need to 'test' off the shelf Mb's. Review the HCL @ vmware website VMware certified Compatibility Guides

[NYSDHCR]

I know it is supported. I am asking if anyone upgraded their ESX host to 3.5 U4 with their VC at 2.5 U2?

[FJF]

Hi bitsorbytes,

Thanks for your reply.

Yes i do appreciate that. However we have a firewall software with speically features for eommerce which didn't work with earlier versions of XenSource but works with version 5.x upwards.

Having said that we throughouly test our products with most popular softwares we can before releasing them. It is actually more cost effective in the long run.

Hence the reason for wishing to test it with ESX. Which at the moment we can't install.

i even double checked the order of SATA hard drives and DVD making sure SATA hard drive is the first one.

ESX still can't recoginse any device it can write to. Any help will be much appreciated.

Kind Regards

[ML-EMP]

Did so - only on ESXi though, directly from U2. So far, so good (DL360 & 380 G5's).

We have also several ESX running on other DL380s, but not planning to upgrade them before VC.

I have to say since last august we always wait a while before any upgrade...

However the matrix shows it's supported, you should not expect any issue with U4 ESX managed by U2 VC.

Regards,

--

ML