m1ghtybob Tracker

Re: Help needed to finish Powerchute configuration on ESXi 4.1

m1ghtybob — Thu, 09 Feb 2012 06:08:08 GMT

I found the answer myself and referenced the solution inside this thread for others to utilise.

Re: Help needed to finish Powerchute configuration on ESXi 4.1

m1ghtybob — Mon, 06 Feb 2012 03:48:37 GMT

Hi all,

I just wanted to let you know that I have found the solution. It was referenced on another website

The suggested configuration contained within that blog worked in my environment and I wanted to share it here in the event that if anyone else happens to have the same issue, they can find the resolution quicker than I could come by it:

Thanks to Canabalooza for his help in this:

Instructions on how to configure PCNS for ESXi (Free Version):

Boot the vMA machine, setup the network settings, and change the vi-admin password from the console window.

At this point I suggest you log out of console and use a telnet client to connect to the vMA so you can copy/paste commands and avoid mistakes.

Connected to the vMA virtual machine press Alt+F2 and login as vi-admin with your password.

Download the new ESXi PCNS 3.0 package from the APC website. You can find it by going to their website and selecting the Software & Firmware link under Popular Links to the left of the page. Once the page loads select the "Software Upgrades - PowerChute Network Shutdown" from the top drop down and click submit. Under the 3.0.0 section click the download button to the right. Select the download button for the ESXi package and login to the APC site. Select the download button on the next page to download the file.

Using SCP or WGET transfer the ESXi PCNS 3.0 package to the new vMA machine. I used /home/vi-admin to avoid permission issues.

Browse to the folder you transferred the package to and extract it from the console window,

tar -xzvf pcns300ESXi.tar.gz
cd ESXi

Run the installer,

sudo ./install_en.sh

Enter your vi-admin password and agree to the license terms.

Press Enter for default install path.

Type Yes and press Enter to install.

Now press Enter to install the included java package.

Next it should ask for the IP and credentials of your ESXi. You can press "q" to skip. If you put your IP in it will still work but you will get an error that it wasn't able to add and that you should manually add your host using vifp addserver. This is for vCLI functionality only and is unimportant when using SOAP. Just ignore it.

Edit the shutdown.pl file attached to the end of this post and enter your ESXi host credentials.

Use SCP or WGET to transfer the shutdown.pl and shutdown files to the vMA machine.

Still connected the the terminal change to the directory where you transferred the files, if same location as the PCNS package use,

cd ..

Copy the shutdown.pl and shutdown files to the default setup of PCNS

sudo cp shutdown.pl /opt/APC/PowerChute/group1/bin
sudo cp shutdown /opt/APC/PowerChute/group1/bin

Create a host file and add the ESXi host IP, for multiple hosts use new lines for each IP,

sudo nano -w /opt/APC/PowerChute/group1/bin/host

Contents should look similar,

10.0.0.4
10.0.0.5

Press Ctrl+X to quit nano, Y to save, Enter to keep same file name.

Make sure the files are owned by root and set chmod 0744.

sudo chown root /opt/APC/PowerChute/group1/bin/shutdown
sudo chown root /opt/APC/PowerChute/group1/bin/shutdown.pl
sudo chmod 744 /opt/APC/PowerChute/group1/bin/shutdown
sudo chmod 744 /opt/APC/PowerChute/group1/bin/shutdown.pl

Next configure the ESXi shutdown order. Access the vSphere client, select the ESXi host, then the configuration tab.

Click on the Virtual Machine Startup/Shutdown link. Click properties at the top right and use "Move Up" to move the VMs in the order you want leaving the vMA VM at the top. Note that the startup order is the only one displayed. Shutdown uses the startup list in reverse. If you want the VMs to start on

their own move them all the way up to Automatic Startup still leaving the vMA VM at the top.

Connect to the IP of the vMA machine ex. https://10.0.0.9:6547/login and setup your connection and events as needed. Make sure you remove the check box in the "Turn off UPS" box under Configure Shutdown unless you have the "Restart when power is restored" option set on the APC.

To test you can either unplug utility power to your APC and use the "UPS: On Battery" event or run the following command from the vMA console window,

cd /opt/APC/PowerChute/group1/bin
sudo ./shutdown.pl host

This will gracefully shutdown the VMs in the order you set before telling the ESXi host to power down. You should see the shutdown commands in the log start when connected to the ESXi host with the vSphere client. Also note that each VM must have VMWare Tools installed for the shutdown to work gracefully and make sure you give each VM ample time to power off before your battery power runs out.

NOTE: As pointed out by Andy some hosts have an issue shutting down the VMs gracefully if you don't touch the order on the startup list. If you don't want to access the ESXi host with the vSphere client whenever the power is restored to manually power on the VMs I suggest you move them all to the Automatic Startup section and setup the APC to power off at the end of the PCNS shutdown and power back on when power is restored in the APC settings. You can find the VMWare article here.

Also a note that PCNS and most management cards do NOT accept special characters in the password. Please see this article on the APC site for details.

Credits to lamw for the SOAP script here.

Re: Help needed to finish Powerchute configuration on ESXi 4.1

m1ghtybob — Sun, 05 Feb 2012 22:28:31 GMT

That is not very helpful. As in my Original post, I already tried that.

Help needed to finish Powerchute configuration on ESXi 4.1

m1ghtybob — Thu, 02 Feb 2012 22:21:14 GMT

Hi guys,

I am trying to configure Powerchute Network Shutdown so that in the event of a power outage it will gracefully shutdown first our virtual machines and then the hosts themselves.

The instructions I have followed thus far are these:

http://cosonok.blogspot.com.au/2011/12/installing-apc-ups-powerchute-network.html

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1007036

Now a quick overview of my infrastructure:

3 x ESXi 4.1 Hosts (free version)

1 x APC Smart UPS (with Network Management Card)

I have got the vMA configured and it is up and running. According to what I've read it should 'just work' once the UPS has kicked in but all I've seen happen is that the vMA itself communicates with the UPS as expected but no virtual servers shutdown nor do the ESXi hosts.

When I log in to the Powerchute Network Shutdown web interface there is a section called 'Configure Shutdown'. Although the instructions don't indicate anything is really required here, I can't help but feel that this might be where my downfall lay. Should I be putting something in this section to initiate the shutdown?

I am sure it is something really silly that I am missing out on here but it is driving me nuts! Are you guys able to help shed some light on what it is I have done wrong?

Cheers!

Re: vSphere ESXi + MD3000i Lun Trespassing issue

m1ghtybob — Fri, 23 Jul 2010 01:08:47 GMT

Hi All,

I just wanted to give you an update. I have found the source of the problem and it looks like the LUN trespass issue has since been resolved.

In the end it came down to the iSCSI initiator not being configured properly for the MD3000i and I could see in Event Manager of the VEEAM backup server that it could only receive I/O down 1 path.

I re-installed the initiator following this guide: http://www.delltechcenter.com/page/MPIO to PowerVault MD3000i with Microsoft iSCSI Initiator?t=anon

And then ran futher tests. I am able to confirm that by doing this, it fixed the issue. I have little niggling error messages in the Event Manager but that is more to do with tweaking settings on the server.

Dell confirmed that Round Robin is the preferred option when using the MD3000i but they couldn't actually clarify why you should ultimately use 2 subnets (I did it in the end just to apease the technician).

I wanted to thank everyone for their help. It has definitely been valuable in seeking resolution.

Cheers all!

Re: vSphere ESXi + MD3000i Lun Trespassing issue

m1ghtybob — Tue, 20 Jul 2010 06:26:55 GMT

I can confirm that it doesn't matter if you used Fixed, MRU or RR the issue still occurs. Waiting on DELL to see what they come up with next.

Re: vSphere ESXi + MD3000i Lun Trespassing issue

m1ghtybob — Mon, 19 Jul 2010 23:06:39 GMT

Hi All,

Sorry I haven't responded, I thought it best to wait until I actually did something.

Last night I reconfigured the MD3000i so it is now on two different subnets. I am going to be doing a few tests today to see what improvements it has made but one thing I should note is that I found the culprit as to why I lost near 50MB of speed. When I logged the call with DELL they got me to reconfigure my iSCSI software adapter on each host, what they also made me do is only put 2 Dynamic discovery targets in (one from each controller) and enter ALL of them in the Static Target Discovery. Upon reading their configuration guide last night I found that their best practice contradicts the actions the DELL technician asked me to complete. I have since set it up so that all ports are in the Dynamic Discovery targets and removed everything from Static Discovery. After doing a rescan everything sorted itself out and I used IOMeter to do more testing and I found that missing 50MB of speed!

So, at this stage everything is configured as per Dell's best practice guide: http://www.delltechcenter.com/page/VMwareESX4.0andPowerVault+MD3000i

If I still get the trespass, the next thing I will do is change the pathing policy to MRU.

An interesting side note is that DELL have told me in the next firmware release for the MD3000i they are fixing the alarm feature for the LUN Trespass (so it wont trigger one) as they deem it to be working properly and is misrepresented as an alarm in the current firmware.

Hopefully I will have good news to report later.

Cheers all!

Re: vSphere ESXi + MD3000i Lun Trespassing issue

m1ghtybob — Fri, 16 Jul 2010 03:07:45 GMT

binoche:

The DELL techs have told me otherwise, they keep asking, time and time again if I am using RR. Do you have any doco that suggests otherwise that I can have a look at?

I will test MRU regardless, keep the suggestions comming!

Re: vSphere ESXi + MD3000i Lun Trespassing issue

m1ghtybob — Thu, 15 Jul 2010 22:58:51 GMT

Thanks for posting some suggestions:

binoche:

I cannot see anywhere within the MD3000i console to disable AVT. I am calling DELL this morning to enquire further about this.
There is no network downtime when this occurs
The pathing policy in use is Round Robin

JohnADCO:

Yeah, its the SAN throwing these errors. There is no alarms or alerts anywhere within VMWare itself.
I don't use SVmotion with this setup
I was thinking VEEAM could be a culprit but I also discovered I can achieve this outcome using IOMETER and setting up a test sending 3MB packets to the SAN. Over time it will fail over to the non preferred path.

I will let you know what DELL say about AVT, I didn't know about this feature until I did some reading up on it.

vSphere ESXi + MD3000i Lun Trespassing issue

m1ghtybob — Thu, 15 Jul 2010 07:08:03 GMT

Hi All,

I have, what I would deem, a frustrating problem that I don't seem to be getting anywhere with so I thought I would post it on here to see what thoughts other readers had:

I have recently implemented a VM environment at the place where I work. The basic setup is this:

2 x Dell R710 Servers
2 x Cisco 2960G switches
1 x Dell MD3000i SAN

Since implementing this kit, from Day 1, I kept experiencing LUN trespasses. I have not had any luck with making progress in resolving this issue and still currently have an ongoing case open with DELL regarding this issue.

A classic example of how these LUN Trespasses occur would be:

Going home from work one night leaving the SAN in an optimal state only to arrive the next day with the dreaded flashing amber light on the front of the SAN indicating the SAN is in a 'non-optimal state'
Performing a VEEAM backup of a virtual machine

Every time this happens I bundle up all the logs from both of the ESXi hosts and the MD3000i and ship it to DELL only to have them tell me that there is no issue with the SAN and it is VMWare that is forcing IO to be sent down certain paths presented to it from the SAN. I have checked this 'theory' with VMWare and they are pretty much saying the only time ESXi would show that sort of characteristic is if you were using a Fixed pathing policy (which I am not, I am using Round Robin).

Quite a bit of troubleshooting has been carried out including:

Reconfiguring the iSCSI ports on each of the hosts
Putting in Dumb Gigabit switches to eliminate any 'bottle neck' issues there may be
Disabling and re-enabling paths in ESXi so that the hosts are properly referencing the correct Storage Processor to access the designated LUN
Replacing the suspect storage processor with a new one

Unfortunately, none of these have been able to releive the issues I am experiencing and every day, without fail, I will experience a LUN trespass. I should also state that currently this environment is only hosting 1 production server and a total of 6 VM's - non of which would be generating an exceptionally high load.

The only thing that was not followed properly when setting up the SAN was that both Storage Processors are in the same subnet as opposed to having their own dedicated subnet (as is Dell's best practice). This was not done by choice but rather dictated to me by our Networking team. I have also had Dell confirm that this is not an issue, especially not for this case.

On top of all of this I am getting what I would consider poor performance from the SAN at this stage. 50MB per second. Before DELL actually got involved and tried to re-do the iSCSI vmkernel ports I was getting around 100MB per second. I am not really having much luck here.

Anyway, what I would like to know is if ANYONE has ever experienced similar type behavior and if they managed to get it resolved in the end? DELL keep saying that the trespass is not a big issue and I shouldn't worry about it regardless but my issue is more to do with why is it doing it in the first place and where is my missing 50MB per second of speed!?

Any input would be greatly appreciated. I'd like to think I am not alone on this one but perhaps I just might be.

Cheers all.