Not getting mailserver's wrf files, from some tile...

kfranks · ‎04-13-2012

I'm having mailserver problems....again.

I had successfully set up and run three tiles and recently added three more to the testbed. Now I'm seeing no wrf files for the odd-numbered tiles; tile1 had been working before, in the original set of three. I'm seeing that the vmmark2-mailserver-x.xml file is not getting updated in the respective failing clients and the mailservergetstats_error-x file(s) in the result directories contains:

Exiting, please wait...

Error: No valid counters.

Note:

In order to use typeperf, you must either be a member of the local

Performance Log Users group, or the command must be executed from an

elevated command window.

And now when I try and run the original three tiles, tile1 fails to produce a mailserver wrf file. Everything else looks ok. The firewall is turn off on all of the clients and all of the mailservers.

I'm attaching my results folder, less the host tgzs.

Please help.

Thanks,

-Kirk

jpschnee · ‎04-13-2012

Are the original 3 mailservers talking to their original 3 clients? IE has client1 has always talked to current mailserver1? It's very odd that a client who was previously able to communicate and drive load for a mailserver cannot after adding tiles, as nothing should have changed for tile1.

I would double check that all of the clients are in their proper domains. I would also double check that your Loadgen installations haven't become corrupted.

-Joshua

kfranks · ‎04-20-2012

I still haven't resolved this issue. I restored client0 to a previously working snapshot and tile0 works fine. I have rebuilt the tile1 client and mailserver from scratch. The databases set up properly. I can run the initialization from client1's loadgen and see activity on mailserver1. When I run the vmmark2-mailserver-1 from client1 it starts doing work then faults out from overload. I ran the simulation for 15 minutes, so I'm not too worried about this. When I run the harness I see mailserver1 initializing the databases and the logs in C: of mailserver1 indicate proper starts, stops and file operations. However when it runs the load there is no activity on mailserver1. The timestamp on the vmmark2-mailserver-1 file coincides with the test run. It's as if the prime client can't get client1 to run loadgen.

I'm going to snoop client1 with wireshark to see if I can capture the command that initiates the workload.

STAF on the prime client can ping client1 and mailserver1. STAF on client1 can ping client0 and mailserver1. STAF on mailserver1 can ping client1 and the prime clent.

Any thoughts?

Thanks,

-Kirk

Here's the log:

20120420-10:10:45	Starting VMmark2.1
20120420-10:10:51	VMmark2 : Reading Configuration File :: C:\vmmark2\VMMARK2.CONFIG
20120420-10:10:51	Default: VMMARK/IMPORTDIR = c:\vmmark2\xml
20120420-10:10:51	Enabled: VMMARK/DEBUGFLAG = 1
20120420-10:10:51	Overriding Required Default: VMMARK/RUNTIME = 900
20120420-10:10:52	Default: VMMARK/PARALLELSETUP = 1
20120420-10:10:52	Overriding Default: VMMARK/PARALLELSETUP = 1
20120420-10:10:52	Default: VMMARK/POSTPROCESSCMD = C:\cygwin\bin\perl -f C:\vmmark2\tilescore.pl -t %u
20120420-10:10:52	Default: VMMARK/TILEDELAY = 60
20120420-10:10:52	Overriding Required Default: VMMARK/REPORTER = 1
20120420-10:10:52	Default: VMMARK/ScriptName = vSphere-get-config.sh
20120420-10:10:52	Default: VMMARK/REPORTERDIR = /cygdrive/c/vmmark2/tools/
20120420-10:10:52	Default: VMMARK/RUNFLAG = 3
20120420-10:10:52	Default: VMMARK/RESCUE = 0
20120420-10:10:52	Default: VMMARK/CLEANUPFLAG = 1
20120420-10:10:52	Default: VMMARK/ERRORIMMEDIATE = 0
20120420-10:10:52	UserSpecified: VMMARK/VCServerUSER = Administrator
20120420-10:10:52	UserSpecified: VMMARK/VCServerPASSWORD *****
20120420-10:10:52	UserSpecified: VMMARK/VCServerCLUSTER = QA-Performance
20120420-10:10:52	UserSpecified: VMMARK/VCServerHOSTNAME = 192.168.101.81
20120420-10:10:52	Default: VMMARK/DoClusterMapping = 1
20120420-10:10:52	Default: VMMARK/INFRAOPSLIST = Vmotion SVmotion Deploy
20120420-10:10:52	Overriding Required Default: VMMARK/WORKLOADLIST = MailServer
20120420-10:10:52	Default: VMMARK/CONFIGDIR = c:\vmmark2
20120420-10:10:52	Default: VMMARK/TMPDIR = c:\vmmark2\tmp
20120420-10:10:52	Default: VMMARK/CLIENTROOTDIR = C:\\vclient%u
20120420-10:10:52	UNIFIED Work List : ['MailServer', 'Vmotion', 'SVmotion', 'Deploy']
20120420-10:10:52	Info: FS Create Directory: c:\vmmark2\results\Results_20120420101045_Two_Tiles Returned: RC = 0, STAFResult =
20120420-10:10:52	Info: COPY FILE C:\vmmark2\VMMARK2.CONFIG TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\VMMARK2.CONFIG
20120420-10:10:52	Info: FS Copy file: VMMARK2.CONFIG Returned: RC = 0, STAFResult =
20120420-10:10:52	Info: Create guestinfofiles subdirectory for guest host information files
20120420-10:10:52	Info: FS Create Directory: c:\vmmark2\results\Results_20120420101045_Two_Tiles\guestinfofiles Returned: RC = 0, STAFResult =
20120420-10:10:52	VMmark2 : Configuring Workloads
20120420-10:10:52	Default: VMMARK/MailServer/SKIPRESTORE = 0
20120420-10:10:52	Default: VMMARK/MailServer/MailServerPassword = *****
20120420-10:10:52	UserSpecified: VMMARK/MailServer/MailDomains = ['Maildomain0', 'Maildomain1']
20120420-10:10:52	UserSpecified: VMMARK/MailServer/MailQualifier = vtg.vce.com
20120420-10:10:52	Default: VMMARK/MailServer/clientpath = C:\Progra~1\Exchan~1\
20120420-10:10:52	Default: VMMARK/MailServer/DELAYTIME = 12
20120420-10:10:52	Default: VMMARK/MailServer/dblocation = E:\vmmark\
20120420-10:10:52	Default: VMMARK/MailServer/backuplocation = f:\
20120420-10:10:52	Default: VMMARK/Vmotion/Workloads = OlioDB
20120420-10:10:52	Default: VMMARK/SVmotion/Workloads = Standby
20120420-10:10:52	UserSpecified: NumSVmotionLUN(s) = '['Performance_Datastore6', 'Performance_Datastore7', 'Performance_Datastore8']'
20120420-10:10:52	Default DeployUpgradeTools : 1
20120420-10:10:52	UserSpecified: OScustomizations = '['VMmark_Customization0', 'VMmark_Customization0']'
20120420-10:10:52	UserSpecified: DeployLUN(s) = '['Performance_Datastore3', 'Performance_Datastore4', 'Performance_Datastore5']'
20120420-10:10:52	UserSpecified: DeployVMs = '['DeployVM0', 'DeployVM1']'
20120420-10:10:52	UserSpecified: Deploy Templates = '['DeployTemplate', 'DeployTemplate']'
20120420-10:10:52	VMmark2 : Starting Environment Setup
20120420-10:10:52	Info: tileindex = 0
20120420-10:10:52	Info: machName = client0
20120420-10:10:52	Info: machName = Mailserver0
20120420-10:10:53	Info: tileindex = 1
20120420-10:10:53	Info: machName = client1
20120420-10:10:55	Info: machName = Mailserver1
20120420-10:10:57	Client client0 : STAF version check : min 3.4.0 : found 3.4.7
20120420-10:10:58	Client client1 : STAF version check : min 3.4.0 : found 3.4.7
20120420-10:10:58	Info: All Client STAF Versions meet minimum requirements
20120420-10:10:58	VMmark2 : Checking Client Time Sync
20120420-10:10:58	Tile 0 : Prime Client Time 1334941858 : Client client0 Time : 1334941858 :: TimeErrors : 0 :: Clock Skew 0
20120420-10:10:59	Tile 1 : Prime Client Time 1334941859 : Client client1 Time : 1334941859 :: TimeErrors : 0 :: Clock Skew 0
20120420-10:10:59	vCServer : Prime Client Time 1334941859 : vCServer 192.168.101.81 Time : 1334941859 :: TimeErrors : 0 :: Clock Skew 0
20120420-10:10:59	Info: All Clients and vCServer Times Within 60 Second Threshold
20120420-10:10:59	InfraOps: Request Cluster ListHosts Info : (QA-Performance, c:\VMmark2\tools\VMmarkPScli.ps1 -f listhosts -vc 192.168.101.81 -c QA-Performance -u Administrator -p 'Acadia123')
20120420-10:11:24	VMmark 2 : Hosts In Cluster : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com']
20120420-10:11:24	InfraOps: sVMotion Request Cluster Info : (QA-Performance, c:\VMmark2\tools\VMmarkPScli.ps1 -f listvms -vc 192.168.101.81 -c QA-Performance -u Administrator -p 'Acadia123')
20120420-10:11:48	VMs In Cluster : ['Mailserver0', 'DS2WebA0', 'DS2WebB0', 'DS2WebC0', 'Standby0', 'OlioWeb0', 'DS2DB0', 'Standby1', 'DS2DB1', 'DS2WebA1', 'DS2WebB1', 'DS2WebC1', 'OlioWeb1', 'Mailserver1', 'DS2DB2', 'DS2WebA2', 'DS2WebB2', 'DS2WebC2', 'OlioDB2', 'OlioWeb2', 'Mailserver2', 'Standby2', 'OlioDB0', 'OlioDB1', 'DS2WebC3', 'OlioDB3', 'OlioWeb3', 'DS2DB3', 'DS2WebA3', 'DS2WebB3', 'Mailserver3', 'Standby3', 'DS2WebC4', 'OlioDB4', 'OlioWeb4', 'DS2DB4', 'DS2WebA4', 'DS2WebB4', 'Mailserver4', 'Standby4', 'DS2WebB5', 'DS2WebC5', 'OlioDB5', 'OlioWeb5', 'Standby5', 'DS2DB5', 'DS2WebA5', 'Mailserver5']
20120420-10:11:48	Checking VM Mapping And Resources
20120420-10:13:41	Warning : Cluster VM Mapping And Resources returned Warnings : InvalidResources 0 : PoweredOnErrors 1 : See c:\vmmark2\results\Results_20120420101045_Two_Tiles\ClusterVMmapping.txt
20120420-10:13:52	Info: Start Timer: client0 : start shell command "mstmr" workdir C:\\vclient0\mstmr workload mstmr0 sameconsole
20120420-10:13:52	Info: Start Timer: client1 : start shell command "mstmr" workdir C:\\vclient1\mstmr workload mstmr1 sameconsole
20120420-10:13:52	VMmark2 : Starting Workload Setup
20120420-10:13:52	Info: Tile0: COPY FILE vmmark2-mailserver-0.xml Returned: RC = 0, STAFResult =
20120420-10:13:52	Info: Tile0: DELETE ENTRY mailserver_exchange2007*.txt Returned: RC = 0, STAFResult = []
20120420-10:13:52	Tile0 : Starting MailServer Restore Process
20120420-10:13:53	Info: Tile1: COPY FILE vmmark2-mailserver-1.xml Returned: RC = 0, STAFResult =
20120420-10:13:53	Info: Tile1: DELETE ENTRY mailserver_exchange2007*.txt Returned: RC = 0, STAFResult = []
20120420-10:13:53	Tile1 : Starting MailServer Restore Process
20120420-10:14:48	Tile0 : MailServer Finished Restore Process
20120420-10:15:05	Tile1 : MailServer Finished Restore Process
20120420-10:15:05	ListVMs already done
20120420-10:15:05	ListHosts already done
20120420-10:15:05	Found ALL needed VMs for VMotion : Powered On VMs : ['Mailserver0', 'DS2WebA0', 'DS2WebB0', 'DS2WebC0', 'Standby0', 'OlioWeb0', 'DS2DB0', 'Standby1', 'DS2DB1', 'DS2WebA1', 'DS2WebB1', 'DS2WebC1', 'OlioWeb1', 'Mailserver1', 'DS2DB2', 'DS2WebA2', 'DS2WebB2', 'DS2WebC2', 'OlioDB2', 'OlioWeb2', 'Mailserver2', 'Standby2', 'OlioDB0', 'OlioDB1', 'DS2WebC3', 'OlioDB3', 'OlioWeb3', 'DS2DB3', 'DS2WebA3', 'DS2WebB3', 'Mailserver3', 'Standby3', 'DS2WebC4', 'OlioDB4', 'OlioWeb4', 'DS2DB4', 'DS2WebA4', 'DS2WebB4', 'Mailserver4', 'Standby4', 'DS2WebB5', 'DS2WebC5', 'OlioDB5', 'OlioWeb5', 'Standby5', 'DS2DB5', 'DS2WebA5', 'Mailserver5']
20120420-10:15:05	vmotion VMs : ['OlioDB0', 'OlioDB1']
20120420-10:15:05	Hosts in cluster : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com']
20120420-10:15:05	gHostNames : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com'] :: NumHosts 6
20120420-10:15:05	Calculated sVMotionBurstQueueSize = 1
20120420-10:15:05	UserSpecified: NumLUNs : 3 : TargetLUN(s) = '['Performance_Datastore6', 'Performance_Datastore7', 'Performance_Datastore8']'
20120420-10:15:05	ListVMs already done
20120420-10:15:05	Found ALL needed VMs for Storage VMotion : Powered On VMs : ['Mailserver0', 'DS2WebA0', 'DS2WebB0', 'DS2WebC0', 'Standby0', 'OlioWeb0', 'DS2DB0', 'Standby1', 'DS2DB1', 'DS2WebA1', 'DS2WebB1', 'DS2WebC1', 'OlioWeb1', 'Mailserver1', 'DS2DB2', 'DS2WebA2', 'DS2WebB2', 'DS2WebC2', 'OlioDB2', 'OlioWeb2', 'Mailserver2', 'Standby2', 'OlioDB0', 'OlioDB1', 'DS2WebC3', 'OlioDB3', 'OlioWeb3', 'DS2DB3', 'DS2WebA3', 'DS2WebB3', 'Mailserver3', 'Standby3', 'DS2WebC4', 'OlioDB4', 'OlioWeb4', 'DS2DB4', 'DS2WebA4', 'DS2WebB4', 'Mailserver4', 'Standby4', 'DS2WebB5', 'DS2WebC5', 'OlioDB5', 'OlioWeb5', 'Standby5', 'DS2DB5', 'DS2WebA5', 'Mailserver5']
20120420-10:15:05	sVMotion VMs : ['Standby0', 'Standby1']
20120420-10:15:37	Storage vMotion TargetLUN located and all VMs are on different LUNs
20120420-10:15:37	ListHosts already done
20120420-10:15:37	gHostNames : ['vxgle2vh0018.vxgle2.vtg.vce.com', 'vxgle2vh0020.vxgle2.vtg.vce.com', 'vxgle2vh0021.vxgle2.vtg.vce.com', 'vxgle2vh0022.vxgle2.vtg.vce.com', 'vxgle2vh0023.vxgle2.vtg.vce.com', 'vxgle2vh0024.vxgle2.vtg.vce.com'] :: NumHosts 6
20120420-10:15:37	Calculated DeployBurstQueueSize = 1
20120420-10:15:37	UserSpecified: Deploy NumVMss : 2 : DeployVM(s) = '['DeployVM0', 'DeployVM1']'
20120420-10:15:37	UserSpecified: Deploy NumLUNs : 3 : DeployLUN(s) = '['Performance_Datastore3', 'Performance_Datastore4', 'Performance_Datastore5']'
20120420-10:15:37	UserSpecified: Deploy NumTemplates : 2 : Deploy Template(s) = '['DeployTemplate', 'DeployTemplate']'
20120420-10:16:02	Info: Template 'DeployTemplate' : Located
20120420-10:16:26	Info: OS customization 'VMmark_Customization0' : Located
20120420-10:16:56	Info: DeployLUN 'Performance_Datastore3' : Located
20120420-10:17:01	VMmark2 : Run Starting
20120420-10:17:01	InfraOps: Collect Events and Perf
20120420-10:17:01	Tile 1 Delay 60000 ms
20120420-10:17:01	Tile 0 Delay 0 ms
20120420-10:17:01	Info: Request: run mailserver 0
20120420-10:17:13	Tile0: LoadGen Delay 300
20120420-10:17:13	Tile 0: Request: Run mailserver
20120420-10:18:01	Info: Request: run mailserver 1
20120420-10:18:01	Info : Calculated VMotionBurstQueueSize = 1
20120420-10:18:01	Info : Calculated sVMotionBurstQueueSize = 1
20120420-10:18:01	Info : Calculated DeployBurstQueueSize = 1
20120420-10:18:13	Tile1: LoadGen Delay 300
20120420-10:18:13	Tile 1: Request: Run mailserver
20120420-10:20:31	InfraOps: VMotion0 #0 : OlioDB0
20120420-10:20:41	InfraOps: Request sVMotion0 #1 : VM Standby0
20120420-10:20:51	DeployBurstQueue = ['DeployVM0'], DeployBurstHostQueue ['vxgle2vh0024.vxgle2.vtg.vce.com']
20120420-10:20:51	InfraOps: Request Deploy0 #0 : DeployVM0 onto SUT vxgle2vh0024.vxgle2.vtg.vce.com in LUN Performance_Datastore3 using VMmark_Customization0
20120420-10:22:13	Tile0 : Collecting MailServer Stats
20120420-10:23:13	Tile1 : Collecting MailServer Stats
20120420-10:23:14	Process: Tile 1: Collecting MailServer Stats failed to start/complete. Returned: RC = 4026531842, STAFResult = None
20120420-10:24:09	InfraOps: VMotion0 #1 : OlioDB1
20120420-10:27:49	InfraOps: VMotion0 #2 : OlioDB0
20120420-10:28:33	InfraOps: Thread Reached MinTime for sVMotions
20120420-10:28:41	InfraOps: Thread Reached MinTime for Deploy
20120420-10:28:41	Info: ping newly Deployed VM = 'DeployVM0'
20120420-10:28:42	Deployed VM #0 : Passed Ping Test
20120420-10:28:42	InfraOps: Request Destroy Deployed VM #0 : DeployVM0
20120420-10:29:11	VM 'DeployVM0' : Destruction completed successfully
20120420-10:31:31	InfraOps: VMotion0 #3 : OlioDB1
20120420-10:32:08	InfraOps : Thread Reached MinTime for VMotions
20120420-10:35:54	VMmark2 : Checking Client Time Sync
20120420-10:35:54	Tile 0 : Prime Client Time 1334943354 : Client client0 Time : 1334943354 :: TimeErrors : 0 :: Clock Skew 0
20120420-10:35:54	Tile 1 : Prime Client Time 1334943354 : Client client1 Time : 1334943354 :: TimeErrors : 0 :: Clock Skew 0
20120420-10:35:54	vCServer : Prime Client Time 1334943354 : vCServer 192.168.101.81 Time : 1334943354 :: TimeErrors : 0 :: Clock Skew 0
20120420-10:35:54	Info: All Clients and vCServer Times Within 60 Second Threshold
20120420-10:35:54	VMmark2 : Collecting Results
20120420-10:35:54	Info: COPY FILE C:\\vclient0\mailserver\Mailserver0.wrf TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\Mailserver0.wrf
20120420-10:35:54	Info: FS Copy file: Mailserver0.wrf Returned: RC = 0, STAFResult =
20120420-10:35:54	Info: COPY DIRECTORY C:\\vclient0\mailserver\ TODIRECTORY c:\vmmark2\results\Results_20120420101045_Two_Tiles\mailserver-debug0
20120420-10:35:55	Info: FS Copy Directory mailserver-debug0.xml Returned: RC = 0, STAFResult =
20120420-10:35:55	Info: COPY FILE C:\\vclient1\mailserver\Mailserver1.wrf TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\Mailserver1.wrf
20120420-10:35:55	Stafcmd: FS Copy file: Mailserver1.wrf failed with RC = 48, STAFResult = File C:\\vclient1\mailserver\Mailserver1.wrf does not exist
20120420-10:35:55	Info: COPY DIRECTORY C:\\vclient1\mailserver\ TODIRECTORY c:\vmmark2\results\Results_20120420101045_Two_Tiles\mailserver-debug1
20120420-10:35:59	Info: FS Copy Directory mailserver-debug1.xml Returned: RC = 0, STAFResult =
20120420-10:36:02	Could not Collect all Results for the following 1 Wklds: ['MailServerTile1 failed results collection']
20120420-10:36:02	Warning: VMmark may not have completed successfully. Check for results manually, see documentation for instructions.
20120420-10:36:02	Info: Stop Timer: client0 : stop workload mstmr0 using SIGTERM
20120420-10:36:02	Info: Stop Timer: client1 : stop workload mstmr1 using SIGTERM
20120420-10:36:02	VMmark2 : Reporter Routines Starting
20120420-10:36:02	ListHosts already done
20120420-10:36:02	Reporter : Generate Cluster Report : QA-Performance
20120420-10:36:03	Info: vxgle2vh0020.vxgle2.vtg.vce.com : Starting Reporting Script Routine
20120420-10:36:03	Info: vxgle2vh0018.vxgle2.vtg.vce.com : Starting Reporting Script Routine
20120420-10:36:03	Info: vxgle2vh0021.vxgle2.vtg.vce.com : Starting Reporting Script Routine
20120420-10:36:03	Info: vxgle2vh0023.vxgle2.vtg.vce.com : Starting Reporting Script Routine
20120420-10:36:03	Info: vxgle2vh0022.vxgle2.vtg.vce.com : Starting Reporting Script Routine
20120420-10:36:03	Info: vxgle2vh0024.vxgle2.vtg.vce.com : Starting Reporting Script Routine
20120420-10:37:27	Info: Tile0 : Mailserver0 : Get Windows guest host information
20120420-10:37:27	Info: Tile1 : Mailserver1 : Get Windows guest host information
20120420-10:37:35	Info: COPY FILE c:\Mailserver1_guestVM.info TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\guestinfofiles\Mailserver1_guestVM.info
20120420-10:37:35	Info: COPY FILE c:\Mailserver0_guestVM.info TOFILE c:\vmmark2\results\Results_20120420101045_Two_Tiles\guestinfofiles\Mailserver0_guestVM.info
20120420-10:37:36	Info: Create Windows guest host files: 1 Returned: RC = 0, STAFResult =
20120420-10:37:37	Info: Create Windows guest host files: 0 Returned: RC = 0, STAFResult =
20120420-10:37:37	Info: Tile1 : Mailserver1 : Get Windows guest host information completed
20120420-10:37:37	Info: Tile0 : Mailserver0 : Get Windows guest host information completed
20120420-10:38:49	scp root@vxgle2vh0021.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0021.vxgle2.vtg.vce.com.txt
20120420-10:38:49	scp root@vxgle2vh0021.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore1/reporter/vxgle2vh0021-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/
20120420-10:38:51	scp root@vxgle2vh0022.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0022.vxgle2.vtg.vce.com.txt
20120420-10:38:52	scp root@vxgle2vh0022.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0022-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/
20120420-10:38:52	vxgle2vh0021.vxgle2.vtg.vce.com : Removing Reporting Result from Host
20120420-10:38:54	vxgle2vh0022.vxgle2.vtg.vce.com : Removing Reporting Result from Host
20120420-10:38:56	scp root@vxgle2vh0023.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0023.vxgle2.vtg.vce.com.txt
20120420-10:38:56	scp root@vxgle2vh0023.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0023-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/
20120420-10:38:58	vxgle2vh0023.vxgle2.vtg.vce.com : Removing Reporting Result from Host
20120420-10:39:04	scp root@vxgle2vh0018.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0018.vxgle2.vtg.vce.com.txt
20120420-10:39:04	scp root@vxgle2vh0018.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0018-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/
20120420-10:39:07	vxgle2vh0018.vxgle2.vtg.vce.com : Removing Reporting Result from Host
20120420-10:39:17	scp root@vxgle2vh0020.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0020.vxgle2.vtg.vce.com.txt
20120420-10:39:18	scp root@vxgle2vh0020.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0020-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/
20120420-10:39:20	vxgle2vh0020.vxgle2.vtg.vce.com : Removing Reporting Result from Host
20120420-10:42:34	scp root@vxgle2vh0024.vxgle2.vtg.vce.com:/tmp/tgzpath /cygdrive/c/vmmark2/tmp//tgzfile-vxgle2vh0024.vxgle2.vtg.vce.com.txt
20120420-10:42:35	scp root@vxgle2vh0024.vxgle2.vtg.vce.com:/vmfs/volumes/Performance_Datastore2/reporter/vxgle2vh0024-20120420-1736.tgz /cygdrive/c/vmmark2/results/Results_20120420101045_Two_Tiles/
20120420-10:42:37	vxgle2vh0024.vxgle2.vtg.vce.com : Removing Reporting Result from Host
20120420-10:42:37	Info: Cleaning up STAF System Variables
20120420-10:42:37	VMmark2 : Ending Normally

jpschnee · ‎04-20-2012

Hi,

Can you zip up the entire results directory for this run? I'm concerned about it faulting due to overload and want to review for task exceptions.

Thanks

-Joshua

kfranks · ‎04-20-2012

Here it is, less the linux tgzs.

Thanks,

-Kirk

jpschnee · ‎04-20-2012

Can you confirm the state of the counters under load?

Start a test and then wait for initialization phase to pass.

Then:

1. Log into your client1 as the domain1 administrator.

2. Open a command prompt and type "perfmon".

3. Select "Performance Monitor" from the left pane (Under performance).

4. Click the green add button.

5. You should be able to scroll up to the "Exchange Load Generator Engine" counters.

6. Press the expand + box to the right of the name and then select the "Task Completed" and "Task Exceptions" counters.

7. Press the "add >>" button and then OK.

Let me know what you see.

-Joshua

kfranks · ‎04-20-2012

Joshua,

After I hit the OK button with these counters, I see a graph with nothing but little blips along the bottom which indicate processor time (when I hover over them).

-Kirk

kfranks · ‎04-20-2012

It doesn't look like these counters are getting added. I also tried adding them via the properties; I see them added with colors for the datasets, but nothing is put into the main graph panel.

-Kirk

jpschnee · ‎04-20-2012

That sounds like loadgen wasn't actually running. I can see that client1' s vmark-mailserver-1.xml is getting updated so I'd suggest that you run Loadgen manually against that file and see why it's crashing.

The process time counter is already there by default btw.

-Joshua

kfranks · ‎04-20-2012

Joshua

I can see those counters listed in the legend now. Loadgen is showing the tasks completing w/o exceptions, and perfmon shows them at 100% completing. I think the overload I saw in the previous run may have been the result of my monkeying with the constraints of the test run. There's about 5 minutes left and the queue length is 0, I don't think it's going to fault out.

So if Loadgen isn't getting started by Client1 (I assume), what should I be looking at/for?

Thanks,

-Kirk

jpschnee · ‎04-20-2012

So are you saying that everything started working fine after you ran loadgen manually with the vmmark2-mailserver-1.xml file?

If so, did you have to change anything beyond time settings?

Based on the timestamps, is Loadgen creating any new files when the harness starts a run?

-Joshua

kfranks · ‎04-20-2012

That is correct, when I didn't alter the parameters on the local execution, it ran correctly w/o errors. That's why I wasn't too concerned earlier, when I had altered them and it overloaded...figured it was the fault of my monkeying with the run parameters.

I'm watching a run right now. When the harness began mailserver1 actions, I saw the timestamp of the vmmark2-mailserver-1 file change to reflect that activity. I see a LoadGenRemoteSvc process running on client1. Client0 shows LoadGenRemoteSvc and LoadGen processes running. Mailserver0 has activity, mailserver1 does not.

-Kirk

jpschnee · ‎04-20-2012

On client1, take a look a the C;\Program Files*\Exchange Load Generator folder. See if Loadgen is creating new files in that directory when the harness starts a run. You should be able to match the timestamps and the look in the files to see what's going on.

-Joshua

kfranks · ‎04-20-2012

The newest files are those coincident with the local run of LoadGen, nothing newer matching the latest harness run. In fact, there appears to be three sets of files. The first, I think, is associated with building the database yesterday, on this new client. The second if the 'overloaded' run of LoadGen earlier. And the third is the passing run of LoadGen.

-Kirk

jpschnee · ‎04-20-2012

OK, so what happens if you start a command prompt on your client1 and type

"dir c:\Progra~1\Exchan~1\LoadGenCmd.exe"

-Joshua

kfranks · ‎04-20-2012

I get a listing of LoadGen.

jpschnee · ‎04-20-2012

OK, so on the prime client start up a cygwin prompt and type the following:

staf client1 process start command "c:\\Progra~1\\Excha~1\\LoadGenCmd.exe /r /c C:\\vclient1\\mailserver\\vmmark2-mailserver-1.xml" returnstdout stderrtostdout wait

-Joshua

kfranks · ‎04-20-2012

I get this:

Administrator@Client0 ~

$ staf client1 process start command "c:\\Progra~1\\Excha~1\\LoadGenCmd.exe /r /c C:\\vclient1\\mailserver\\vmmark2-mailserver-1.xml" returnstdout stderrtostdout wait

Error submitting request, RC: 10

Additional info

---------------

Error starting the process. CreateProcess failed with OS RC 2: The system cannot find the file specified.

Invalid command: c:\Progra~1\Excha~1\LoadGenCmd.exe /r /c C:\vclient1\mailserver\vmmark2-mailserver-1.xml

Administrator@Client0 ~

kfranks · ‎04-20-2012

There was syntax error in the path string (Excha~1 => Exchan~1). Correcting it, I got:

Administrator@Client0 ~

$ staf client1 process start command "c:\\Progra~1\\Exchan~1\\LoadGenCmd.exe /r /c C:\\vclient1\\mailserver\\vmmark2-mailserver-1.xml" returnstdout stderrtostdout wait

Response

--------

{

Return Code: 1

Key : <None>

Files : [

{

Return Code: 0

Data : Microsoft Exchange Load Generator

Configuring LoadGen task engine...

Loading settings...

ERROR -- Caught exception UnauthorizedAccessException:

Access to the registry key 'Global' is denied.

at Microsoft.Exchange.Swordfish.TaskEngine.initializeUserGroups()

at Microsoft.Exchange.Swordfish.TaskEngine..ctor(GlobalSettings settingsObject)

at Microsoft.Exchange.Swordfish.Cmd.SwordfishCmd.Main(String[] args)

}

]

}

Administrator@Client0 ~

jamesz08 · ‎04-23-2012

The problem is that the user that is running staf on client1 is not part of the Domain Admin group. Make sure the staf process is started by the domain administrator. The local administrator can work also, but I have seen some issues with it. Usually the local administrator account is automatically added to the Domain Admin group, but sometimes it doesn't seem to work. If you verify that staf is started by the domain administrator and it is still not working try rejoining the domain on that client.

All

Not getting mailserver's wrf files, from some tiles