Resxtop with VMA 4.1 batch mode capture not opening in perfmon

[ncolyer]

I've created an hour long batch capture using resxtop with the following command:

resxtop -b -n 720 --server servername.domain > vmacapture.csv

However when I add the Data Source to perfmon I get:

"Data Source C:\filename.csv is either invavlid or cannot be found. System message: The specified log file type has not been installed on this system.

I completed another small capture earlier, 9 iterations and that loaded fine.

I've tried capturing more batches, 3 more attempts of 2 minutes but all of them come up with the same error..

Any ideas on why this is failing? I can open the data in excel fine...

[DuncanArmstrong]

It's possible you're exceeding limitations of the performance monitor, but I haven't tried reading esxtop batch outputs in the newer performance monitors. If you tried it in Windows XP, for example, would it work?

Either way, you might want to consider esxplot, perhaps? http://labs.vmware.com/flings/esxplot

[ncolyer]

Thanks for the reply.

Unfortunately my .CSV files that ESXTOP/RESXTOP exports do not open in ESXPLOT either.

I get the following error

C:/filename.csv doesn't seem to be an estop data set, Corrupt Header Information.

The strange thing is, i managed to get one CSV to work a little while ago, and then every subsequent RESXTOP batch mode capture gives this same problem in perfrmon and ESXPLOT.

[VTsukanov]

Strange ...

Does the service "Performance Counter DLL Host" work?

[ncolyer]

i'm not sure exactly what you mean? How do i find this service?

I'm not familiar with the vMA command line tools that well.

I've tried loading the .CSV file into Perfmon, ESXplot, and again in perfmon on a windows 7 and another windows xp machine. Same problem

[DuncanArmstrong]

Hm, how are the files being transferred off of the vMA? I'm wondering if we're looking at encoding issues, but that really should not happen.

Not sure what to tell you. Perhaps you can take the first few KB of the output and attach it. For instance: `dd if=file.csv of=sample.csv bs=1k count=32`

I don't truly think there's anything wrong with the .csv, but it's worth checking.

[ncolyer]

I'm transferring the files off using WINSCP.

I will remove our company info and attach one of the CSV files here.

[VTsukanov]

Sorry, I misprinted.

I meant windows service "Performance Logs and Alerts" on the host where you are trying to start perfmon.

[ncolyer]

Attaching vmatest2.csv

The performance service did work for 2 of the CSV files. So i believe it is working. Also i'm getting errors in ESXPLOT as well not just perfmon. And i tried in perfmon on another newly built windows 7 machine which had the same error when trying to open it.

Thanks again for the help so far.

[VTsukanov]

There is an error in your cvs file. It starts as

[2010-09-10 12:17:05.902 F0DD4B90 warn...

instead

"(PDH-CSV 4.0) (PDT)(0)..."

[ncolyer]

Yeah I saw that on a few of them. However even one of them that didn't have that error also had a problem.

Nevertheless do you know what is causing this particular error? or any ideas how i can find out?

I also tried pasting the data into one that was working, which didn't have that error on the top but had all the columns present, but this didn't work either.

[DuncanArmstrong]

Yep, definitely broken. As it was pointed out, we need specific parts in the header.

Read the file:

$ hexdump -C stripped-esxtop.csv | less

00000000 22 28 50 44 48 2d 43 53 56 20 34 2e 30 29 20 28 |"(PDH-CSV 4.0)* (*|

00000010 45 44 54 29 28 30 29 22 xx xx xx xx xx xx xx xx |EDT)(0)" ...

I've had to strip some content for confidentiality, so try not to read into it too much. Note the bolded part.

Looking at the CSV you have (and I'm not sure if it's due to stripping confidential data or not):

00000000 5b 32 30 31 30 2d 30 39 2d 31 30 20 31 32 3a 31 |[2010-09-10 12:1|

00000010 37 3a 30 35 2e 39 30 32 20 46 30 44 44 34 42 39 |7:05.902 F0DD4B9|

00000020 30 20 77 61 72 6e 69 6e 67 20 27 41 70 70 27 5d |0 warning 'App']|

00000030 20 43 6c 6f 73 69 6e 67 20 52 65 73 70 6f 6e 73 | Closing Respons|

00000040 65 20 70 72 6f 63 65 73 73 69 6e 67 20 69 6e 20 |e processing in |

00000050 75 6e 65 78 70 65 63 74 65 64 20 73 74 61 74 65 |unexpected state|

00000060 3a 20 33 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c |: 3,,,,,,,,,,,,,|

00000070 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c 2c |,,,,,,,,,,,,,,,,|

*

00001d80 2c 2c 2c 0d 0a 28 50 44 48 2d 43 53 56 20 34 2e |,,,..(PDH-CSV 4.|

00001d90 30 29 20 28 45 44 54 29 28 30 29 2c 5c 5c 73 72 |0) (EDT)(0)

Note: The asterisk character implies that everything between 00000070 and 00001d80 was all zeroes.

The series of 2c (comma) characters is probably not a good thing. It looks like we're having a vMA to host communication error that's breaking the CSV file. On the bright side, you can probably repair and correct the files by hand, though I don't know if you're happy with the idea in general.

Erase everything from the beginning of the file until you get to (PDH-CSV 4.0), file should play.

Looks like a communication problem. Now to dig into that.

2010-09-10 12:17:05.902 F0DD4B90 warning 'App' Closing Response processing in unexpected state: 3

Message was edited by: DuncanArmstrong: WYSIWYG problems!

[ncolyer]

Thanks for the response. Absolutely I don't want to have to edit by hand every time. Is the corrupt data just in a few columns or is it something easy to find in excel? I tried deleting just the top line and it still errors out.

I'm also moving the vMA appliance to the same Host now. This should hopefully resolve any host/vma communication problems if they exist...

I should know the results of the next resxtop batch job soon.

[ncolyer]

Same problem with the next capture even with the vMA on the same host.

Also is there a reason why deleting that top line up until the part you mentioned still doesn't work?

I saved it in Excel as a .CSV file with that error line removed.

Just to add to the confusion also. If i open one of my .csv files that does work in perfmon, open it in excel, then save it straight away as a different file name in the same .csv format...it doesn't open in perfmon now :S

[DuncanArmstrong]

I can't find much on this issue here. If you have a support agreement, please file a Support Request so we can get a better look at it, and have a bug filed on it. Right now I don't think resxtop is handling this error, or ESX is passing on a hostd communication issue into the communication to vMA.

What build number is your ESX host? It's definitely later than ESX 3.5 Update-2, correct? Your next test should help to eliminate networking concerns, though it depends on your logical networking arrangement (e.g. are we still forced to have the vMA communicate to the network, then route to the management network or vCenter?).

Another workaround is to run esxtop in batch mode (with the same arguments) on the respective ESX host(s). This defeats the purpose of a vMA and vSphere CLI, arguably.

You can also try the vSphere CLI from other hosts/locations and see if it's limited to the vMA or not.

[DuncanArmstrong]

Take a peek at the saved file in a text editor real quick and see if the header is intact, or if there's anything still preceding. I also don't know yet if Excel throws anything else in, but I don't think it will.

I'm going to try the same on your sample now.

Edit: Outright removal of the first line in Excel will work fine. I think the problem is all the fields are not populated with double-quotes. Would you kindly have another sample created from a known-good CSV you took? If you take a look at the two (repair attempt via Excel vs Known-Good), there should be a visible difference if I'm on the right track.

[ncolyer]

I have an SR open right now 1569628651. The support agent is looking at this thread as well.

[DuncanArmstrong · Accepted solution]

Just talked with your assigned TSE (as you know by now) - we may end up relying on some workarounds like running localized instances of esxtop, until we get to the bottom of the issue. For now by comparison, indeed the only difference is a loss of encapsulating quotation marks around every field name and data value. In my opinion it shouldn't matter when in CSV format, but it... appears to.

Edit: Upon reflection, that is a standard for a CSV, actually. We really want those quotation marks!

[ncolyer]

Thanks for all the help you provided.

After loading up the vMA 4.0 everything seems to be working properly, 2 batch jobs managed to complete without error and open in perfrmon. I'm now starting our 1 hour baseline capture of a server which hopefully will not have any issues. If so we will have to move to ESXTOP temporarily as suggested.

[DuncanArmstrong]

I'm glad we were able to help. You can run SSH commands via the vMA if you desire and achieve a similar effect to what it was supposed to provide.

Essentially it breaks down as:

1) SSH command to remotely execute a given command.

2) When the command completes, grab the given filename via SCP and save the file on the vMA.

3) Relocate the file to your processing server. You could possibly do this in one sequence using the above command via CIFS mount.

But to clarify one item: Did vMA 4.0 not have the issue, where vMA 4.1 does?