VMware Cloud Community
JDLangdon
Expert
Expert

SPLUNK for VMware

Has anyone had any luck at getting SPLUNK for VMware setup? I'm trying to setup the VMware sourcetypes but I'm obviously doing something wrong configured correctly.

________________________________

Jason D. Langdon

Reply
0 Kudos
17 Replies
travesty
Contributor
Contributor

I'm also trying to setup the VMware application for Splunk but can't get it to work. Can anybody provide some problems they ran into and the resolutions to these problems?

Reply
0 Kudos
vanberg
Contributor
Contributor

I was able to get it up and running for our setup in about 45 min's (forgot to enable the firewall settings)

Works great

Eric

Reply
0 Kudos
JDLangdon
Expert
Expert

I was able to get it up and running for our setup in about 45 min's (forgot to enable the firewall settings)

I can get the syslog portion working fine but I cannot get the VMware api's to work correctly. I have two guys from SPLUNK look at my log files and config files and neither of them could offer any workable suggestions either.

________________________________

Jason D. Langdon

Reply
0 Kudos
travesty
Contributor
Contributor

Where should I be looking for firewall settings? I believe the problem I've run into is that VCenter is expecting a certificate from the Splunk server but for whatever reason the Splunk server isn't sending one. I'm stuck.

Reply
0 Kudos
vanberg
Contributor
Contributor

Depends....where are you in the setup? Are you installing on Windows? What part are you stuck at?

Assuming its windows:

goto a command prompt and type

echo%JAVAHOME%
echo %SPLUNK_HOME%

And paste the results in here...we can start from there.

Also, please paste in the url from your vmware.conf file (C:\Program Files\Splunk\etc\apps\vmware\default)

I will try and help you get it up and running.

Eric

Reply
0 Kudos
travesty
Contributor
Contributor

I have Splunk installed on a Linux distribution, Fedora 10. Both of those variables are set, so when I ran the command a blank line was followed. The url from the vmware.conf file is, address of VCenter)/sdk

Reply
0 Kudos
vanberg
Contributor
Contributor

I dont have a fedora box setup, but I should still be able to help get it going, plus i might build one up.

More questions:

1. Can you see the main splunk page when you go through a web browser?

2. Do you have the VMWare Application already installed?

3. From the splunk page, restart the splunk service.

3. Run the command cd $SPLUNK_HOME/etc/apps/vmware

4. Run the command java -jar lib/splunk.jar

5. Paste the first 20 lines from step 4.

Eric

Reply
0 Kudos
travesty
Contributor
Contributor

  1. yes

  2. yes, it was installed from splunkbase through our splunk interface

Here is what I grabbed from the test.

Started

Caught Exception : Exception : org.apache.axis.AxisFault Message : ; nested exception is:

gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL StackTrace :

AxisFault

faultCode: {[http://schemas.xmlsoap.org/soap/envelope/}Server.userException|http://schemas.xmlsoap.org/soap/envelope/%7DServer.userException]

faultSubcode:

faultString: gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

faultActor:

faultNode:

faultDetail:

{[http://xml.apache.org/axis/}stackTrace:gnu.javax.net.ssl.provider.AlertException|http://xml.apache.org/axis/%7DstackTrace:gnu.javax.net.ssl.provider.AlertException]: UNEXPECTED_MESSAGE: remotely generated; FATAL

at gnu.javax.net.ssl.provider.SSLEngineImpl.unwrap(libgcj.so.9)

at javax.net.ssl.SSLEngine.unwrap(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl.doHandshake(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl$SocketOutputStream.write(libgcj.so.9)

at java.io.BufferedOutputStream.flush(libgcj.so.9)

at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:516)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.retrieveServiceContent(VimBindingStub.java:23449)

at com.vmware.apputils.vim.ServiceConnection.connect(ServiceConnection.java:54)

at com.vmware.apputils.vim.ServiceUtil.clientConnect(ServiceUtil.java:36)

at com.vmware.apputils.AppUtil.connect(AppUtil.java:389)

at com.splunk.VMWareHostConnection.init(Splunk4VMI.java:275)

at com.splunk.Splunk4VMI.init(Splunk4VMI.java:393)

at com.splunk.Splunk4VMI.main(Splunk4VMI.java:573)

{[http://xml.apache.org/axis/}hostname:APP-07-SPLUNK.gripa.local|http://xml.apache.org/axis/%7Dhostname:APP-07-SPLUNK.gripa.local]

gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

at org.apache.axis.AxisFault.makeFault(AxisFault.java:101)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:154)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.retrieveServiceContent(VimBindingStub.java:23449)

at com.vmware.apputils.vim.ServiceConnection.connect(ServiceConnection.java:54)

at com.vmware.apputils.vim.ServiceUtil.clientConnect(ServiceUtil.java:36)

at com.vmware.apputils.AppUtil.connect(AppUtil.java:389)

at com.splunk.VMWareHostConnection.init(Splunk4VMI.java:275)

at com.splunk.Splunk4VMI.init(Splunk4VMI.java:393)

at com.splunk.Splunk4VMI.main(Splunk4VMI.java:573)

Caused by: gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

at gnu.javax.net.ssl.provider.SSLEngineImpl.unwrap(libgcj.so.9)

at javax.net.ssl.SSLEngine.unwrap(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl.doHandshake(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl$SocketOutputStream.write(libgcj.so.9)

at java.io.BufferedOutputStream.flush(libgcj.so.9)

at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:516)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

...16 more

Exception running : Splunk4VMI

Caught Exception : Exception : org.apache.axis.AxisFault Message : ; nested exception is:

gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL StackTrace :

AxisFault

faultCode: {[http://schemas.xmlsoap.org/soap/envelope/}Server.userException|http://schemas.xmlsoap.org/soap/envelope/%7DServer.userException]

faultSubcode:

faultString: gnu.javax.net.ssl.provider.AlertException: UNEXPECTED_MESSAGE: remotely generated; FATAL

faultActor:

faultNode:

faultDetail:

{[http://xml.apache.org/axis/}stackTrace:gnu.javax.net.ssl.provider.AlertException|http://xml.apache.org/axis/%7DstackTrace:gnu.javax.net.ssl.provider.AlertException]: UNEXPECTED_MESSAGE: remotely generated; FATAL

at gnu.javax.net.ssl.provider.SSLEngineImpl.unwrap(libgcj.so.9)

at javax.net.ssl.SSLEngine.unwrap(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl.doHandshake(libgcj.so.9)

at gnu.javax.net.ssl.provider.SSLSocketImpl$SocketOutputStream.write(libgcj.so.9)

at java.io.BufferedOutputStream.flush(libgcj.so.9)

at org.apache.axis.transport.http.HTTPSender.writeToSocket(HTTPSender.java:516)

at org.apache.axis.transport.http.HTTPSender.invoke(HTTPSender.java:138)

at org.apache.axis.strategies.InvocationStrategy.visit(InvocationStrategy.java:32)

at org.apache.axis.SimpleChain.doVisiting(SimpleChain.java:118)

at org.apache.axis.SimpleChain.invoke(SimpleChain.java:83)

at org.apache.axis.client.AxisClient.invoke(AxisClient.java:165)

at org.apache.axis.client.Call.invokeEngine(Call.java:2784)

at org.apache.axis.client.Call.invoke(Call.java:2767)

at org.apache.axis.client.Call.invoke(Call.java:2443)

at org.apache.axis.client.Call.invoke(Call.java:2366)

at org.apache.axis.client.Call.invoke(Call.java:1812)

at com.vmware.vim.VimBindingStub.retrieveServiceContent(VimBindingStub.java:23449)

at com.vmware.apputils.vim.ServiceConnection.connect(ServiceConnection.java:54)

at com.vmware.apputils.vim.ServiceUtil.clientConnect(ServiceUtil.java:36)

at com.vmware.apputils.AppUtil.connect(AppUtil.java:389)

at com.splunk.VMWareHostConnection.init(Splunk4VMI.java:275)

at com.splunk.Splunk4VMI.init(Splunk4VMI.java:393)

at com.splunk.Splunk4VMI.main(Splunk4VMI.java:573)

{[http://xml.apache.org/axis/}hostname:APP-07-SPLUNK.gripa.local|http://xml.apache.org/axis/%7Dhostname:APP-07-SPLUNK.gripa.local]

Reply
0 Kudos
JDLangdon
Expert
Expert

Looks like you're using the wrong version of java.

________________________________

Jason D. Langdon

Reply
0 Kudos
travesty
Contributor
Contributor

I'm using version 1.5, which is supposed to be compatible.

Reply
0 Kudos
JDLangdon
Expert
Expert

which did you install, java or jdk? I had to download and install jdk1.6.0.13 before it would work.

________________________________

Jason D. Langdon

Reply
0 Kudos
SimonSplunk
Contributor
Contributor

Hi there - Simon from Splunk here. I would try upgrading to Java 1.6 if you can.

You can also contact me directly: simon at splunk dot com

Reply
0 Kudos
travesty
Contributor
Contributor

We upgraded our Java version and installed the matching JDK version. It appears to have fixed the problem. How long does it take for Splunk to index all the data in order for me to see results in the VMware dashboards?

Reply
0 Kudos
JDLangdon
Expert
Expert

I never did get the VMware dashboards working.

________________________________

Jason D. Langdon

Reply
0 Kudos
travesty
Contributor
Contributor

It ran over night and it collected a good deal of data but none of the data was populated for the VMware dashboards. I'll install some ESX updates to see if that solves the problem.

Reply
0 Kudos
SimonSplunk
Contributor
Contributor

So you're seeing data when you search: sourcetype=vmware_api ?

If so, whats the latest timestamp you see? Can you click on "Report on Results"? Do you see fields on the left?

Regarding the dashboards, do you even see them in the pulldown? If not, are you using LDAP or not logging in as "admin"?

Reply
0 Kudos
travesty
Contributor
Contributor

All of the saved searches for the VMware app were disabled, so none of the information was being populated in the dashboards. Is this by default? I've enabled all of the saved searches and my dashboards are now being populated.

Reply
0 Kudos