VMware Modern Apps Community
AbhishekSK
Hot Shot
Hot Shot

server error 407

My wavefront proxy is running in a docker container and is continually reporting:

Jun 08, 2016 6:27:18 PM com.wavefront.agent.QueuedAgentService handleTaskRetry

WARNING: Cannot post push data result to Wavefront servers. Will enqueue and retry later: java.lang.RuntimeException: Server error:

407

Jun 08, 2016 6:27:18 PM com.wavefront.agent.AbstractAgent fetchConfig

INFO: fetching configuration from server at: https://try.wavefront.com/api/

Jun 08, 2016 6:27:20 PM com.wavefront.agent.QueuedAgentService$3 run

WARNING: [RETRY THREAD 0] cannot submit data to Wavefront servers. Will re-attempt later

java.lang.RuntimeException: Server error: 407

        at com.wavefront.agent.QueuedAgentService.parsePostingResponse(QueuedAgentService.java:470)

        at com.wavefront.agent.QueuedAgentService.access$700(QueuedAgentService.java:65)

        at com.wavefront.agent.QueuedAgentService$PostPushDataResultTask.execute(QueuedAgentService.java:557)

        at com.wavefront.agent.QueuedAgentService$3.run(QueuedAgentService.java:174)

        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)

        at java.util.concurrent.FutureTask.run(FutureTask.java:266)

The API token and URL seem to be set correctly in the config file (/opt/wavefront/wavefront-proxy/conf/wavefront.conf). Any ideas?

Reply
0 Kudos
6 Replies
AbhishekSK
Hot Shot
Hot Shot

Hi Stephan,

Error 407 is what you would normally get if your http traffic is routed through a proxy server and it requires authentication - if bypassing the proxy is not an option, could you please try setting the HTTP_PROXY environment variable and restart the container? The following template could be used to configure the proxy with authentication:

export HTTP_PROXY="http://{username}:{password}@{proxy_host}:{proxy_port}"

Please let us know if it helps!

-Vasily

Reply
0 Kudos
AbhishekSK
Hot Shot
Hot Shot

We're not using a proxy. I can run a shell in the same docker container and make requests to https://metrics.wavefront.com/api using the same auth token as the proxy without issue.

Reply
0 Kudos
AbhishekSK
Hot Shot
Hot Shot

That makes sense - in this case, could you please update the wavefront.conf file to point to metrics.wavefront.com instead of try.wavefront.com and restart the proxy? The default setting in the config points to the trial instance, but your account is actually set up on a production "metrics" cluster. 

-Vasily

Reply
0 Kudos
AbhishekSK
Hot Shot
Hot Shot

The config file says "server=https://metrics.wavefront.com/api". There's no mention of try.wavefront.com in the config, though that does appear in the logs.

Reply
0 Kudos
AbhishekSK
Hot Shot
Hot Shot

I restarted the proxy *within* the container. I think there may be a race between writing the config file and starting the proxy when the docker container starts. It now reports that it's retrieving config from metrics.wavefront.com (instead of try.wavefront.com) and it draining its queue.

Reply
0 Kudos
AbhishekSK
Hot Shot
Hot Shot

This turns out to be caused by a bug in the install script. I published a pull request with a fix:

start wavefront-proxy after writing config file by zuercher · Pull Request #21 · wavefrontHQ/install...

Reply
0 Kudos