Another frustrating update:
After working with 3.7.2 for a couple of times I decided lets give the upgrade to 3.8.0 a shot and see how it goes. Turns out the issue returns as soon as SDDC manager gets migrated! Any ideas or help on this would be greatly appreciated.
Here is a look at the sddcManagerServer.log:
2019-08-31T16:05:09.278Z ERROR [3fa23d81b8c24ee6] [services/errorHandling.js, http-get-ui, productionErrorRoute:85] Sending error response: {"id":"152af7ca-4cbc-4ae4-aba0-6451d9b4df89","requestedPath":"/ui/","error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"},"stack":"error: Session middleware wrapper failed initial and all retry attempts. Cannot continue. Throwing error from session middleware; password authentication failed for user \"vcfui\"\n at Connection.parseE (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:546:11)\n at Connection.parseMessage (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:371:19)\n at Socket.<anonymous> (/opt/vmware/vcf/sddc-manager-ui-app/server/node_modules/pg/lib/connection.js:114:22)\n at Socket.emit (events.js:189:13)\n at addChunk (_stream_readable.js:284:12)\n at readableAddChunk (_stream_readable.js:265:11)\n at Socket.Readable.push (_stream_readable.js:220:10)\n at TCP.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)"}
2019-08-31T16:05:12.263Z VERBOSE [3bec8569d3b84280] [services/util.js, http-get-ui, logRequest:262] Incoming request {"url":"/ui/","method":"GET","id":"c17554df-3198-4f18-95af-fb230ea985a3"}
2019-08-31T16:05:12.263Z DEBUG [3bec8569d3b84280] [services/util.js, http-get-ui, fixIncomingUrlPrefix:376] Start fixIncomingUrlFormat() {"id":"c17554df-3198-4f18-95af-fb230ea985a3"}
2019-08-31T16:05:12.264Z VERBOSE [3bec8569d3b84280] [services/util.js, http-get-ui, logRequestBody:287] Incoming request /ui/ has body {"body":{},"id":"c17554df-3198-4f18-95af-fb230ea985a3"}
2019-08-31T16:05:12.264Z INFO [3bec8569d3b84280] [../node_modules/express/lib/router/layer.js, http-get-ui, handle:95] Start sessionMiddlewareWrapper middleware()
2019-08-31T16:05:12.264Z INFO [3bec8569d3b84280] [../node_modules/express/lib/router/layer.js, http-get-ui, handle:95] Starting sessionMiddleware function; current retry count = 0
2019-08-31T16:05:12.264Z VERBOSE [3bec8569d3b84280] [services/logging/opentrace.js, http-get-ui, _finish:227] Span http-get-ui complete: Total time: 1ms {"duration":1}
2019-08-31T16:05:12.266Z WARN [3bec8569d3b84280] [../node_modules/express-session/index.js, http-get-ui, null:450] Session middleware produced an error while validating session. Retrying
2019-08-31T16:05:12.266Z ERROR [3bec8569d3b84280] [../internal/process/next_tick.js, http-get-ui, _tickCallback:68] undefined {"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed"}
2019-08-31T16:05:12.266Z ERROR [3bec8569d3b84280] [services/wrappers/pgSessionWrapper.js, http-get-ui, sessionMiddlewareWrap:96] Error from session middleware caught in middleware wrapper 0 {"error":{"name":"error","length":101,"severity":"FATAL","code":"28P01","file":"auth.c","line":"328","routine":"auth_failed","level":"error"}}
