DexG83
Enthusiast
Enthusiast

my vRO-Test is gone! [ / ]# kubectl -n prelude get pods - No resources found.

Hello Community,

do someone have any idea about the following...

vRO 8.0.1.7355 Build 15296158

after some workflow testing this day, at some point in time my orchestrator client was not reachable anymore.

looking for the pods shows the following output:

[ /var/log/vmware/prelude ]# kubectl -n prelude get pods

NAME                                   READY   STATUS    RESTARTS   AGE

orchestration-ui-app-86947f69d-jdx4l   0/1     Pending   0          104m

pgpool-6cf944ff54-lqjjb                0/1     Pending   0          109m

postgres-0                             0/1     Pending   0          104m

proxy-service-wnz7f                    0/1     Evicted   0          15s

symphony-logging-daemonset-nr2df       0/1     Evicted   0          15s

vco-app-5486dbc885-49txd               0/2     Pending   0          109m

so I tried tor redeploy

cd /opt/scripts/

./deploy.sh

The last lines of the deployment:

....

Error: could not find a ready tiller pod

+ ret=1

+ echo 'helm failed to upgrade '\''postgres'\'' in namespace '\''prelude'\'''

helm failed to upgrade 'postgres' in namespace 'prelude'

++ helm get postgres

++ head -n 1

++ sed -e 's/: /=/' -e 's/^\s*//' -e 's/\s*$//'

++ grep app:

Error: could not find a ready tiller pod

+ label=

+ [[ '' != '' ]]

+ return 1

+ local ret=1

+ echo 'Maximum number of retries exceeded.'

Maximum number of retries exceeded.

+ return 1

+ return 1

....

Error: could not find a ready tiller pod

+ ret=1

+ echo 'helm failed to upgrade '\''ingress-ctl'\'' in namespace '\''ingress'\'''

helm failed to upgrade 'ingress-ctl' in namespace 'ingress'

++ helm get ingress-ctl

++ grep app:

++ head -n 1

++ sed -e 's/: /=/' -e 's/^\s*//' -e 's/\s*$//'

Error: could not find a ready tiller pod

+ label=

+ [[ '' != '' ]]

+ return 1

+ local ret=1

+ echo 'Maximum number of retries exceeded.'

Maximum number of retries exceeded.

+ return 1

+ return 1

Now I have no pods at all.

[ / ]# kubectl -n prelude get pods

No resources found.

[ / ]#

Can anyone help me Smiley Sad

thx alot

0 Kudos
4 Replies
iiliev
VMware Employee
VMware Employee

Hi,

I've seen the following suggestion in another discussion about a similar error. Could you try it to see if it helps:

./deploy.sh --onlyClean

<reboot the appliance>

./deploy.sh

0 Kudos
DexG83
Enthusiast
Enthusiast

Hi,

here is the status after testing your suggestions.

NAME                                   READY   STATUS    RESTARTS  

orchestration-ui-app-86947f69d-wzzz2   0/1     Pending   0         

pgpool-6cf944ff54-lschw                0/1     Pending   0         

postgres-0                             0/1     Pending   0         

proxy-service-74tmj                    0/1     Evicted   0         

symphony-logging-daemonset-gb2x2       0/1     Evicted   0         

vco-app-5486dbc885-zj5x4               0/2     Pending   0     

another reboot and a lot of time waiting did not solve the issue.

thank you for your help!

0 Kudos
tchristin
Enthusiast
Enthusiast

Hi,

Please follow VMware Knowledge Base

I had the same issue and now it's solved !

Cheers,

Tim.

0 Kudos
DexG83
Enthusiast
Enthusiast

Hey Tim,

thanks for your suggestion, in my special case it was a "nearly full" fs:

Filesystem Size  Used Avail Use% Mounted on

devtmpfs 5.9G     0  5.9G   0% /dev

tmpfs 5.9G     0  5.9G   0% /dev/shm

tmpfs 5.9G  2.2M  5.9G   1% /run

tmpfs 5.9G     0  5.9G   0% /sys/fs/cgroup

/dev/sda4 11G  9.1G  1.1G 90% /

tmpfs 5.9G     0  5.9G   0% /tmp

/dev/sda2 120M   22M   93M  19% /boot

/dev/mapper/metrics_vg-metrics 20G  250M   19G   2% /metrics

/dev/mapper/logs_vg-services--logs 7.9G  164M  7.3G   3% /services-logs

/dev/mapper/data_vg-data 59G   16G   41G  28% /data

increasing the corresponding disk resolved my issue ;D

Extend Filesystem: https://kb.vmware.com/s/article/76933

Best regards,

Dex

0 Kudos