my vRO-Test is gone! [ / ]# kubectl -n prelude get pods - No resources found.

[DexG83]

Hello Community,

do someone have any idea about the following...

vRO 8.0.1.7355 Build 15296158

after some workflow testing this day, at some point in time my orchestrator client was not reachable anymore.

looking for the pods shows the following output:

[ /var/log/vmware/prelude ]# kubectl -n prelude get pods

NAME                                   READY   STATUS    RESTARTS   AGE

orchestration-ui-app-86947f69d-jdx4l   0/1     Pending   0          104m

pgpool-6cf944ff54-lqjjb                0/1     Pending   0          109m

postgres-0                             0/1     Pending   0          104m

proxy-service-wnz7f                    0/1     Evicted   0          15s

symphony-logging-daemonset-nr2df       0/1     Evicted   0          15s

vco-app-5486dbc885-49txd               0/2     Pending   0          109m

so I tried tor redeploy

cd /opt/scripts/

./deploy.sh

The last lines of the deployment:

....

Error: could not find a ready tiller pod

+ ret=1

+ echo 'helm failed to upgrade '\''postgres'\'' in namespace '\''prelude'\'''

helm failed to upgrade 'postgres' in namespace 'prelude'

++ helm get postgres

++ head -n 1

++ sed -e 's/: /=/' -e 's/^\s*//' -e 's/\s*$//'

++ grep app:

Error: could not find a ready tiller pod

+ label=

+ [[ '' != '' ]]

+ return 1

+ local ret=1

+ echo 'Maximum number of retries exceeded.'

Maximum number of retries exceeded.

+ return 1

+ return 1

....

Error: could not find a ready tiller pod

+ ret=1

+ echo 'helm failed to upgrade '\''ingress-ctl'\'' in namespace '\''ingress'\'''

helm failed to upgrade 'ingress-ctl' in namespace 'ingress'

++ helm get ingress-ctl

++ grep app:

++ head -n 1

++ sed -e 's/: /=/' -e 's/^\s*//' -e 's/\s*$//'

Error: could not find a ready tiller pod

+ label=

+ [[ '' != '' ]]

+ return 1

+ local ret=1

+ echo 'Maximum number of retries exceeded.'

Maximum number of retries exceeded.

+ return 1

+ return 1

Now I have no pods at all.

[ / ]# kubectl -n prelude get pods

No resources found.

[ / ]#

Can anyone help me

thx alot

[iiliev]

Hi,

I've seen the following suggestion in another discussion about a similar error. Could you try it to see if it helps:

./deploy.sh --onlyClean
<reboot the appliance>
./deploy.sh

[DexG83]

Hi,

here is the status after testing your suggestions.

NAME                                   READY   STATUS    RESTARTS   
orchestration-ui-app-86947f69d-wzzz2   0/1     Pending   0          
pgpool-6cf944ff54-lschw                0/1     Pending   0          
postgres-0                             0/1     Pending   0          
proxy-service-74tmj                    0/1     Evicted   0          
symphony-logging-daemonset-gb2x2       0/1     Evicted   0          
vco-app-5486dbc885-zj5x4               0/2     Pending   0      

another reboot and a lot of time waiting did not solve the issue.

thank you for your help!

[tchristin]

Hi,

Please follow VMware Knowledge Base

I had the same issue and now it's solved !

Cheers,

Tim.

[DexG83]

Hey Tim,

thanks for your suggestion, in my special case it was a "nearly full" fs:

Filesystem Size  Used Avail Use% Mounted on
devtmpfs 5.9G     0  5.9G   0% /dev
tmpfs 5.9G     0  5.9G   0% /dev/shm
tmpfs 5.9G  2.2M  5.9G   1% /run
tmpfs 5.9G     0  5.9G   0% /sys/fs/cgroup
/dev/sda4 11G  9.1G  1.1G 90% /
tmpfs 5.9G     0  5.9G   0% /tmp
/dev/sda2 120M   22M   93M  19% /boot
/dev/mapper/metrics_vg-metrics 20G  250M   19G   2% /metrics
/dev/mapper/logs_vg-services--logs 7.9G  164M  7.3G   3% /services-logs
/dev/mapper/data_vg-data 59G   16G   41G  28% /data

increasing the corresponding disk resolved my issue ;D

Extend Filesystem: https://kb.vmware.com/s/article/76933​

Best regards,

Dex