mashio
Contributor
Contributor

failed to get image with harbor

I have vSphere with tanzu environment which was work properly.

after a downtime which I had to reboot all nodes( ESXi,vCenter,NSX)

now I cant get images when deploying new pods.

the error I get when apply new yaml file:

failed to get images: Image fitness/redis-8d0bcad8bdda90b91a150fdec3d9c6fe94e3dcd7-v68881 has failed. Error: Failed to resolve on node. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head https://registry-1.docker.io/v2/bitnami/redis/manifests/

status of the new pod is "ErrImagePull "

#kubectl get pods

NAME                          READY   STATUS         RESTARTS   AGE

cart-795fb45db5-cs8lg         0/1     ErrImagePull   0          61s

cart-795fb45db5-pbpzd         0/1     Pending        0          23s

cart-redis-75b6495979-brrjm   1/1     Running        0          7m29s

all seems to be working properly. all NSX devices appear green, I can login to the supervisor cluster and manage pods, Harbor appear healthy.

any suggestion how to fix it?

Thanks.

0 Kudos
9 Replies
daphnissov
Immortal
Immortal

That error means it's trying to contact Docker hub, so you have networking/firewall issues preventing resolution of that registry.

0 Kudos
mashio
Contributor
Contributor

how you suggest to start troubleshoot it?

what component in the system is trying to connect to docker hub? worker node?

0 Kudos
daphnissov
Immortal
Immortal

Check DNS resolution for docker hub from the ESXi hosts, check port access second.

0 Kudos
nachogonzalez
Expert
Expert

Do you have a proxy?

0 Kudos
sghose0007
Enthusiast
Enthusiast

SSH to the worker nodes or Control nodes and see if you can ping the https://registry-1.docker.io/v2/bitnami/redis/manifests/ and see if its reachable..

Also try to use the Embedded harbor and see if its get deployed

Run Kubectl describe pods cart-795fb45db5-cs8lg and see what the events say about..

0 Kudos
mashio
Contributor
Contributor

there is no DNS issue I can resolve external address from worker node

0 Kudos
mashio
Contributor
Contributor

I can wget one of the images from worker node.

wget  https://gcr.io/v2/vmwarecloudadvocacy/acmeshop-cart/manifests/stable

Connecting to gcr.io (64.233.160.82:443)

stable               100% |***************************************************************************************************************************************************************| 10044  0:00:00 ETA

any other suggestion except the embedded harbor?

0 Kudos
daphnissov
Immortal
Immortal

You're testing against the wrong registry. Look at the pull error again.

0 Kudos
mashio
Contributor
Contributor

im having the same issue on both images, just tried the second one.

0 Kudos