failed to get image with harbor

[mashio]

I have vSphere with tanzu environment which was work properly.

after a downtime which I had to reboot all nodes( ESXi,vCenter,NSX)

now I cant get images when deploying new pods.

the error I get when apply new yaml file:

failed to get images: Image fitness/redis-8d0bcad8bdda90b91a150fdec3d9c6fe94e3dcd7-v68881 has failed. Error: Failed to resolve on node. Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head https://registry-1.docker.io/v2/bitnami/redis/manifests/

status of the new pod is "ErrImagePull "

#kubectl get pods

NAME                          READY   STATUS         RESTARTS   AGE

cart-795fb45db5-cs8lg         0/1     ErrImagePull   0          61s

cart-795fb45db5-pbpzd         0/1     Pending        0          23s

cart-redis-75b6495979-brrjm   1/1     Running        0          7m29s

all seems to be working properly. all NSX devices appear green, I can login to the supervisor cluster and manage pods, Harbor appear healthy.

any suggestion how to fix it?

Thanks.

[daphnissov]

That error means it's trying to contact Docker hub, so you have networking/firewall issues preventing resolution of that registry.

[mashio]

how you suggest to start troubleshoot it?

what component in the system is trying to connect to docker hub? worker node?

[daphnissov]

Check DNS resolution for docker hub from the ESXi hosts, check port access second.

[nachogonzalez]

Do you have a proxy?

[sghose0007]

SSH to the worker nodes or Control nodes and see if you can ping the https://registry-1.docker.io/v2/bitnami/redis/manifests/ and see if its reachable..

Also try to use the Embedded harbor and see if its get deployed

Run Kubectl describe pods cart-795fb45db5-cs8lg and see what the events say about..

[mashio]

there is no DNS issue I can resolve external address from worker node

[mashio]

I can wget one of the images from worker node.

wget  https://gcr.io/v2/vmwarecloudadvocacy/acmeshop-cart/manifests/stable

Connecting to gcr.io (64.233.160.82:443)

stable               100% |***************************************************************************************************************************************************************| 10044  0:00:00 ETA

any other suggestion except the embedded harbor?

[daphnissov]

You're testing against the wrong registry. Look at the pull error again.

[mashio]

im having the same issue on both images, just tried the second one.

[BrunoA7]

Hi All

Do you still have the issue ?
How did you resolve it ?

Regards

[ggtselingas]

Same issue:

Reason: Http request failed. Code 400: ErrorType(2) failed to do request: Head "https://projects.registry.vmware.com/v2/tkg/contour/manifests/sha256:8c5c66410ccca423b3b1635401a0fb3...": dial tcp: lookup projects.registry.vmware.com: i/o timeout
Normal Image 102s image-controller Image contour-204b221aceed9528140334ab567d869a62181e99-v54266 bound successfully
Normal BackOff 102s image-controller Backoff pulling images for pod. Retrying after 10m0s.
Normal Pulling 99s kubelet Waiting for Image svc-contour-domain-c26/contour-204b221aceed9528140334ab567d869a62181e99-v54266

Any help would be appreciated.