Pods cannot download images from external repositories

[sjesse]

I've been trying to setup a lab with tanzu and nsx-t enabled, and I can't download any images because the pods can't download the images. Pretty sure its because the "namespace network" isn't routing correctly, and I'm not sure why. I have my own T1 with a vm in it, and I can reach the ingress and egress ip range, but not the namespace network. I've tried adding static routes and I'm must be missing something

[sjesse]

This is what I'm trying to fix

Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 23s default-scheduler Successfully assigned default/grafana-9df5bb7db-vmt6w to grafana-cluster-workers-c8wfs-76b8bcb45c-ksxz2
Normal SuccessfulAttachVolume 22s attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-62a9dc7c-67c9-46f3-8318-a5e71a0bfc34"
Warning Failed 15s kubelet Failed to pull image "grafana/grafana:8.4.4": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/grafana/grafana:8.4.4": failed to resolve reference "docker.io/grafana/grafana:8.4.4": failed to do request: Head "https://registry-1.docker.io/v2/grafana/grafana/manifests/8.4.4": dial tcp 52.200.78.26:443: connect: network is unreachable
Normal BackOff 15s kubelet Back-off pulling image "grafana/grafana:8.4.4"
Warning Failed 15s kubelet Error: ImagePullBackOff
Normal Pulling 4s (x2 over 15s) kubelet Pulling image "grafana/grafana:8.4.4"
Warning Failed 4s (x2 over 15s) kubelet Error: ErrImagePull
Warning Failed 4s kubelet Failed to pull image "grafana/grafana:8.4.4": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/grafana/grafana:8.4.4": failed to resolve reference "docker.io/grafana/grafana:8.4.4": failed to do request: Head "https://registry-1.docker.io/v2/grafana/grafana/manifests/8.4.4": dial tcp 34.206.128.73:443: connect: network is unreachable

[Rober1]

When you see these kinds of errors, it means that Kubernetes cannot find your container image. The most common causes are:

The image value in your Domain is set incorrectly, meaning Kubernetes will be trying to pull the wrong image.
The image requires authentication or permission to pull it and you have not configured Kubernetes with the necessary credentials, for example in an imagePullSecret.
You built the image on a machine that is not where your kubelet is running and Kubernetes cannot see the image, meaning you need to copy the image to the worker nodes or put it in a container registry that is accessible the to all of the worker nodes. visit official site

[sjesse]

This turned out to be an external nat issue, where the egress range wasn't being NAT'd properly so it couldn't access anything externally.