Air Gapped TKG installation - waiting for cert-manager to be available

[MrSnowCake]

Hi, Good day!

I am testing TKG on an Air-gap server and was having a problem during the management cluster installation using UI.

The installation stop at "waiting for cert-manager to be available" then timed out.

I checked the pod using "kubectl get all -A --kubeconfig ~/.kube-tkg/tmp/config_aumYqnIn" and the status have "ImagePullBackOff"

NAMESPACE            NAME                                                                      READY   STATUS             RESTARTS   AGE
cert-manager         pod/cert-manager-8678dbf479-dmh5w                                         0/1     ImagePullBackOff   0          83s
cert-manager         pod/cert-manager-cainjector-587b86b44-86kl2                               0/1     ImagePullBackOff   0          83s
cert-manager         pod/cert-manager-webhook-c67547cbf-rhrr2                                  0/1     ImagePullBackOff   0          83s

The describe shows:

  Type     Reason     Age                 From               Message
  ----     ------     ----                ----               -------
  Normal   Scheduled  113s                default-scheduler  Successfully assigned cert-manager/cert-manager-8678dbf479-dmh5w to tkg-kind-cdvhvi0ej4ell81tha80-control-plane
  Normal   Pulling    38s (x4 over 112s)  kubelet            Pulling image "192.168.54.50:5000/bootstrap/cert-manager-controller:v1.5.3_vmware.4"
  Warning  Failed     38s (x4 over 112s)  kubelet            Failed to pull image "192.168.54.50:5000/bootstrap/cert-manager-controller:v1.5.3_vmware.4": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.54.50:5000/bootstrap/cert-manager-controller:v1.5.3_vmware.4": failed to resolve reference "192.168.54.50:5000/bootstrap/cert-manager-controller:v1.5.3_vmware.4": failed to do request: Head "https://192.168.54.50:5000/v2/bootstrap/cert-manager-controller/manifests/v1.5.3_vmware.4": http: server gave HTTP response to HTTPS client
  Warning  Failed     38s (x4 over 112s)  kubelet            Error: ErrImagePull
  Normal   BackOff    11s (x6 over 112s)  kubelet            Back-off pulling image "192.168.54.50:5000/bootstrap/cert-manager-controller:v1.5.3_vmware.4"
  Warning  Failed     11s (x6 over 112s)  kubelet            Error: ImagePullBackOff

The docker daemon is already configured to allow insecure-registries but it seems tanzu is using other.

Can I configure tanzu to use http instead of https when pulling the image?

TIA

[MrSnowCake]

The other option I can think is to configure the docker registry to use https, but I am sad to say I am also having an issue making it https "error:1408F10B:SSL routines:ssl3_get_record:wrong version number" using self signed certificate.

So I am hoping tanzu can pull the image in http instead.

[bdamian]

The registry must respond to HTTPS requests. If you use a self signed certificate, then your OS must trust the certificate.

If you are using Ubuntu, there is a simple procedure to trust the certificate. For other Linux distro, I can't help you though.

[MrSnowCake]

Hi, bdamian.

I am using RHEL OS, but is it possible to just let tanzu use http instead?

Thank you 🙂