Depolying Harbor into a Workload or a Shared Services Cluster for Container Service Extension (CSE)

[marcus-noto]

Depolying Harbor into a Workload or a Shared Services Cluster for Container Service Extension (CSE) 3.1.4 with Tanzu Kubernetes Grid (TKG) 1.5.4.

Problem description:
I have deployed Harbor into a Workload or a Shared Services Cluster for Container Service Extension (CSE) 3.1.4 with Tanzu Kubernetes Grid (TKG) 1.5.4, but I experienced problems when following the documentation. Following the documentation, step-for-step did not successfully install the Harbor Tanzu cli package and I had to do my own custom changes to make it work.

1. Did anything change in the environment prior to the issue occurring?
Answer: Not to my knowledge.

2. What date/time (and time zone!) did the issue begin?
Answer: Not relevant since this is an installation problem. I started installation on the 28/10/2022 12.00 (UTC+1)

3. Provide the date/time (and time zone!) of a specific instance of the problem
Answer: Not relevant since this is an installation problem. I started installation on the 28/10/2022 12.00 (UTC+1)

4. What was the full error message displayed on the screen or in the logs?
Answer: There are at least two errors encountered throughout the installation process:

First issue.
The first error was received when trying to install the package into the designated namespace, but the problem is that Tanzu packages are only installed into the default namespace. According to the documentation the flag --namspace is supposed to be the target namespace for the installation, but in fact it is the namespace from which the package is retrieved from. This should be updated to avoid any further confusion.
command and error:
- $ tanzu package install harbor --package-name harbor.tanzu.vmware.com --version 2.5.3+vmware.1-tkg.1 --values-file harbor-data-values.yaml --namespace harbor-test
- Error: failed to find a package with name 'harbor.tanzu.vmware.com' in namespace 'harbor-test': internalpackagemetadatas.internal.packaging.carvel.dev "harbor.tanzu.vmware.com" not found

Second issue:
Running the same command with the repository namespace included starts the installation of the Harbor package, but there is a new error.
Command: - $ tanzu package install harbor --package-name harbor.tanzu.vmware.com --version 2.5.3+vmware.1-tkg.1 --values-file harbor-data-values.yaml --namespace default
Error (a long error message before this): - kapp: Error: Timed out waiting after 5m0s. Reconcile failed: Error (see .status.usefulErrorMessage for details)
- Error: exit status 1

5. What was the name of the host/cluster/VM/datastore/vCenter that was impacted?
Answer: Data center: s34-test-tanzu-vdc, Vapps: shared01

6. What product versions (and build numbers!) are being used?
Answer: Container service Extension (CSE) 3.1.4 with Tanzu Kubernetes Grid (TKG) 1.5.4

7. What are the steps to reproduce the problem?
Answer: Follow the docs explained here for the TKG 1.5.4 using CSE 3.1.4. It seems like the documentation has not been tested for this setup, and I need to use the additional steps below to make it work.

8. What troubleshooting steps have you performed prior to contacting VMware?

After updating the –namespace to be the default namespace where the Tanzu package is located, the installation can start. The issue arrives when the installation process is stopped since some pods are not able to create or read directories.

To troubleshoot the steps done were:
1. First check the error message
Commands:
$ tanzu package installed list -n harbor-test
Output:
NAME PACKAGE-NAME PACKAGE-VERSION STATUS harbor harbor.tanzu.vmware.com 2.5.3+vmware.1-tkg.1 Reconcile failed: Error (see .status.usefulErrorMessage for details)

2. Check if pods are running:
Command:
$ -mnoto@tenantmgt:~$ k get pods -n harbor-test
Output:
NAME READY STATUS RESTARTS AGE
harbor-core-5b5c656fb8-8ql9g 0/1 Running 5 (105s ago) 8m43s
harbor-database-0 0/1 CrashLoopBackOff 6 (107s ago) 8m42s
harbor-jobservice-6d5f6c9bf-ff4st 0/1 Running 4 (111s ago) 8m43s
harbor-notary-server-7659cdcb7c-8khzx 0/1 CrashLoopBackOff 5 (2m35s ago) 8m42s
harbor-notary-signer-57f7bfd68b-lt74t 0/1 CrashLoopBackOff 5 (2m34s ago) 8m42s
harbor-portal-996fd68d-j9x5j 1/1 Running 0 8m43s
harbor-redis-0 1/1 Running 0 8m42s
harbor-registry-6b649589c5-6w6hx 2/2 Running 0 8m41s
harbor-trivy-0 0/1 CrashLoopBackOff 6 (66s ago) 8m41s

3. Check logs of not running pods:
Command: $ k logs -n harbor-test harbor-database-0
Log message:
- Defaulted container "database" out of: database, data-migrator (init), data-permissions-ensurer (init)
- ls: cannot access '/var/lib/postgresql/data/pgdata': No such file or directory
- The files belonging to this database system will be owned by user "postgres".
- This user must also own the server process.
- The database cluster will be initialized with locales
- COLLATE: en_US.UTF-8
- CTYPE: en_US.UTF-8
- MESSAGES: C
- MONETARY: C
- NUMERIC: C
- TIME: C
- The default text search configuration will be set to "english".
- Data page checksums are disabled.
- initdb: error: could not create directory "/var/lib/postgresql/data/pgdata": Permission denied

4. The problem seems to be that the pods are not able to create or read directories. By implementing the Harbor FSGroup Overlay that is recommended when using vSphere with Tanzu, we can patch the Harbor packages to give the containers permissions. But this will not fix the problem entirely. The problem will remain for the pods Harbor-database and harbor-trivy. Therefore, one must modify the overlay by giving the “Init container” for the harbor-database pod the correct directory. From "/var/lib/postgresql/data” to "/var/lib/postgresql/data/pgdata. Then create a generic secret with the overlay and patch it to the Harbor package as shown in the documentation for step 5 under “deploy Harbor into a Cluster”.

5. Delete the pods and update the Tanzu packages with the overlay:
Command:
- $ kubectl delete pods --all -n harbor-test
- $ tanzu package installed update harbor --version 2.5.3+vmware.1-tkg.1 --values-file harbor-data-values.yaml

Output:
- | Updating installed package 'harbor'
- Getting package install for 'harbor' I1028 13:54:03.873558 3244463 request.go:665] Waited for 1.040583978s due to client-side throttling, not priority and fairness, request: GET:https://10.223.30.223:6443/apis/crd.antrea.io/v1alpha1?timeout=32s
- | Getting package install for 'harbor'
- | Getting package metadata for 'harbor.tanzu.vmware.com'
- | Updating secret 'harbor-default-values'
- | Updating package install for 'harbor'
- / Waiting for 'PackageInstall' reconciliation for 'harbor'
- Error: resource reconciliation failed: I1028 11:39:46.080118 542845 request.go:665] Waited for 1.00697258s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/discovery.k8s.io/v1beta1?timeout=32s
- I1028 11:39:56.735050 542845 request.go:665] Waited for 1.049089542s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/core.antrea.tanzu.vmware.com/v1alpha2?timeout=32s
- I1028 11:40:06.740061 542845 request.go:665] Waited for 7.046122002s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/batch/v1?timeout=32s
- I1028 11:40:16.787783 542845 request.go:665] Waited for 8.797858432s due to client-side throttling, not priority and fairness, request: GET:https://100.64.0.1:443/apis/metrics.k8s.io/v1beta1?timeout=32s
- kapp: Error: waiting on reconcile deployment/harbor-notary-server (apps/v1) namespace: harbor-test:
- Finished unsuccessfully (Deployment is not progressing: ProgressDeadlineExceeded (message: ReplicaSet "harbor-notary-server-7659cdcb7c" has timed out progressing.)). Reconcile failed: Error (see .status.usefulErrorMessage for details)

6. The error shows that there is a problem with the connection of the harbor-notary-server, but now all the pods are running. And when checking on the package installation:
Command: $ tanzu package installed list -n defualt
Output:
NAME PACKAGE-NAME PACKAGE-VERSION STATUS
harbor harbor.tanzu.vmware.com 2.5.3+vmware.1-tkg.1 Reconcile succeeded

7. The Harbor package is installed and we can continue with the rest of the steps in the documentation, which hopefully will solve the connection error for the harbor-notary-server pod. Nevertheless, as we have seen here the documentation is lacking and there has to be done custom implementation to make it work with our setup.

[scott28tt]

As your post needs moving to the area for Tanzu, I have reported it to the moderators.