We use API to publish (mostly update) actions to vRO from local disk using POST to /vco/api/actions endpoint
Works as expected, but it runs slower over time
Currently it takes ~20s to post single action on our most busy server.
The content of /services-logs/prelude/vco-app/file-logs/vco-server-app.log
2023-03-20T11:28:37.041Z INFO vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.service.impl.ScriptModuleManagementServiceImpl - Action 'FgApi' saved.
2023-03-20T11:28:37.381Z DEBUG vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.git.StagingRepositoryService - Pulling /usr/lib/vco/app-server/data/git/__SYSTEM.git
2023-03-20T11:28:37.607Z DEBUG vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.git.StagingRepositoryService - Staging area[/usr/lib/vco/app-server/data/git/__SYSTEM.git] head: 9ff0ad400c3b188fda7f9960552b02c12202e975
2023-03-20T11:28:38.219Z WARN vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.git.PushCommandBuilder - Remote refs/heads/main push returned status: UP_TO_DATE
2023-03-20T11:28:38.706Z DEBUG vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.git.StagingRepositoryService - Staging area[/usr/lib/vco/app-server/data/git/__SYSTEM.git] head: 9ff0ad400c3b188fda7f9960552b02c12202e975
2023-03-20T11:28:59.002Z INFO vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.git.PushCommandBuilder - Remote refs/heads/main push returned status: OK
2023-03-20T11:28:59.004Z WARN vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.service.version.ContentVersionManagementServiceImpl - Acquisition of the content update lock and pushing the changes took 19 sec
2023-03-20T11:28:59.005Z INFO vco [host='vco-app-5898689979-4dpcp' thread='contentUpdatesExecutor-3' user='' org='' trace=''] {} com.vmware.o11n.service.impl.ScriptModuleManagementServiceImpl - Action 'FgApi' saved (via import).
It looks like the problem is in git repo located in /data/vco/usr/lib/vco/app-server/data/git/__SYSTEM.git
Counting all git commits is ~47K.
We have 2 other vRO servers. The timing problem is ~proportional to number of git commits:
- Server A #commits: 2.5K ==>. time <1s
- Server B #commits: 17K ==> time 2s
- Server C #commits: 47K ==> time 20s
How can we remediate this issue?
