feat: add hpa and async worker to enterprise-catalog.

[Jacatove]

Description

Enterprise catalog has been enabled in our platform and there is a need to anticipate future intensive usage of this service, that's why HPA and worker support is being added. Currently, an operation like adding an enterprise customer catalog to a an enterprise customer, takes a considerable amount of time, this is thought as a result of running tasks synchronously.

Changes

• Support HPA.
• Add enterprise-catalog worker deployment for both k8s and docker-compose.

Test 1: You can trigger an async task by adding an enterprise customer catalog to a an enterprise customer, the update_catalog_metadata_task task is executed by the worker.

Test 2:

1. Pull the current branch
2. Install the plugin pip install -e tutor-contrib-enterprise and verify that the plugin is installed tutor plugins list
3. Run tutor config save
4. Run tutor local launch
5. Open the logs tutor local logs --follow
6. Run tutor local do sync-enterprise-catalog-metadata
7. Verify that the worker is working

[Squirrel18]

Remove these limits as we don't have any idea of the ideal limits. @Jacatove

[Jacatove]

@Squirrel18 "Any HPA target can be scaled based on the resource usage of the pods in the scaling target. When defining the pod specification the resource requests like cpu and memory should be specified. This is used to determine the resource utilization and used by the HPA controller to scale the target up or down". Copied from https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/#support-for-resource-metrics.

So I think it is worth to give some thought about resources:

This is what I have gathered so far. I ran the enterprise-catalog-update-content-job and this is the result.

MEMORY:

CPU

Checking the resourses used by the pods, I got this:

enterprise-catalog-64554f5895-8kl52                         0m           501Mi           
enterprise-catalog-update-content-job-1703786342004-kmzmg   60m          89Mi            
jacato:~$ kubectl top pod | grep enterprise
enterprise-catalog-64554f5895-8kl52                         0m           501Mi           
enterprise-catalog-update-content-job-1703786342004-kmzmg   5m           93Mi