IFC-873 Remove shared storage requirements for git workers

[gmazoyer]

Fixes #4789
Fixes #4296

[codspeed-hq]

CodSpeed Performance Report

Merging #4867 will not alter performance

_{Comparing gma-20241106-4789 (c521456) with develop (c5e1456)}

Summary

✅ 10 untouched benchmarks

[ogenstad]

Just a small comment on the naming here. We've always followed the naming conventions so that the class matches the routing key.

So following the current convention it would be:

    "refresh.git.clone": RefreshGitClone,
    "refresh.git.fetch": RefreshGitFetch,

This might sound a bit off and it could be that we should find other names. In this case it would probably have been better if we hadn't already "used" git.*.* as a binding.

[dgarros]

I think it make sense to create a new binding for all git messages like git.*.* that would be reserved for the task-workers
Currently all consumers even the API server are subscribing to refresh.*.*

[ogenstad]

It's a bit different:

class InfrahubMessageBus:
    DELIVER_TIMEOUT: int = 30 * 60  # 30 minutes
    worker_bindings: list[str] = [
        "check.*.*",
        "event.*.*",
        "finalize.*.*",
        "git.*.*",
        "refresh.webhook.*",
        "request.*.*",
        "send.*.*",
        "schema.*.*",
        "transform.*.*",
        "trigger.*.*",
    ]
    event_bindings: list[str] = ["refresh.registry.*"]

As it is not anything under refresh.registry.* would be sent to all worker types both API workers and Prefect workers. And anything under git.*.* would be sent to the competing work queue that gets pick up by the first Prefect worker available. Which means that in order to use anything under "git." for broadcasts we'd have to modify the existing routing (i.e. have a call to .unbind() for that queue at startup, alternatively if it's a fresh instance of RabbitMQ everything would be recreated from scratch)

[dgarros]

thanks for the clarification
I think we need to convert git.*.* to a broadcasts queue and ensure that only the task workers will subscribe to it

[ogenstad]

We'd need to move some of the tasks, for instance we wouldn't want git.file.get to be a broadcast task. That one we'd only want to arrive at one worker. I.e. it should be sent to the queue that all workers subscribe to but only one worker should process the task. Perhaps we can spend a few minutes on this in the backend meeting.

[dgarros]

Looks great