[BUG] Remote Publication Cluster State Download Task Management.

[SwethaGuptha]

Describe the bug

A node at any point in time should only have one task in-going for downloading latest cluster state. For a cluster with remote-publication, this management of task has gaps due to which the node can have more than one ongoing cluster state download task. We can re-visit the task management to address the common scenarios like:

1. If a node receive a new cluster state download task with the version that is same as the ongoing cluster state download task then it can be de-duped.
2. If the new cluster state task has higher version than the version of ongoing download task then the ongoing task must be cancelled.

Related component

Cluster Manager

To Reproduce

1. Create a cluster with data nodes having 2vcpu
2. Create 10k indices
3. Simulate node drop

Expected behavior

Have only one cluster state download task at any point in time.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[rajiv-kv]

[Triage Attendees - 1, 2]

Thanks for filing the issue. Please feel free to propose options as to how download task could be tracked.
The ClusterState downloads are in a dedicated threadpool and cannot be preempted from execution.