[BUG] Remote state read thread-pool build up on download task timeout

[SwethaGuptha]

Describe the bug

With remote publication, in case of an full cluster state download timeout, threads continue to work on the read tasks in thread pool even though the cluster state download attempt has been marked as failed. This causes the queue build up and impacts the threads available for subsequent downloads.

Related component

Cluster Manager

To Reproduce

1. Create a cluster with data nodes having 2vcpu
2. Create 10k indices
3. Simulate node drop

Expected behavior

Tasks from timed-out cluster state download should be terminated.

Additional Details

Plugins
Please list all plugins currently enabled.

Screenshots
If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context
Add any other context about the problem here.

[rajiv-kv]

[Triage Attendees - 1, 2]

Thanks for filing the issue. Please feel free to raise a fix for the clean-up