Get job participating clients from Job Meta

[yanchengnv]

Fixes # .

Description

At the start of a job, the CJ sends a sync message to the server to get information about the job's participating clients. This mechanism has a few issues:

• CJs are started at about the same time. If there are many clients (e.g. there could be thousands of clients for handling edge devices), the SP will be under a lot of pressure to process so many messages at the same time.

• In case of Simulator, if the number of subprocesses is smaller than the number of clients, then some clients will be swapped in and out. Each time a client is swapped in, it needs to send the sync message again. But if this happens after the job is already done, it will fail, as reported by QA.

This PR gets rid of the sync mechanism from CJs. Once the SP starts to run a job, it already knows what clients will participate in the job. SP then includes such info in the job's metadata in the message to CPs to start the job. Once started, the CJ no longer needs to send the sync message to SP to get participating clients since it can get this info from the job's metadata.

This PR also cleans up some log messages by changing their log level to DEBUG.

NOTE: this PR does not change SJ's logic to get job clients from SP via the sync message, due to the large scope of changes. Since there is always only a single SJ for each job, it won't cost much for the SP to process one message from the SJ.

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Quick tests passed locally by running ./runtest.sh.
• In-line docstrings updated.
• Documentation updated.

[yanchengnv]

/build

[yanchengnv]

/build