[BUG] Update metadata and job documents in one run when handle change policy

[bowenlan-amzn]

Regarding this part of the logic for handling change policy

index-management/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexRunner.kt

Lines 784 to 795 in 930157b

	/*
	* Try to update the ManagedIndexMetaData in cluster state, we need to do this first before updating the
	* ManagedIndexConfig because if this fails we can fail early and still retry this whole process on the next
	* execution whereas if we do the update to ManagedIndexConfig first we lose the ChangePolicy on the job and
	* could fail to update the ManagedIndexMetaData which would put us in a bad state
	* */
	val updated = updateManagedIndexMetaData(updatedManagedIndexMetaData)
	if (!updated.metadataSaved || policy == null) return
	// Change the policy and user stored on the job from changePolicy, this will also set the changePolicy to null on the job
	savePolicyToManagedIndexConfig(managedIndexConfig, policy.copy(user = changePolicy.user))

I feel it's possible to combine these 2 into one bulk call, so they always fail or succeed at same time.

If only metadata update succeed, it seems possible to fall into this check and stop running

index-management/src/main/kotlin/org/opensearch/indexmanagement/indexstatemanagement/ManagedIndexRunner.kt

Lines 306 to 307 in 930157b

	if (managedIndexMetaData.hasVersionConflict(managedIndexConfig)) {
	val info = mapOf("message" to "There is a version conflict between your previous execution and your managed index")

[vikasvb90]

Bulk doesn't guarantee atomic ingestion of all documents it carries. There can be partial failures. Looking at partial failures of the response will also not work because rollback (deletion) of ingested doc or ingestion of failed doc is again not guaranteed. The only possible solution of this which requires a major effort would be to maintain a parent-child relationship to designate a group of docs and accordingly modify search queries and subsequent updates.

[bowenlan-amzn]

Got it.
Just note down another approach — moving metadata as a field into job document. The small downside is anything in job document change will trigger the listener from Job Scheduler to reschedule the job.