Mark evaluations with (outdated) invalid algorithm as failed

[amickan]

When re-evaluating submissions for a challenge phase, it is possible that a submission is no longer valid for the challenge. This can happen if the interface for the phase changed in the meantime (i.e. the algorithm inputs are different now). Re-evaluating such a submission will currently leave the linked evaluation stuck in Executing algorithm state, because:

• in create_algorithm_jobs we filter_civs_for_algorithm based on the inputs that are set on the algorithm. So even if new archive items are passed to create_algorithm_jobs those get filtered out --> no new jobs to create --> set_evaluation_inputs gets called
• set_evaluation_inputs does nothing if not self.inputs_complete, where we check that there is exactly 1 successful job per valid archive item

The newly created evaluation for such submission should be marked as Failed (or Cancelled?) with an appropriate error message (i.e. "Algorithm inputs don't match those defined for the phase"). We should check and catch that early, probably in create_algorithm_jobs.

First discovered here: https://github.com/DIAGNijmegen/rse-grand-challenge-admin/issues/428#issuecomment-2615173942

[jmsmkn]

I think you mean evaluations and not submissions throughout? create_algorithm_jobs is the correct place to check.

[amickan]

Hm, we're technically re-evaluating submissions here (because the evaluation container changed), which creates a new evaluation with the updated method, which then gets stuck in the executing algorithm state. Or am I wrong?

I have updated the description to reflect that.

[amickan]

And it's the submitted algorithm that is no longer a valid submission for the phase.

[jmsmkn]

Indeed, but a submission does not have a status, only evaluations do.

[amickan]

Indeed, but a submission does not have a status, only evaluations do.

True, which is why I suggested The newly created evaluation for such submission should be marked as Failed