Initially our “naive” resolution was a easy polling logic each x minutes
Checking if successful/failure file was created by SageMaker.
Based mostly on a file existence or not — we knew what inference occasion(s) succeeded/failed and up to date their corresponding job standing:
Sadly, the Implementation by the AWS SageMaker staff has edge-cases — It’s doesn’t create a “failure” file for the next situations:
- Occasions timing out inside AWS managed queue of person requests
- Didn’t end processing inside timeout restrict (e.g max 1h)
This left us BLIND for some uncommon but trivial edge-cases inflicting us:
- couldn’t discover a success/failure file
- couldn’t perceive the end result of an inference activity
- couldn’t resolve if a job was completed
This was the second we understood we wanted an structure change.
