You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
_parse_progress_from_resource_manager() assumes that there will be at most one job running on a cluster at the same time, which is wrong now that clusters can run steps concurrently.
If we know a step's StartTime from the ListSteps API, that seems to only be a few seconds off of Start Time in the resource manager UI. So that's a way we could possibly match up step progress correctly.
It would be really nice if there EMR API would tell us the mapping between EMR step IDs and YARN application IDs, but so far I haven't found one.
The text was updated successfully, but these errors were encountered:
Since we now have code to talk to the resource manager API, we can guess the application ID for the step from the apps API (based on start time) and then get its progress from the app API.
_parse_progress_from_resource_manager()
assumes that there will be at most one job running on a cluster at the same time, which is wrong now that clusters can run steps concurrently.If we know a step's
StartTime
from theListSteps
API, that seems to only be a few seconds off ofStart Time
in the resource manager UI. So that's a way we could possibly match up step progress correctly.It would be really nice if there EMR API would tell us the mapping between EMR step IDs and YARN application IDs, but so far I haven't found one.
The text was updated successfully, but these errors were encountered: