[GOBBLIN-2148] Add temporal workflow cancel support

[abhishekmjain]

Dear Gobblin maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Gobblin JIRA issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
  • https://issues.apache.org/jira/browse/GOBBLIN-2148

Description

• Here are some details about my PR, including screenshots (if applicable):
  Currently, when GaaS Temporal job are cancelled, only the Azkaban job is stopped, which stops the subsequent YARN application. However, the Temporal job that is submitted by the YARN application is not stopped.
  As part of this PR, we have implemented the cancel flow which should cancel the Temporal workflow upon cancellation of Azkaban job.
  The support for cancellation is added in base class GobblinTemporalJobLauncher. The workflowId is needed for cancellation, thus some changes are made to initialize the workflowId in submitJob, which can then be used for cancellation.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
• testCancelWorkflowIfFailed
• testCancelWorkflowIfCompleted
• testCancelWorkflowIfRunning

I have also tested the cancel flow in personal Az project for all 3 cases:

• Cancellation after job completion: Doesn't trigger cancel event to Temporal
• Cancellation before worker started: Results in termination event to Temporal
• Cancellation during workflow execution:
  a) If cancellation completes within 3 seconds, successful cancellation of Temporal workflow
  b) If cancellation takes more than 3 seconds, termination of Temporal workflow

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[Will-Lo]

Commenting on the PR description here, but 3 seconds for cancellation seems quite aggressive. Is there a way for us to increase this timeout to say, 30 seconds?