[Batch] Add n_max_attempts against infinite retries/preemptions

[cjllanwarne]

Adds:

• A n_max_attempts field to the client job object (defaulting to 20)
• A n_max_attempts reader in the front_end which stores the field into the database
  • And a database field to store it in
• A change in the batch driver to fail jobs which have already had their allowed maximum number of attempts

Some judgement call justification -

• Why fail the job on attempt "n+1" instead of during the tidy-up of job "n"?
  • Personally I prefer a code-driven max-out over adding any more business logic into database procedures.
  • The mark job complete action is triggered from the worker VM not the driver, and I didn't want to add max-out business logic into the worker either.

[chrisvittal]

Looks great. Just a couple of questions.

[chrisvittal]

Why is this default 404?

[cjllanwarne]

Oops, this was a placeholder while testing this out. This is slightly different semantically from the defaults in the code (this is a value to assign any pre-existing batches - which had de-facto infinite retries - instead of a default going forward), but for the sake of consistency, I think changing this to 20 as well is probably most sensible

[chrisvittal]

If n_max_attempts defaults to 20, why is it 5 here?

[cjllanwarne]

Good spot, this should probably be 20 too

[patrick-schultz]

Could you give some context on the two different places in the driver we're checking the number of attempts? What are those two places, and why do we need two?

Also, have you done anything to test how much adding joins with the attempts table slows down those queries?

[cjllanwarne]

@patrick-schultz -

1. Assuming you mean pool.py and job-private.py as "the two different places in the driver" - those are two types of a more general "instance collection" concept that the driver has. Pool represents the standard machine type pools that jobs can be assigned to, and job-private looks after jobs that need their own dedicated (ie "job private") VMs
2. I haven't... both that and double checking the ALGORITHM=INSTANT claims (given that jobs is so large) are things I'd like to do. Do you have any ideas on the best way to do that? Maybe I could copy @ehigham 's tactic of duplicating the production DB temporarily and running custom queries against it? Or is there an easier/better/cheaper way?