Pilot release procedure

Pilot development usually starts in personal GitHub repositories. When a pilot version is deemed stable, a pull request to the official PanDA Pilot GitHub area, specifically to the 'next' branch. When it is time for release, the 'next' branch is merged with the 'master' branch.

For ATLAS, an independent cron job checks for updates in the 'master' branch (once per hour) and pulls down the source code when needed. The cron job script creates a new source area in cvmfs and also creates a corresponding tarball. It is not used in production, however, until the RC prod version is stable (below). When the pilot_version for PQ.ALL in CRIC is updated, all ATLAS PQs will use the corresponding version. There is also a default pilot version on CVMFS that need to be updated manually, in case the pilot_version set in CRIC does not exist. The pilot wrapper should default to the CVMFS version.

Whenever a build of the development version of the pilot ('the dev pilot') is stable, the RC dev pilot is updated. The RC dev pilot is used for automated testing using HammerCloud on all active PanDA queues. No dev pilot is ever released until it has been running RC test jobs to minimise problems on the grid. When a release is in progress, i.e. after the GitHub PR to the master branch, an RC prod pilot version is created and copied manually to the PanDA server. The pilot wrapper will run 1% of all production jobs with this version. When it is believed to work properly, the pilot_version for PQ.ALL in CRIC is updated.

A pilot release consists of the following steps

1. Master branch update, which triggers the automatic build of pilot tarball on CVMFS (pilot3-VERSION_STRING.tar.gz)
2. Tag creation (see GitHub master branch code area)
3. Pypi update
4. Send release notes to [email protected]
5. Update RC prod pilot (pilot3-rc.tar.gz), copy to PanDA servers
6. ATLAS: When jobs are running with the new pilot version, send an update to the [email protected] and [email protected] lists with all relevant code changes (explanations, motivations, related JIRA tickets, etc)
7. ATLAS: When RC prod pilot version is stable, update PQ.ALL in CRIC