art – an artifact uploader

Art is a simple tool to manage artifacts to deploy.

The original use case was to transition an ops workflow from cloning full Git repositories with deploy keys, but there are probably other uses.

Art supports creating artifacts from local sources as well as remote Git repositories.

Currently supported destination types are local directories and S3, but adding support for other cloud storage (or, say, SFTP) should be trivial.

Usage

Art is configured by a combination of command line parameters and a YAML configuration file in the source; by default, art.yaml.

An example configuration file describing all current features:

wrap: all.tar
prepare:
  - make all
file_map:
  - source: bin/**
    strip: 1
    rename:
    - from: [0-9]+
      to: ''

Stanza	Description
wrap	Optional filename for the wrapfile, a tarball containing all other files.
prepare	A list of shell commands to prepare a distribution.
file_map	A list of file map entries (see below) to select files.
file_map[].source	A glob pattern to select files.
file_map[].strip	Number of path components to strip from the beginning. With a value of 1, bin/foo becomes foo, etc.
file_map[].rename[].from	A Python regular expression to rename parts of file paths.
file_map[].rename[].to	A replacement for the regular expression. Reference expressions are allowed.

Additionally, these options are typically configured on the command line.

• Source: a directory tree where the artifacts will be generated from.
• Destination: the place where artifacts will be gathered into (or rather, the base of such a place).
• Suffix: A path segment appended to the destination to create a "tag".

Given the above configuration file, and assuming make all creates bin/fibulate039 and bin/fibulatedata/foo.dat, and a destination of s3://fibulate-artifacts/fib and the suffixes prod and foo, the following files would be pushed to S3:

• s3://fibulate-artifacts/fib/foo/.manifest.json
• s3://fibulate-artifacts/fib/foo/all.tar
• s3://fibulate-artifacts/fib/foo/fibulate
• s3://fibulate-artifacts/fib/foo/fibulatedata/foo.dat
• s3://fibulate-artifacts/fib/prod/.manifest.json
• s3://fibulate-artifacts/fib/prod/all.tar
• s3://fibulate-artifacts/fib/prod/fibulate
• s3://fibulate-artifacts/fib/prod/fibulatedata/foo.dat

Misc. notes

• The .manifest.json files are autogenerated and contain information about the distribution, such as file sizes and original names.
• Wrapfiles are useful with e.g. Python wheels, which need to have their original filenames intact. Wrapping wheel artifacts into a wrapfile makes it possible to download a single file with a well-known name, unpack it and do e.g. pip install *.whl. It should be noted that the wrapfile is, on purpose, a tarbomb.
• S3 access assumes your S3 credentials are available in one of the ways supported by the Boto library.
• The --suffix-description option makes Art autogenerate an additional suffix based on the git describe --always --long --dirty output of the source, so you get a canonical URL for the particular version. (That description output is also saved in the manifest file, so you can cross-correlate between an arbitrary suffix and this.)
• For S3 destinations, you can add an ?acl=... parameter, where ... must be one of S3's pre-canned ACLs: 'private'|'public-read'|'public-read-write'|'authenticated-read'|'aws-exec-read'|'bucket-owner-read'|'bucket-owner-full-control' The default is whatever is the default for your bucket.

Dev / Testing

Use py.test in general.

Self-test:

pip install -e .
art --local-source . --dest ./dist/artifact --suffix latest