lal spec

lal is a simple command line tool that works on folders with a valid manifest.json, and accepts the following commands:

lal fetch - fetch dependencies from manifest.json into INPUT
lal update - update arbitrary dependencies into INPUT
lal status - print current INPUT dependencies with origin
lal verify - verify manifest validity + verify flat lockfile dependency tree
lal env - control build environment
lal build [name] - run canonical build in docker with current directory mounted
lal shell - enter container environment mounting current directory
lal run - runs a non-build script through lal shell
lal configure - generate configuration file
lal init - generate manifest file
lal stash - copies current OUTPUT to cache
lal upgrade - performs an upgrade check
lal clean - cleans up cache directory
lal export - obtain a raw tarball from artifactory
lal query - list versions of a component on artifactory
lal remove - remove components from INPUT and manifest.json
lal publish - publish release builds to artifactory
lal propagate - works out steps to propagate dependencies

Manifest

A per-repo file. Format looks like this (here annotated with illegal comments):

{
  "name": "libwebsockets",  // name of repo
  "environment": "centos",  // name of environment found in the lal config
  "supportedEnvironments": ["centos", "xenial"],
  "components": {           // map of components and default configuration
    "libwebsockets": {
      "defaultConfig": "release",
      "configurations": ["release", "clang"]
    },
    "websockets_tests": {
      "defaultConfig": "coverage",
      "configurations": ["coverage", "release", "clang"]
    }
  },
  "dependencies": {
    "ciscossl": 42
  },
  "devDependencies": {
    "gtest": 42
  }
}

Lockfile

A per-build file auto-generated by lal build and will reduce the lockfiles generated from dependencies to provide aggregated information.

{
  "name": "edonus",
  "config": "release",
  "container": {
    "name": "edonusdevelopers/centos_build",
    "tag": "2016-04-03"
  },
  "environment": "centos",
  "tool": "0.10.0", // from `lal --version`
  "version": "5",  // from --with-version or "EXPERIMENTAL-{randomhex}"
  "sha": "0ee0ee225d107076ed4b00368805d987baac9c4d", // from --with-sha
  "dependencies": {
    "libwebsockets": {
      "name": "libwebsockets",
      "config": "release",
      "container": {
        "name": "edonusdevelopers/centos_build",
        "tag": "2016-04-03"
      },
      "environment": "centos",
      "version": "47",
      "tool": "0.10.0",
      "dependencies": {
        "ciscossl": {
          "name": "ciscossl",
          "config": "release",
          "container": {
            "name": "edonusdevelopers/centos_build",
            "tag": "2016-04-03"
          },
          "environment": "centos",
          "version": "200",
          "tool": "0.10.0",
          "dependencies": {}
        }
      }
    },
    "ciscossl": {
      "name": "ciscossl",
      "config": "release",
      "container": {
        "name": "edonusdevelopers/centos_build",
        "tag": "2016-04-03"
      },
      "environment": "centos",
      "version": "200",
      "tool": "0.10.0",
      "dependencies": {}
    }
  }
}

This struct is fully recursive in the sense that every value in the dependencies hash is also a valid lockfile.

Config

A per-machine configuration file in ~/.lal/config generated by lal configure. This is an example of environments, artifactory settings and mounts for a hypothetical edonus team.

{
  "artifactory": {
    "server": "https://engci-maven-master.cisco.com/artifactory",
    "group": "CME-release",
    "vgroup": "https://engci-maven.cisco.com/artifactory/CME-group"
  },
  "cache": "/home/devuser/.lal/cache",
  "environments": {
    "centos": { "container": "edonusdevelopers/centos_build", "tag": "latest" },
    "xenial": { "container": "edonusdevelopers/build_xenial", "tag": "latest" }
  },
  "upgradeCheck": "2016-06-30T12:20:10.126707483+00:00",
  "mounts": [
    {
      "src": "/mnt/tools",
      "dest": "/tools",
      "readonly": true
    }
  ]
}

Every repository is required to specify the name of one of the specified environments in their manifest.json.

The upgradeCheck value is updated automatically by lal upgrade.

.lal/opts

A per-repo temporary file primarily for lal env that overrides the current environment.

{
  "env": "xenial"
}

When this file exists, every lal command will use this environment rather than the default one. It is created by lal-env.

This file is intended to be gitignored because it overrides manifest.environment.

Caching

The local cache is populated by fetches from the registry, or calls to stash them.

~/.lal/cache $ tree .
├── environments
│   ├── centos
│   │   └── ciscossl
│   │       └── 6
│   │           └── ciscossl.tar.gz
│   └── xenial
│       └── ciscossl
│           └── 6
│               └── ciscossl.tar.gz
└── stash
    └── ciscossl
        └── asan
            └── ciscossl.tar.gz

Sources:

environments are components from the registry under a specific environment namespace
stash are tarballs of OUTPUT of builds when doing lal stash <name>

Versioning

As implied by the structure of the Manifest, Lockfile, and cache directories, the only versioning scheme supported by lal is a monotonically increasing integer sequence.

Subcommands

lal status

Provides list of dependencies currently in INPUT. If they are not in the manifest they will be listed as extraneous. If they are stashed dependencies they will be listed in yellow origin

Extra flags:

--full or -f: print the full dependency tree
--origin or -o: print version and environment origin of artifact
--time or -t: print build time of artifact

Alias: lal ls

lal env [environment]

Subcommand that controls the current environment. This is a sticky, repo-wide setting stored in $PWD/.lal/opts when working with non-standard environments.

$ lal env
xenial
# every lal command will defer the environments key in `manifest.json` by default

$ lal env set zesty # writes { "environment": "zesty" } to .lal/opts
$ lal env update # invokes docker pull of the zesty image
# now every lal command will warn if `manifest.environment != lal env`
$ lal fetch # fetches from zesty
$ lal build # build using zesty components
$ lal shell # enters zesty shell
$ lal run unit-test websocket_server_test 5 asan # runs unit test in zesty shell

$ lal env reset # deletes `.lal/opts` if it exists

# lal now behaves as usual, doing all commands in the described environment in manifest.json

This is an advanced command for people developing on temporary, non-standard environments. If you would like to override the environment on a command-by-command basis, there is an option for that as well.

lal build [name] [flags]

Runs the BUILD script in the current directory in the container.

If no arguments are suppplied it will run ./BUILD $name $config where name is the value of name in the manifest, and config is the name of the component's defaultConfig.

E.g. lal build in say a gtest repo will probably call ./BUILD gtest release in the container.

lal build will run lal verify and abort if this fails. When using stashed components, you should build with --simple-verify or -s for short. This will allow stashed versions to pass, but still not cripple the verifier so that you accidentally include things built in different environments.

Any further verify blocks can be overridden with -f or --force. There are very few legit developer reasons why you would want to completely ignore lal verify, but maybe you have such a special case.

Release specific flags:

--release: Generate a tarball and lockfile in ./ARTIFACT folder after building
--with-version n: Jenkins specific option which will specify lockfile version
--with-sha str: Jenkins specific option which will set revision id

Typically jenkins would do:

lal build --release --with-version=$BUILD_NUMBER --with-sha=$(git rev-parse HEAD)

And publish that with lal publish.

Passing configuration flags:

--config=name: Passes a named config to BUILD as $2.

This allows multiple blessed configurations of the same component, i.e. lal build dme-unit-tests --config=asan and lal build dme-unit-tests --config=debug. Both are valid provided dme-unit-tests provides those configurations in the components part of the manifest.

lal update [components..]

Find the latest available version of a component that is available in all currently supportedEnvironments from the manifest.

lal update component [--save]: fetches the latest version of a component. The optional --save flag will also update the manifest file locally.
lal update component=version [--save]: fetches a specific version. If the version is parsable as an integer, it is fetched from artifactory. Otherwise, it is assumed to be a stashed version.

Many component or component=version arguments can be used in one invocation.

lal fetch

lal fetch [--core]: fetches all versions corresponding to the manifest from the registry and puts them into INPUT. The optional --core flag will disregard any devDependencies.

Any components already found in INPUT are reused if they are present at the right version and correct environment.

Any extraneous versions found in INPUT are removed.

lal shell

Enters an interactive shell in the container corresponding to the environment key in the manifest mounting the current directory.

Useful for experimental builds using internal scripts in a repo. Assumes you have run lal fetch or equivalent so that INPUT is ready for this.

lal shell should basically run:

docker run \
  -v $PWD:/home/lal/volume \
  -w /home/lal/volume \
  --user lal \
  -it ${SOME_CONTAINER} \
  /bin/bash

Note that the config can be edited to pass in extra mounts.

lal shell should allow passing in trailing arguments to run arbitrary commands:

lal shell ./BUILD something # runs this command rather than /bin/bash and removes -i
lal shell -p ./BUILD something # same, but adds --privileged to docker
lal shell bash -c "cmd1; cmd2" # multiple commands in one go
lal shell --print-only prints above command
lal shell --print-only ./BUILD something # prints what would have been done

lal shell should also allow making it easy to forward the X11 socket:

lal shell -X # mounts /tmp/.X11-unix, ~/.Xauthority and forwards the DISPLAY evar

Host networking should be convenience flag:

lal shell -n # passes --net=host to docker.

A combination of these two flags allows forwarding X through ssh and lal:

ssh -X somemachine # ssh with X11 forwarding
lal sh -X -n xcalc # run xcalc and forward X11 all the way through ssh

The X11 forwarding setup requires xhost installed, and also xauth installed if you need it through ssh as well. You may need to run xhost local:docker to allow docker to access X.

Alias: lal sh

lal run [name]

Runs scripts in the local .lal/scripts/ folder via lal shell. Because lal shell mounts $PWD, the scripts folder can contain parametrised scripts such as:

#!/bin/bash
# contents of .lal/scripts/subroutine
main() {
  echo "hi $1 $2"
}

completer() {
  echo "foo bar"
}

Which could be invoked with lal run subroutine there mr, which would echo hi there mr in the container. An optional completer function can be supplied for autocomplete of values.

Alias: lal script

lal stash [name]

Stashes the current OUTPUT folder to in ~/.lal/cache/stash/${component}/${NAME} for future reuse. This can be put into another repository with lal update component=name

Alias: lal save

lal verify

Helper command used by lal build exposed for convenience/sanity. Verifies that:

manifest.json exists in $PWD and is valid JSON
dependencies in INPUT match manifest.json
the dependency tree is flat
dependencies in INPUT contains only published dependencies
dependencies in INPUT were built using the correct environment

lal build normally guards on this command.

An optional --simple or -s can be passed to lal verify to not check for published dependencies and a flat dependency tree.

lal configure [defaults]

Sets up a default config with a set of pre-configured defaults from a seperately supplied file with default values:

lal configure configs/edonus.json

Will set up the docker environments, artifactory downnload settings, and common mounts to scan for for the edonus team.

To tweak different settings, edit ~/.lal/config after the original configure call, then manage it yourself.

lal init [environment]

Creates a basic manifest.json in the current directory, assuming directory name as the name of the main component.

A -f flag can be supplied to force overwrite the manifest.

lal init -f centos

lal upgrade

Performs an upgrade check of lal. If new versions are found, it reports how to upgrade your lal tool. This is normally checked daily. But it can be done manually with this command.

This is currently disabled awaiting a redesign.

lal clean

Deletes artifacts in the cache directory older than 14 days. The day is configurable with -d <days>.

lal export [component]

Exports a build artifact from the storage backend in the current directory or a directory of choice.

The component can be either the name of the component for latest version, or suffixed with =version for a specific version:

lal -e xenial export gtest -o mystorage/
test -f mystorage/gtest.tar.gz

lal -e xenial export liblzma=6
test -f ./liblzma.tar.gz

NB: export does not read the manifest.json for environment overrides.

lal query [component]

Lists the availble versions in the storage backend that were built in a speific environent.

lal -e xenial query libwebsockets

NB: query does not read the manifest.json for environment overrides.

lal remove [components..]

Removes and optionally saves a removal of a component from INPUT and the manifest.

lal remove libwebsockets --save
lal remove gtest --save-dev

Note you can only use one of save or save-dev at a time. Without either save flag, this subcommand simply deletes the corresponding subdirectory of INPUT.

Alias: lal rm

lal publish

Publishes a release build in the local ARTIFACT subdirectory provided it is built with a correct version and proper credentials are presented.

lal env set xenial
lal fetch
lal build libldns --release --with-version=20 --with-sha=$(git rev-parse HEAD)
# specific builds should push immediately (even if there are more builds)
lal publish libldns

The publish command will upload to a bucket named after the environment used to build it (found in ./ARTIFACT/lockfile.json). It will also verify that the version is set with --with-version.

The uploaded artifact will in this case end up the following location:

https://artifactory.host/artifactory/group/env/xenial/libldns/20/

If you have more supportedEnvironments then lal update will look in all the buckets corresponing to your environments before finding a version that can be useg in all environments.

lal propagate [component]

Retraces a dependency tree in reverse to figure out steps needed to propagate a leaf dependency properly. This is useful for satisfying the full version strictness checks of lal verify in a large dependency tree (recall that we enforce a flat dependency tree).

Given a component with the following example dependency tree:

~ > mycomponent on master $ lal ls -f
mycomponent
├── openssl
├─┬ libcurl
│ ├── c-ares
│ └── openssl
├─┬ cucumber-cpp (dev)
│ └── gtest
├── gtest (dev)
├─┬ qt
│ ├── openssl
│ └── freetype
└── zlib

If we need to propagate a new version of openssl, we need to do it in two stages:

~ > mycomponent on master $ lal propagate openssl
Assuming openssl has been updated:
Stage 1:
- update [openssl] in libcurl
- update [openssl] in qt
Stage 2:
- update [libcurl, openssl, qt] in mycomponent

Every step in each stage is paralellizable, but every stage must wait for the previous stage. A simple web service to perform this scheduling and upgrade can be set up if you are willing to hook this up to your CI infrastructure.

Universal Options

--help or -h
-v
--env or -e

Note that -v is a global option that gradually increases verbosity (allows multiple uses), and goes before subcommands.

lal update zlib # update with only standard logging (info!, warn!, and error!)
lal -v fetch # fetch with debug! messages
lal -vv build # build with debug! and trace! messages

The --env flag will override the default environment in both manifest.environment and .lal/opts for the current command:

lal --env xenial fetch
lal -e xenial verify
lal -e xenial build
lal -e xenial shell
lal -e xenial script unit-test websocket_server_test 5 asan

Because these commands are often used together you can instead make it sticky with lal env.

For full autogenerated help of all flags of every subcommand help can be requested:

lal build -h
lal help build # equivalent

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPEC.md

SPEC.md

lal spec

Manifest

Lockfile

Config

.lal/opts

Caching

Versioning

Subcommands

lal status

lal env [environment]

lal build [name] [flags]

lal update [components..]

lal fetch

lal shell

lal run [name]

lal stash [name]

lal verify

lal configure [defaults]

lal init [environment]

lal upgrade

lal clean

lal export [component]

lal query [component]

lal remove [components..]

lal publish

lal propagate [component]

Universal Options

Files

SPEC.md

Latest commit

History

SPEC.md

File metadata and controls

lal spec

Manifest

Lockfile

Config

.lal/opts

Caching

Versioning

Subcommands

lal status

lal env [environment]

lal build [name] [flags]

lal update [components..]

lal fetch

lal shell

lal run [name]

lal stash [name]

lal verify

lal configure [defaults]

lal init [environment]

lal upgrade

lal clean

lal export [component]

lal query [component]

lal remove [components..]

lal publish

lal propagate [component]

Universal Options