forked from tensorflow/tflite-micro
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
feat(py): add scripts for building and uploading to PyPI (tensorflow#…
…2236) Add mechanism and scripts for building and uploading the Python distribution package `tflite_micro` to PyPI. These scripts are intended mainly for use by CI when generating packages for distribution via for PyPI, and won't be used by most developers. Building a package for local use is still done via a normal Bazel build. Heavily comment the scripts with rationale and technical details of the implementation. Make significant updates to python/tflite_micro/README.md which explain building, installing, and uploading the package to PyPI. Leave some cleanup of the existing text for later. Add a build setting `--//python/tflite_micro:compatibility_tag` for setting :whl's platform compatibility tag. Unfortunately, it cannot derived automatically from the execution environment by the current implementation of @rules_python. BUG=part of tensorflow#1484
- Loading branch information
Showing
5 changed files
with
354 additions
and
19 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| # Use the Python Packaging Authority's reference build environment | ||
| # for binary extensions. Binary extensions are typically built and distributed | ||
| # for each target Python version and OS platform. The reference build | ||
| # environment contains Python installations for each version, and a C/C++ | ||
| # toolchain specified for maximum compatibility among x86_64 Linux paltforms. | ||
| FROM quay.io/pypa/manylinux_2_28_x86_64 | ||
|
|
||
| # Install bazel (via bazelisk) | ||
| ENV BAZELISK=https://github.com/bazelbuild/bazelisk/releases/download/v1.18.0/bazelisk-linux-amd64 | ||
| ENV BAZEL=/usr/local/bin/bazel | ||
| RUN curl --output $BAZEL --location $BAZELISK && chmod 755 $BAZEL | ||
|
|
||
| # Append the location of the C/C++ toolchain to the default PATH, where | ||
| # bazel expects to find it. The reference environment provides the location | ||
| # (typically somewhere under /opt) in DEVTOOLSET_ROOTPATH. | ||
| RUN echo "PATH="${PATH}:/${DEVTOOLSET_ROOTPATH}"" >>/etc/environment |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,114 @@ | ||
| #!/bin/sh | ||
|
|
||
| # Copyright 2023 The TensorFlow Authors. All Rights Reserved. | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| set -e | ||
|
|
||
| OUT_DIR_DEFAULT=bazel-pypi-out | ||
|
|
||
| USAGE="$(basename $0) <python-tag> [<output-directory>] | ||
| Build a Python wheel for public release to PyPI using a special Docker build | ||
| container. Uses bazel, but does not pollute the WORKSPACE's default cache. | ||
| <python-tag> must be one of the supported interpreters: | ||
| cp310 | ||
| cp311 | ||
| <output-directory> defaults to $OUT_DIR_DEFAULT. | ||
| " | ||
|
|
||
| case "$1" in | ||
| cp310|cp311) | ||
| PY_TAG=$1 | ||
| OUTDIR=$(realpath ${2:-$OUT_DIR_DEFAULT}) | ||
| mkdir -p $OUTDIR | ||
| break | ||
| ;; | ||
| *) | ||
| echo usage: "$USAGE" >&2 | ||
| exit 1 | ||
| esac | ||
|
|
||
| SRCDIR=$(realpath .) | ||
| if ! test -f $SRCDIR/WORKSPACE; then | ||
| echo "error: must run from the top of the source tree" >&2 | ||
| exit 1 | ||
| fi | ||
|
|
||
| # Remove Bazel's workspace symlinks so they'll be rewritten below, pointing into | ||
| # OUTDIR. | ||
| find . -maxdepth 1 -type l -name bazel-\* | xargs rm -f | ||
|
|
||
| # Build the Docker image from its source file. Don't pollute the public list of | ||
| # images by tagging; just use the image's ID. | ||
| DOCKERFILE=python/tflite_micro/pypi_build.dockerfile | ||
| IMAGE_ID_FILE=$OUTDIR/image-id | ||
| docker build - --iidfile $IMAGE_ID_FILE <$DOCKERFILE | ||
| IMAGE_ID=$(cat $IMAGE_ID_FILE) | ||
|
|
||
| # Build the Python package within an ephemeral container. | ||
| docker run \ | ||
| --rm \ | ||
| --interactive \ | ||
| --mount type=bind,source=$SRCDIR,destination=$SRCDIR \ | ||
| --mount type=bind,source=$OUTDIR,destination=$OUTDIR \ | ||
| --workdir $SRCDIR \ | ||
| --env USER=$(id -un) \ | ||
| $IMAGE_ID \ | ||
| /bin/bash -s -e -x -u \ | ||
| <<EOF | ||
| # Setup the Python compatibility tags. The PY_ABI always matches the Python | ||
| # interpreter tag. The platform tag is supplied by the build image in the | ||
| # environment variable AUDITWHEEL_PLAT. | ||
| PY_ABI=$PY_TAG | ||
| PY_PLATFORM=\$AUDITWHEEL_PLAT | ||
| PY_COMPATIBILITY=${PY_TAG}_\${PY_ABI}_\${PY_PLATFORM} | ||
| # Link the desired Python version in the PATH where bazel will find it. The | ||
| # build image contains many differnet Python installations as options. | ||
| ln -sf /opt/python/$PY_TAG-$PY_TAG/bin/* /usr/bin | ||
| # Bazelisk fails if it can't check HOME for a .rc file. | ||
| export HOME=$OUTDIR | ||
| # Bazelisk, bazel, and pip all need a writable cache directory. | ||
| export XDG_CACHE_HOME=$OUTDIR/cache | ||
| # Build the wheel via bazel, using the Python compatibility tag matching the | ||
| # build environment. Drop root privledges and run as the invoking user. | ||
| # Relocate the bazel cache to keep the cache used for each toolchain | ||
| # separate. | ||
| setpriv --reuid=$(id -u) --regid=$(id -g) --clear-groups \ | ||
| bazel --output_user_root=$OUTDIR/$PY_TAG-out \ | ||
| build \ | ||
| //python/tflite_micro:whl.dist \ | ||
| --//python/tflite_micro:compatibility_tag=\$PY_COMPATIBILITY | ||
| # Test, in the container environment | ||
| setpriv --reuid=$(id -u) --regid=$(id -g) --clear-groups \ | ||
| bazel --output_user_root=$OUTDIR/$PY_TAG-out \ | ||
| test \ | ||
| //python/tflite_micro:whl_test \ | ||
| --//python/tflite_micro:compatibility_tag=\$PY_COMPATIBILITY | ||
| EOF | ||
|
|
||
| # Make the output directory tree writable so it can be removed easily by the | ||
| # user with `rm -rf $OUTDIR`. Bazel leaves it write-protected. | ||
| chmod -R +w $OUTDIR | ||
|
|
||
| # Copy the generated wheel file to the root of the $OUTDIR. | ||
| cp bazel-bin/python/tflite_micro/whl_dist/*.whl $OUTDIR | ||
| echo "Output:\n$(ls $OUTDIR/*.whl)" |
Oops, something went wrong.