Python Build Changes

Change

Starting 03/01/2021 (March 1st 2021) Python build and runtime workflow is changing for the apps built on App Service. This may be a Breaking Change for your app.

Current build process

The app builds are carried out in a build container/agent by a Build System called Oryx. This container produces artifacts which are stored on an NFS volume and later run by a Runtime container defined by the user(eg: Python 3.8).

Oryx running in the Build Agent installs the dependencies listed in requirements.txt into a virtual environment named antenv. This build is carried out in a local build directory. This virtual environment produced by the above build is compressed and extracted by the runtime at the root of the container to enable the users to have an optimal build and startup time.

This caused a problem where the Virtual Environment could not resolve certain app dependencies. The reason for this is that during the creation of the Virtual Environment pip hardcodes certain paths for dependencies. Since, the builds are carried out in an Build Container and Run on a Runtime container at a different path, this causes a mismatch between the directory structure for the virtual environment.

Change/New Build+Runtime Process

The build workflow remains the same with respect to the use of a temporary build directory as described above. But now the whole app output(including the Virtual Environment) is compressed into a tarball and stored at /home/site/wwwroot with a manifest file oryx-manifest.toml. This manifest file describes how the build was carried out in the build container. The runtime container on startup reads this manifest and recreates a local copy of the folder structure on every instance. This change also ensures that the runtime during the app startup will already have the virtual environment activated and users can run commands like migrate without the need to activate the environment.

As a part of this change, the app would no longer run from /home/site/wwwroot but would have a dynamic runtime directory. This path can be retrieved by using the variable $APP_PATH.

Potential impact

We do not expect apps execution to break due to this change. However, for those customers who use a bash/shell script as the startupScript, there could be potential compatibility issues when the new builds are carried out after the next platform update.

Few scenarios to consider

• Apps which have hardcoded the path to their virtual environment as "/antenv/bin/activate" in their startup scripts may get an error. We recommend that these customers rebuild the app, review their startup script and modify all these paths relative to $APP_PATH.

• Apps which are writing to a local path relative to the app path would no longer have their data persisted. Only the data written under the /home directory is persisted. This can include libraries which persist session state.

• Files used by the app can be viewed using the SSH functionality of the SCM site and navigating to the $APP_PATH location. As the files are no longer in /home/site/wwwroot, they will not be viewable through other UI elements on the SCM site.

Impacted Versions:

Python 2.7, 3.6, 3.7, 3.8

How can this be tested?

Testing on Azure:

1. Presently, the new behavior is opt-in. To try the new version, you can add the app setting: KUDU_BUILD_VERSION=1.0.0.
2. If you want to continue to pin to the old version - This can be done by adding an app setting: KUDU_BUILD_VERSION=0.0.1. This app setting would ensure the build behavior remains the same even when we change default behavior on 03/01/2021.

Testing Locally:

Alternatively, you can download our latest images and test your app’s runtime locally.

• Steps to test locally:

1. Pull the Build Image: docker pull mcr.microsoft.com/appsvc/kudulite:20201229.1
2. Create a local folder which would be used by both the build and the runtime container.
3. Start the build image locally: docker run -v <local-directory>:/home -e KUDU_BUILD_VERSION=1.0.0 -p <local-build-port>:8181 mcr.microsoft.com/appsvc/kudulite:20201229.1
4. Clone the App Locally: git clone http://localhost:<local-build-port>/localsite.git
5. Push the app code to the build container: git push
6. Start the runtime container: docker run -p <local-runtime-port>:8080 -v <local-directory>:/home mcr.microsoft.com/appsvc/python:<ver>_20201229.1
7. Browse your app: http://localhost:<local-runtime-port>/

The latest image tags are:

• Build Container: mcr.microsoft.com/appsvc/kudulite:20201229.1
• Python 2.7: mcr.microsoft.com/appsvc/python:2.7_20201229.1
• Python 3.6: mcr.microsoft.com/appsvc/python:3.6_20201229.1
• Python 3.7: mcr.microsoft.com/appsvc/python:3.7_20201229.1
• Python 3.8: mcr.microsoft.com/appsvc/python:3.8_20201229.1

We are hopeful that this set of changes will ensure a better experience for our customers on App Service Linux.

How can I file an issue?

Open an issue in ImageBuilder Issues for any issues encountered on this regard.

Thanks,
App Service Linux team,
Microsoft.