Support for custom image builds?

[agile]

We are using a custom image build in order to support installing dependencies from private repos as described here but when attempting to use the extension with that setup, ie: something like this using a modification of the example in those docs:

# syntax=quay.io/astronomer/airflow-extensions:v1
LABEL maintainer="Astronomer <[email protected]>"
ARG BUILD_NUMBER=-1
LABEL io.astronomer.docker=true
LABEL io.astronomer.docker.build.number=$BUILD_NUMBER
LABEL io.astronomer.docker.airflow.onbuild=true
# Install OS-Level packages
COPY packages.txt .
USER root
RUN apt-get update && cat packages.txt | xargs apt-get install -y
USER astro

FROM stage1 AS stage2
# Install Python packages
ARG PIP_EXTRA_INDEX_URL
ENV PIP_EXTRA_INDEX_URL=${PIP_EXTRA_INDEX_URL}
COPY requirements.txt .
RUN pip install --no-cache-dir -q -r requirements.txt

# Create VENV
ARG EX_REQUIREMENTS_FILE=ex-requirements.txt
COPY ${EX_REQUIREMENTS_FILE} .
PYENV 3.8 ex ${EX_REQUIREMENTS_FILE}

FROM stage1 AS stage3
# Copy requirements directory
COPY --from=stage2 /usr/local/lib/python3.9/site-packages /usr/local/lib/python3.9/site-packages
COPY --from=stage2 /usr/local/bin /home/astro/.local/bin
# Also copy the venvs from stage2
COPY --from=stage2 /home/astro/.venv /home/astro/.venv
ENV PATH="/home/astro/.local/bin:$PATH"

COPY dags/ dags/
COPY include/ include/
COPY plugins/ plugins/
COPY tests/ tests/

This does not work because with the extension it is appending additional statements after the final COPY statements defined at the end of the build that replicate tasks performed in prior stages such as installing packages.txt and requirements.txt but without the necessary credentials to do so successfully.

In the meantime we're working on replicating what the extension does manually in our build but would be nice if we didn't need to do that.

[ashb]

@agile Gotcha. Do you want private repos for the modules installed in the venv, or in the normal requirements.txt? I.e. would you like a way to pass the credentials/flags to the RUN pip install -r requirements.txt/to not have those steps automatically added?