ERROR: Python dependency not found (on Windows)

[scott-guthridge]

We're trying to build a wheel for a Cython project through github workflows on Windows. The build is failing pretty early on. The question spans github actions, cibuildwheel, and meson-python -- hopefully, it's not off-topic here.

Top level meson.build begins with:

project('libvna', 'cython',
  version: '0.0.5-13',
  license: 'GPL-3',
  default_options: [
    'buildtype=debugoptimized',
    'b_ndebug=if-release',
    'c_std=c99',
  ],
)

py_mod = import('python')
py3 = py_mod.find_installation(pure: false)
dep_py = py3.dependency()                                               <----- line 13 FAILING HERE
dep_vna = dependency('libvna', version: '>=0.3.9')

srcdir = meson.source_root()
testdir = join_paths(srcdir, 'tests')
builddir = meson.current_build_dir()

Here's the error:

The Meson build system
Version: 1.3.1
Source dir: D:\a\pylibvna-cicdtest\pylibvna-cicdtest
Build dir: D:\a\pylibvna-cicdtest\pylibvna-cicdtest\.mesonpy-mc6o1dxn
Build type: native build
Project name: libvna
Project version: 0.0.5-13
Cython compiler for the host machine: cython (cython 3.0.7)
Host machine cpu family: x86_64
Host machine cpu: x86_64
Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\cibw-run-wcok67kg\cp310-win_amd64\build\venv\Scripts\python.exe)
Run-time dependency python found: NO
..\meson.build:13:13: ERROR: Python dependency not found

We're following the example of SciPy pretty closely. Our project is simpler (no Fortran or C++), but we don't see what we're doing differently in this area.

Here's part of the github workflows script:

yaml
jobs:
  build_wheels:
    name: Build wheel for ${{matrix.python[0]}}-${{matrix.buildplat[1]}} ${{matrix.buildplat[2]}}
    runs-on: ${{ matrix.buildplat[0] }}
    strategy:
      matrix:
        buildplat:
        # - [ubuntu-22.04, manylinux, x86_64]
        # - [macos-11, macosx, x86_64]
        - [windows-2019, win, AMD64]
        python:
        # - ["cp38",  "3.8"]
        # - ["cp39",  "3.9"]
        - ["cp310", "3.10"]
        # - ["cp311", "3.11"]
        # - ["cp312", "3.12"]
    steps:
    - name: Checkout Code
      uses: actions/checkout@v3

    - uses: actions/setup-python@v4
      with:
        python-version: 3.9

    - name: Install Rtools (Windows)
      # Rtools provides pkg-config on Windows
      if: ${{ runner.os == 'Windows' }}
      run: |
        choco install rtools -y --no-progress --force --version=4.0.0.20220206
        echo "c:\rtools40\ucrt64\bin;" >> $env:GITHUB_PATH

    - name: Install cibuildwheel and twine
      run: |
        python -m pip install --upgrade pip
        pip install cibuildwheel twine

    - name: Install Libvna (Windows)
      if: ${{ runner.os == 'Windows' }}
      run: |
        python3 .github/tools/download-libvna-windows
        New-Item -Path C:\opt -ItemType Directory -Force
        Expand-Archive -Path libvna-windows-x86_64.zip -DestinationPath C:\opt
        echo 'C:\opt\libvna-windows-x86_64\bin;' >> $env:GITHUB_PATH
        Write-Host "GITHUB_PATH=$env:GITHUB_PATH"

    - name: Build Wheels
      env:
        CIBW_BUILD: ${{ matrix.python[0] }}-${{ matrix.buildplat[1] }}*
        CIBW_ARCHS: ${{ matrix.buildplat[2] }}
        CIBW_ENVIRONMENT: "PIP_NO_BUILD_ISOLATION=false PIP_PRE=1"
        CIBW_ENVIRONMENT_PASS_LINUX: RUNNER_OS
        CIBW_PRERELEASE_PYTHONS: True
        CIBW_BEFORE_BUILD_LINUX: |
          echo "WHOAMI=`whoami`"
          echo "UNAME=`uname -a`"
          set -e
          yum install -y gcc libyaml-devel
          pip install requests
          python3 .github/tools/install-libvna-rpm
        CIBW_ENVIRONMENT_WINDOWS: >
          PKG_CONFIG_PATH=c:/opt/libvna-windows-x86_64/lib/pkgconfig
          PIP_PRE=1
          PIP_NO_BUILD_ISOLATION=false
        CIBW_BEFORE_BUILD_WINDOWS:
          pip install numpy>=2.0.0.dev0 meson-python cython!=3.0.3 ninja &&
          bash {project}/tools/test-script &&
          powershell -Command "Write-Host GITHUB_PATH=$env:GITHUB_PATH"
        CIBW_BEFORE_BUILD_MACOS: |
          pip install numpy>=2.0.0.dev0 meson-python cython!=3.0.3 ninja
          brew tap scott-guthridge/extra
          brew install scott-guthridge/extra/libvna
      run: python -m cibuildwheel --output-dir wheelhouse

Linux and MacOs builds are running without errors.

[dnicolodi]

The only thing that stands out as possibly causing the issue is setting PKG_CONFIG_PATH. Can you try to do not set this environment variable ans see if the build goes past this point? Other than that, we would need the Meson build log to see what exactly goes wrong.

Other than that, why are you setting the PIP_NO_BUILD_ISOLATION=false environment variable? This should be the default. And if you do this, why are you installing numpy, meson-python, cython, and ninja? These would need to be installed again in the build environment anyhow.

[rgommers]

I don't see it either. On Windows I'd expect the detection to go via the python-config script rather than pkg-config, so probably PKG_CONFIG_PATH doesn't do anything either way.

The answer will be in meson-logs/meson-log.txt in the build directory, which can be obtained with something like this step in your GitHub Actions workflow file:

- name: Meson Log
  shell: bash
  if: always()  # or failure()
  run: |
    cat build/meson-logs/meson-log.txt

Extra complication: you have to convince cibuildwheel to build in a fixed build dir so that the log isn't cleaned up before you can get to it in this debug step. See at the bottom of #246 for how.

In the meson-log.txt content, you should see content like:

Program python3 found: YES (/Users/rgommers/mambaforge/envs/scipy-dev/bin/python3.10)
Searching for 'python-3.10' via pkgconfig lookup in LIBPC
Pkg-config binary for 1 is not cached.
...

which will tell you how Meson is searching for Python.h & co.

[scott-guthridge]

Thanks for the quick replies. Not setting PKG_CONFIG_PATH at all, or setting it to exactly what scipy sets it to made no difference.

why are you setting the PIP_NO_BUILD_ISOLATION=false environment variable? This should be the default. And if you do this, why are you installing numpy, meson-python, cython, and ninja?

Those were both copied from SciPy. I keep pulling in more and more of their settings and environment trying to determine which are actually needed. I notice that the numpy build scripts look a bit cleaner -- they may be a better example to follow.

Got the log:

 cat build/meson-logs/meson-log.txt
  shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
  env:
    pythonLocation: C:\hostedtoolcache\windows\Python\3.9.13\x64
    PKG_CONFIG_PATH: C:\hostedtoolcache\windows\Python\3.9.13\x64/lib/pkgconfig
    Python_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.9.13\x64
    Python2_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.9.13\x64
    Python3_ROOT_DIR: C:\hostedtoolcache\windows\Python\3.9.13\x64
Build started at 2024-01-12T07:17:25.505091
Main binary: C:\Users\runneradmin\AppData\Local\Temp\build-env-xms2ccuf\Scripts\python.exe
Build Options: -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md '--native-file=D:\a\pylibvna-cicdtest\pylibvna-cicdtest\build\meson-python-native-file.ini'
Python system: Windows
The Meson build system
Version: 1.3.1
Source dir: D:\a\pylibvna-cicdtest\pylibvna-cicdtest
Build dir: D:\a\pylibvna-cicdtest\pylibvna-cicdtest\build
Build type: native build
Project name: libvna
Project version: 0.0.5-13
-----------
Detecting compiler via: `cython -V` -> 0
stdout:
Cython version 3.0.8
-----------
Running compile:
Working directory:  C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3
Code:
 print("hello world")
-----------
Command line: `cython C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3\testfile.pyx -o C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3\output.exe --fast-fail` -> 0
stderr:
C:\Users\runneradmin\AppData\Local\Temp\build-env-xms2ccuf\lib\site-packages\Cython\Compiler\Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3\testfile.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)
-----------
Cython compiler for the host machine: cython (cython 3.0.8)
-----------
Detecting compiler via: `cython -V` -> 0
stdout:
Cython version 3.0.8
-----------
Using cached compile:
Cached command line:  cython C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3\testfile.pyx -o C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3\output.exe --fast-fail 

Code:
 print("hello world")
Cached compiler stdout:
 
Cached compiler stderr:
 C:\Users\runneradmin\AppData\Local\Temp\build-env-xms2ccuf\lib\site-packages\Cython\Compiler\Main.py:381: FutureWarning: Cython directive 'language_level' not set, using '3str' for now (Py3). This has changed from earlier releases! File: C:\Users\RUNNER~1\AppData\Local\Temp\tmpql_436h3\testfile.pyx
  tree = Parsing.p_module(s, pxd, full_module_name)

Cython compiler for the build machine: cython (cython 3.0.8)
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: x86_64
Host machine cpu: x86_64
Target machine cpu family: x86_64
Target machine cpu: x86_64
'utf-8' codec can't decode byte 0x90 in position 2: invalid start byte
Unusable script 'C:\\Users\\runneradmin\\AppData\\Local\\Temp\\build-env-xms2ccuf\\Scripts\\python.exe'
Program python found: YES (C:\Users\runneradmin\AppData\Local\Temp\build-env-xms2ccuf\Scripts\python.exe)
Dependency lookup for python with method 'sysconfig' failed: no toolchain found
Run-time dependency python found: NO 

..\meson.build:13:13: ERROR: Python dependency not found

I'm trying to understand what it means. Could the 'utf-8' error be a UTF-8 vs UTF-16 thing? But I don't provide that python.exe -- it's getting set up by some part of the build -- and aren't .exe files binaries, anyway? Is that causing the no toolchain error, or are these unrelated?

[rgommers]

What @dnicolodi suggests below should indeed fix this I think. If you change the first line of your meson.build to:

project('libvna', 'c', 'cython',

things should work.

The dependency detection for Python is going to look for the Python.h header and it needs a C (or C++) compiler for that.

[dnicolodi]

I really don't get how threading is supposed to work in GitHub Discussions: when I use "quote reply" I get a new thread instead of a reply.

Anyhow, would there be any downside in having Meson also pull in a C compiler when cython is specified as project language? I don't see any. Even if the Cython output is then compiled as C++, I think all compilers able to compile Cython code have both C and C++ front-ends integrated. I can see how actually implementing this in Meson may look like. @eli-schwartz, what do you think?

[eli-schwartz]

I believe we already do that. That's not the problem. The problem is that detecting python ahead of time may need a C compiler but only on Windows. Windows does NOT provide pkg-config files for python, so we fall back to sysconfig and probing with find_library.

[dnicolodi]

What I meant is to automatically translate project('foo', 'cython') into the equivalent of project('foo', 'cython', 'c'). Do you see any drawback in doing that? The only use case that would be affected by that are project where cython is only used as a compiler to transpile to C without producing object files, but in that case I expect users to simply use find_program('cython').

[dnicolodi]

I keep pulling in more and more of their settings and environment trying to determine which are actually needed.

I would go the other way around: start with a barebone setup and add the things you need one after the other. Building wheels using meson-python does not require anything special. For a simple example you can look here https://github.com/dnicolodi/python-siphash24/blob/master/.github/workflows/wheels.yml

[dnicolodi]

project('libvna', 'cython',
  version: '0.0.5-13',
  license: 'GPL-3',
  default_options: [
    'buildtype=debugoptimized',
    'b_ndebug=if-release',
    'c_std=c99',
  ],
)

It may not be the source of your issue, but you need to also have c or cpp as project languages to compile Cython extensions. Cython only transpiles to C, you need also the C or C++ compiler toolchain to be initialized.

[eli-schwartz]

Cython will automatically add C or C++ if needed, IIRC, but only after the target is created.

[dnicolodi]

Yes, you are right. I had a vague recollection of this being an issue, and indeed I already filed mesonbuild/meson#11389 Where we didn't come to any conclusion. However, having c in the project languages is a requirement when using cython on Windows, because the sysconfig discovery method is always attempted to lookup the Python dependency on Windows.

[dnicolodi]

Having the c toolchain is required also on other platforms when pkg-config is not available. This is actually how I first bumped into the issue, see mesonbuild/meson#11389.

[scott-guthridge]

Thank you, everyone. Adding 'c' to the project line fixed this problem.
Now onto the next...