Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hooks: add hook for intake #853

Merged
merged 11 commits into from
Jan 14, 2025
Merged

Conversation

mo-dkrz
Copy link
Contributor

@mo-dkrz mo-dkrz commented Jan 14, 2025

Add hook for intake to ensure that package's data and binaries of the drivers plugin are collected.

@mo-dkrz mo-dkrz marked this pull request as draft January 14, 2025 13:04
@rokm
Copy link
Member

rokm commented Jan 14, 2025

In general, please avoid using collect_all - instead, you should use collect_submodules, collect_data_files and/or collect_dynamic_libs (plus copy_metadata) as required.

Do any of these plugins actually contain binaries and/or data files? The couple from the start of the list that I've looked at all seem to be pure-python packages without binaries and/or data.

@mo-dkrz mo-dkrz marked this pull request as ready for review January 14, 2025 13:51
@mo-dkrz mo-dkrz requested a review from rokm January 14, 2025 13:51
.github/workflows/pr-test.yml Outdated Show resolved Hide resolved
_pyinstaller_hooks_contrib/stdhooks/hook-intake.py Outdated Show resolved Hide resolved
news/853.new.rst Outdated Show resolved Hide resolved
tests/test_libraries.py Outdated Show resolved Hide resolved
@rokm
Copy link
Member

rokm commented Jan 14, 2025

Hmm, actually, it looks like intake discovers its plugins via intake.drivers entry points. Does your use-case still work if you simplify the hook into:

# hook-intake.py
from PyInstaller.utils.hooks import collect_entry_point
datas, hiddenimports = collect_entry_point('intake.drivers')

@mo-dkrz
Copy link
Contributor Author

mo-dkrz commented Jan 14, 2025

Hmm, actually, it looks like intake discovers its plugins via intake.drivers entry points. Does your use-case still work if you simplify the hook into:

# hook-intake.py
from PyInstaller.utils.hooks import collect_entry_point
datas, hiddenimports = collect_entry_point('intake.drivers')

it was completely to-the-point, yes it works perfectly on my use case and rationally speaking it has to work for other cases as well since as you correctly said intake discovers them through the driver entrypoint. This could simplify the hook a lot. I will commit the change req in a moment

@mo-dkrz mo-dkrz requested a review from rokm January 14, 2025 16:21
@rokm
Copy link
Member

rokm commented Jan 14, 2025

Looks like the PR was not opened with "Allow changes from maintainers", so I cannot push my follow-up changes.

Therefore, can you apply the following changes yourself:


  1. The copyright year in the hook should be 2025

  1. The news fragment should be updated to better reflect the nature of the new hook:
    Add hook for ``intake`` to collect its plugins (registered via the ``intake.drivers`` entry-point).

  1. Insert
from PyInstaller import isolated

at line 17 of the test file, and modify the tests:

@importorskip('intake')
def test_intake(pyi_builder):
    pyi_builder.test_source("""
        import intake

        print(f"intake version: {intake.__version__}")
        catalog = intake.Catalog()
    """)


@importorskip('intake')
def test_intake_plugins(pyi_builder, tmp_path):
    # Get the list of plugins in unfrozen python
    @isolated.decorate
    def _get_intake_plugins():
        import intake
        return sorted(list(intake.registry))

    plugins_unfrozen = _get_intake_plugins()
    print(f"Unfrozen plugins: {plugins_unfrozen}")

    # Obtain list of plugins in frozen application.
    output_file = tmp_path / "output.txt"

    pyi_builder.test_source("""
        import sys
        import intake

        with open(sys.argv[1], 'w') as fp:
            for plugin in intake.registry:
                print(f"{plugin}", file=fp)
    """, app_args=[str(output_file)])

    with open(output_file, "r") as fp:
        plugins_frozen = sorted(line.strip() for line in fp)
    print(f"Frozen plugins: {plugins_frozen}")

    assert plugins_frozen == plugins_unfrozen

  1. Insert

          # Install additional intake plugins for test_intake_plugins
          pip show -qq intake && pip install --prefer-binary intake-xarray

at line 162 of .github/workflows/pr-test.ym.

@mo-dkrz
Copy link
Contributor Author

mo-dkrz commented Jan 14, 2025

done

@bwoodsend bwoodsend merged commit 8d1e528 into pyinstaller:master Jan 14, 2025
26 checks passed
github-actions bot pushed a commit to wxx9248/Pickle-Rush that referenced this pull request Jan 16, 2025
…25.0 (#121)

Bumps
[pyinstaller-hooks-contrib](https://github.com/pyinstaller/pyinstaller-hooks-contrib)
from 2024.11 to 2025.0.
<details>
<summary>Release notes</summary>
<p><em>Sourced from <a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/releases">pyinstaller-hooks-contrib's
releases</a>.</em></p>
<blockquote>
<h2>v2025.0</h2>
<p>Please see the <a
href="https://www.github.com/pyinstaller/pyinstaller-hooks-contrib/tree/v2025.0/CHANGELOG.rst">changelog</a>
for more details</p>
</blockquote>
</details>
<details>
<summary>Changelog</summary>
<p><em>Sourced from <a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/blob/master/CHANGELOG.rst">pyinstaller-hooks-contrib's
changelog</a>.</em></p>
<blockquote>
<h2>2025.0 (2025-01-16)</h2>
<p>New hooks</p>
<pre><code>
* Add hook for ``fsspec`` to collect the package's submodules
and ensure the protocol plugins are working.
(`[#856](pyinstaller/pyinstaller-hooks-contrib#856)

&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/856&gt;`_)
* Add hook for ``intake`` to collect its plugins (registered via the
``intake.drivers`` entry-point).
(`[#853](pyinstaller/pyinstaller-hooks-contrib#853)

&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/853&gt;`_)
* Add hook for ``ruamel.yaml`` to collect its plugins, and ensure that
  plugins' ``__plug_in__`` modules are collected as source .py files
(which is necessary for their discovery).
(`[#844](pyinstaller/pyinstaller-hooks-contrib#844)

&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/844&gt;`_)
* Add hook for ``sam2`` (Segment Anything Model 2).
(`[#847](pyinstaller/pyinstaller-hooks-contrib#847)

&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/847&gt;`_)
* Add hook for ``zarr`` to collect the package's metadata.
(`[#855](pyinstaller/pyinstaller-hooks-contrib#855)

&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/855&gt;`_)
<p>Updated hooks
</code></pre></p>
<ul>
<li>Revise the search for OpenSSL shared library and
<code>ossl-modules</code> directory
in the <code>cryptography</code> hook, in order to mitigate issues with
unrelated
copies of OpenSSL ending up being pulled into the build. Most notably,
the hook should not be searching for OpenSSL shared library when
<code>cryptography</code> PyPI wheel is installed, because those ship
with
extensions that are statically linked against OpenSSL.
(<code>[#846](pyinstaller/pyinstaller-hooks-contrib#846)
&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/846&gt;</code>_)</li>
<li>Rewrite <code>pygraphviz</code> hook to fix discovery and collection
of <code>graphviz</code>
files under various Linux distributions, in Anaconda environments
(Windows, Linux, and macOS), and msys2 environments (Windows).
(<code>[#849](pyinstaller/pyinstaller-hooks-contrib#849)
&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/849&gt;</code>_)</li>
<li>Update <code>dask</code> hook to collect template files from
<code>dask/widgets/templates</code>
directory; these file become mandatory when using
<code>dask.array</code> and
<code>jinja2</code> is available.
(<code>[#852](pyinstaller/pyinstaller-hooks-contrib#852)
&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/852&gt;</code>_)</li>
<li>Update <code>triton</code> hook for compatibility with
<code>triton</code> &gt;= 3.0.0; the
hook should now collect backend-specific modules and data files from
<code>triton.backends</code>.
(<code>[#848](pyinstaller/pyinstaller-hooks-contrib#848)
&lt;https://github.com/pyinstaller/pyinstaller-hooks-contrib/issues/848&gt;</code>_)</li>
</ul>
</blockquote>
</details>
<details>
<summary>Commits</summary>
<ul>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/06928e9cfd1f1003892002cc26e79e5d2d957189"><code>06928e9</code></a>
Release v2025.0</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/2c162e9417db3ca686b11662485dc42bc71d66aa"><code>2c162e9</code></a>
hooks: add hook for fsspec (<a
href="https://redirect.github.com/pyinstaller/pyinstaller-hooks-contrib/issues/856">#856</a>)</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/6c7df30fe94341457032ced8ac2ee79e5a6b6cd6"><code>6c7df30</code></a>
hooks: add hook for zarr (<a
href="https://redirect.github.com/pyinstaller/pyinstaller-hooks-contrib/issues/855">#855</a>)</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/8d1e52848ec92d194604b613bf4916cfeefc290e"><code>8d1e528</code></a>
hooks: add hook for intake (<a
href="https://redirect.github.com/pyinstaller/pyinstaller-hooks-contrib/issues/853">#853</a>)</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/0a113f8a5b696e949c1096c3e593cbfecf4e6106"><code>0a113f8</code></a>
Scheduled weekly dependency update for week 02 (<a
href="https://redirect.github.com/pyinstaller/pyinstaller-hooks-contrib/issues/851">#851</a>)</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/717223dd3580bb5a9cf993609b3f7dba58e18d6e"><code>717223d</code></a>
hooks: dask: collect template files from dask/widgets/templates</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/10c53079a55e33bcf82d26ffca1e9a2ca19bc759"><code>10c5307</code></a>
hooks: prettytable does not query its version from metadata after
3.12.0</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/f73f04aa62c2b752f6d2bfa8d77fe0fcddcf4ac2"><code>f73f04a</code></a>
ci: add Homebrew headers and shared libraries to search path</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/ef78a3f40c397fdf4160a0d33ec087057cfc954a"><code>ef78a3f</code></a>
tests: enable pygraphviz tests</li>
<li><a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/commit/fef9193c4403cf46d4c37edcd2c3f3b9e4ed9165"><code>fef9193</code></a>
hooks: rewrite pygraphviz hook</li>
<li>Additional commits viewable in <a
href="https://github.com/pyinstaller/pyinstaller-hooks-contrib/compare/v2024.11...v2025.0">compare
view</a></li>
</ul>
</details>
<br />


[![Dependabot compatibility
score](https://dependabot-badges.githubapp.com/badges/compatibility_score?dependency-name=pyinstaller-hooks-contrib&package-manager=pip&previous-version=2024.11&new-version=2025.0)](https://docs.github.com/en/github/managing-security-vulnerabilities/about-dependabot-security-updates#about-compatibility-scores)

Dependabot will resolve any conflicts with this PR as long as you don't
alter it yourself. You can also trigger a rebase manually by commenting
`@dependabot rebase`.

[//]: # (dependabot-automerge-start)
[//]: # (dependabot-automerge-end)

---

<details>
<summary>Dependabot commands and options</summary>
<br />

You can trigger Dependabot actions by commenting on this PR:
- `@dependabot rebase` will rebase this PR
- `@dependabot recreate` will recreate this PR, overwriting any edits
that have been made to it
- `@dependabot merge` will merge this PR after your CI passes on it
- `@dependabot squash and merge` will squash and merge this PR after
your CI passes on it
- `@dependabot cancel merge` will cancel a previously requested merge
and block automerging
- `@dependabot reopen` will reopen this PR if it is closed
- `@dependabot close` will close this PR and stop Dependabot recreating
it. You can achieve the same result by closing it manually
- `@dependabot show <dependency name> ignore conditions` will show all
of the ignore conditions of the specified dependency
- `@dependabot ignore this major version` will close this PR and stop
Dependabot creating any more for this major version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this minor version` will close this PR and stop
Dependabot creating any more for this minor version (unless you reopen
the PR or upgrade to it yourself)
- `@dependabot ignore this dependency` will close this PR and stop
Dependabot creating any more for this dependency (unless you reopen the
PR or upgrade to it yourself)


</details>

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants