Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: ✨ numpy-array-use-type-var flag #188

Merged
merged 7 commits into from
Nov 21, 2023

Conversation

ringohoffman
Copy link
Contributor

Fixes #187

Adds the flag --numpy-array-use-type-var which reformats the pybind11-generated numpy.ndarray[numpy.float32[m, 1]] annotation as numpy.ndarray[tuple[M, Literal[1]], numpy.dtype[numpy.float32]] using TypeVars; this allows for type-checker supported shape checking

Notable changes are:

  • The addition of FixNumpyArrayDimTypeVar, corresponding to the --numpy-array-use-type-var flag
  • TypeVar_ struct (appended _ to avoid conflicting with from typing import TypeVar), and new Module.type_vars field
  • FixTypingTypeNames.parse_annotation_str() was extracted into a helper method so that it could be additionally be called on every item of result.parameters in order to expand the Literal inside the result.parameters into e.g. typing_extensions.Literal
  • FixScipyTypeArguments was added to remove type arguments from anything from scipy.sparse, since they are not generic and do not accept any type arguments; the shape annotations provided by --numpy-array-wrap-with-annotated are left intact
  • FixNumpyDtype forces numpy.dtype without parameters to be numpy.dtype[typing.Any] since it expects one type argument

reformat numpy.ndarray[numpy.float32[m, 1]] as numpy.ndarray[tuple[M, Literal[1]], numpy.dtype[numpy.float32]] using TypeVars; this allows for shape checking
@ringohoffman
Copy link
Contributor Author

Let me know what you want to do in the way of testing this. Does it make sense to create another set of sub-folders under each of tests/stubs/python-*/pybind11-master?

  • tests/stubs/python-3.7/pybind11-master/numpy-array-wrap-with-annotated
  • tests/stubs/python-3.7/pybind11-master/numpy-array-use-type-var

?

Would we loop over these flags in tests/check-demo-stubs-generation.sh?

it accepts a type argument, and if it is unknown it is typing.Any
@sizmailov
Copy link
Owner

Does it make sense to create another set of sub-folders under each of tests/stubs/python-*/pybind11-master?

* `tests/stubs/python-3.7/pybind11-master/numpy-array-wrap-with-annotated`

* `tests/stubs/python-3.7/pybind11-master/numpy-array-use-type-var`

Unfortunately yes. Let's limit the tests for numpy-array-use-type-var to the latest pybind11 and python.

Would we loop over these flags in tests/check-demo-stubs-generation.sh?

No, let's add an extra command line option to check-demo-stubs-generation.sh for the numpy array transformation.
Let's create a separate entry for numpy-array-use-type-var in gh-actions yml.

since we will be testing both --numpy-array-wrap-with-annotated and --numpy-array-use-type-var (at least for 3.12 on pybind11 master), we put each in separate subfolders to make the tests easy to configure
…on.sh

namely adding the option to be able to test --numpy-array-use-type-var
@ringohoffman
Copy link
Contributor Author

I gave it my best effort! Please let me know what you would have me change!

@sizmailov
Copy link
Owner

Thanks! Looks good to me!

The last minor thing. Let's make the CI job name for numpy-array-use-type-var different from the default one.

@sizmailov sizmailov enabled auto-merge (squash) November 21, 2023 09:02
@sizmailov sizmailov merged commit 60728d3 into sizmailov:master Nov 21, 2023
16 checks passed
@sizmailov
Copy link
Owner

Many thanks for contribution. I'll publish it as v2.4

@ringohoffman ringohoffman deleted the numpy-shape-typevar branch November 21, 2023 09:06
@ringohoffman
Copy link
Contributor Author

Does this affect the status of #113 or #115?

@sizmailov
Copy link
Owner

Indeed, #115 could be closed.
I think #113 is not affected, since nptyping uses its own types.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Option to use TypeVars for numpy.ndarray's shape type argument
2 participants