[#5139] feat: (python-client): Support GCS fileset in the python GVFS client #5160
Conversation
|
This PR is not ready for review until #5079 is merged. |
|
@jerryshao @xloya |
|
@xloya would you please help to review this code, thanks. |
| @@ -27,6 +27,8 @@ | |||
| from fsspec.implementations.arrow import ArrowFSWrapper | |||
| from fsspec.utils import infer_storage_options | |||
| from pyarrow.fs import HadoopFileSystem | |||
| from pyarrow.fs import GcsFileSystem | |||
There was a problem hiding this comment.
In the current implementation, that all storage Python lib dependencies will be introduced into PyGVFS. Although there will be no conflicts in most cases, it may be better for users to only load the underlying storage dependencies they need. I wonder if there is a better way to introduce other FileSystem dependencies on demand, could you take a time to research this?
There was a problem hiding this comment.
Okay, I will do it today.
There was a problem hiding this comment.
@xloya
I have updated the code and used importlib to dynamically import file system classes according to needs.
What changes were proposed in this pull request?
Why are the changes needed?
It's user needs.
Fix: #5139
Does this PR introduce any user-facing change?
Modify the Python GVFS client.
How was this patch tested?
Test locally and add an IT that can't run automatically.
Modify mode as the following picture and execute
./gradlew :clients:client-python:test -PskipDockerTests=falsesuccess.