Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabled pyarrow AzureFileSystem for Windows #44655

Open
davlee1972 opened this issue Nov 5, 2024 · 0 comments
Open

Enabled pyarrow AzureFileSystem for Windows #44655

davlee1972 opened this issue Nov 5, 2024 · 0 comments

Comments

@davlee1972
Copy link

davlee1972 commented Nov 5, 2024

Describe the enhancement requested

There are several resolved issues around enabling and packaging pyarrow.fs.AzureFileSystem for linux, but with the latest pyarrow 18.0 release there still isn't any packaging for windows..

#44347

No AzureFileSystem in windows.

ImportError: The pyarrow installation is not built with support for 'AzureFileSystem'

C:\>python
Python 3.9.18 (main, Sep 11 2023, 14:09:26) [MSC v.1916 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> import pyarrow.fs as fs
>>> pa.__version__
'18.0.0'
>>> dir(fs)
['AwsDefaultS3RetryStrategy', 'AwsStandardS3RetryStrategy', 'FSSpecHandler', 'FileInfo', 'FileSelector', 'FileStats', 'FileSystem', 'FileSystemHandler', 'FileType', 'GcsFileSystem', 'HadoopFileSystem', 'LocalFileSystem', 'PyFileSystem', 'S3FileSystem', 'S3LogLevel', 'S3RetryStrategy', 'SubTreeFileSystem', '_MockFileSystem', '__builtins__', '__cached__', '__doc__', '__file__', '__getattr__', '__loader__', '__name__', '__package__', '__spec__', '_copy_files', '_copy_files_selector', '_ensure_filesystem', '_filesystem_from_str', '_is_path_like', '_not_imported', '_resolve_filesystem_and_path', '_stringify_path', 'atexit', 'copy_files', 'ensure_s3_finalized', 'ensure_s3_initialized', 'finalize_s3', 'initialize_s3', 'resolve_s3_region']
>>> hasattr(fs, "AzureFileSystem")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\xxxxx\Anaconda3\lib\site-packages\pyarrow\fs.py", line 75, in __getattr__
    raise ImportError(
ImportError: The pyarrow installation is not built with support for 'AzureFileSystem'

This works fine in linux:

(base) bash-4.2$ python
Python 3.9.19 (main, May  6 2024, 19:43:03)
[GCC 11.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pyarrow as pa
>>> import pyarrow.fs as fs
>>> pa.__version__
'18.0.0'
>>> dir(fs)
['AwsDefaultS3RetryStrategy', 'AwsStandardS3RetryStrategy', 'AzureFileSystem', 'FSSpecHandler', 'FileInfo', 'FileSelector', 'FileStats', 'FileSystem', 'FileSystemHandler', 'FileType', 'GcsFileSystem', 'HadoopFileSystem', 'LocalFileSystem', 'PyFileSystem', 'S3FileSystem', 'S3LogLevel', 'S3RetryStrategy', 'SubTreeFileSystem', '_MockFileSystem', '__builtins__', '__cached__', '__doc__', '__file__', '__getattr__', '__loader__', '__name__', '__package__', '__spec__', '_copy_files', '_copy_files_selector', '_ensure_filesystem', '_filesystem_from_str', '_is_path_like', '_not_imported', '_resolve_filesystem_and_path', '_stringify_path', 'atexit', 'copy_files', 'ensure_s3_finalized', 'ensure_s3_initialized', 'finalize_s3', 'initialize_s3', 'resolve_s3_region']
>>> hasattr(fs, "AzureFileSystem")
True

Component(s)

Packaging, Python

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant