Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Register object stores via SessionStateBuilder #12553

Closed
Tracked by #12550
alamb opened this issue Sep 20, 2024 · 2 comments · Fixed by #12578
Closed
Tracked by #12550

Support Register object stores via SessionStateBuilder #12553

alamb opened this issue Sep 20, 2024 · 2 comments · Fixed by #12578
Assignees
Labels
enhancement New feature or request good first issue Good for newcomers

Comments

@alamb
Copy link
Contributor

alamb commented Sep 20, 2024

Is your feature request related to a problem or challenge?

Part of #12550

While working on https://github.com/datafusion-contrib/datafusion-dft I want to register various types of extensions while configuring the SessionContext, ideally adding each extension to the SessionStateBuilder each time.

However, I found that there were a few APIs missing on SessionStateBuilder so I had to implement my own workaround builder here: https://github.com/datafusion-contrib/datafusion-dft/blob/8247555f9464058c1ac3370196739ac2b19343ee/src/extensions/builder.rs#L102-L1078

And then call it
https://github.com/datafusion-contrib/datafusion-dft/blob/8247555f9464058c1ac3370196739ac2b19343ee/src/extensions/s3.rs#L59-L62

SessionStateBuilder has no way to register an object store.

Describe the solution you'd like

I would like a way to register object stores

Also it should have

  1. Documentation
  2. Tests (ideally a doc test with an example of how to use to use it)

Describe alternatives you've considered

I recommend adding two new functions

  1. SessionStateBuilder::with_object_store that calls through to RuntimeEnv::register_object_store
  2. SessionStateBuilder::runtime_env() that returns the current RuntimeEnv (follow model here) which would permit access to the underlying RuntimeEnv for access to other more advanced features

So this would be used like

let state = SessionStateBuilder::new() 
  .with_object_store(url, object_store)
  .build()

Or

let mut builder = SessionStateBuilder::new();
bulder.runtime_env().register_object_store(url, object_store)
let state = builder.build()

Additional context

No response

@alamb alamb added the enhancement New feature or request label Sep 20, 2024
@alamb alamb changed the title Register object stores via SessionStateBuilder Support Register object stores via SessionStateBuilder Sep 20, 2024
@alamb
Copy link
Contributor Author

alamb commented Sep 20, 2024

I think this is a good first issue as it is clearly described and straightforward to implement. It would be a good introduction to DataFusion I think

@alamb alamb added the good first issue Good for newcomers label Sep 20, 2024
@OussamaSaoudi
Copy link
Contributor

take

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants