feat: Initial support for Pandas timeseries backend. #240
Conversation
Basic support for loading dataframes using Pandas. This is implemented using a Python callback function that loads the dataframe using Pandas, and then converts it to Polars.
There was a problem hiding this comment.
Looks good though to get it to work with my venv I had to add the path to the site-packages folder as suggested here - PyO3/pyo3#3726 It looks like pyo3 doesn't yet have great support for virtual environments.
pywr-schema/src/timeseries/pandas.rs
Outdated
|
|
||
| let mut df = df.0; | ||
|
|
||
| println!("Loaded Pandas dataset: {}", df); |
|
This worked fine for me when running the tests with my virtual environment activated. However, I appreciate that may not be working for everyone. The CI has the same issue with the Rust workflows not having the correct Python libraries available. If this is to go ahead we might need to merge the workflows or only run some of the tests in the Python workflows. One general option is to add a "python" feature to the schema crate that would include things like this Pandas support. |
|
What do you think is the best way forward with this? |
This is used to gate tests which need Python env available while running the Rust tests. Also some tweaks to the Pandas implementation.
|
I've featured gated the Rust tests that need a Python environment to work. They are not currently run in the CI. |
Basic support for loading dataframes using Pandas. This is implemented using a Python callback function that loads the dataframe using Pandas, and then converts it to Polars.