Discussion on port to polars

[paulf81]

Hi @Bartdoekemeijer , @misi9170 and @ejsimley , I started a new request on porting to polars (#97 ) and I wanted to have a place to discuss a little more that pull request. Specifically, what should be ported to polars? I had a couple of thoughts:

• Option 1, maybe only the energy ratio code and analysis code? The filtering code is already mature, and since it is more "run once" perhaps the pain of porting it all over is not ultimately worth it?
• Option 1B, I've also ported the SQL-interaction code, I found that this code works better/faster integrating with the postgres database and our data explorer GUI
• Option 2: Try to move all pandas code over for consistency. In this pull request I tried not to break anything the filters dataframe_filtering, dataframe_manipuation and ws_pow_filtering, instead offering a polars version of the file (ws_pow_filtering_pl), or else a polars version of just a few functions. What do you all think? We have a code spring coming

[Bartdoekemeijer]

Thanks for starting this discussion! In principle, I am not a big fan of having two versions of the same functions: one for pandas and one for polars. This duplicates a lot of code and also means double the work when modifying existing code, since the changes need to be mirrored. Also, I'd love to have Polars for the raw data filtering. I agree that it's only one step, but it's a step that can easily take 10+ hours. It would be really valuable to speed that up by a factor 5-10 with Polars, if we can.

One way to go about this without requiring a ton of work initially, is:

1. Update all the front-end (examples and analysis scripts) to use Polars
2. At the start of every FLASC function, convert the inserted dataframe into Pandas, and at the end of the function, convert it back to Polars.
3. As time goes on, replace functions one by one with Polars code, prioritizing functions that we use often and that are computationally expensive.

Though I haven't yet worked with Polars at all. How feasible do you think this would be?

[paulf81]

This is a great idea! But thinking it over, I think we make a smoother transition for most users if we flip it around, and keep the front end all pandas (since I think more people are used to it, and it is the better library at the moment for visualization). Then on the backend we do what you say and quietly move things to polars for any function we want to speed up or squash memory warnings. I believe the polars-> pandas -> polars functions aren't much bottlkeneck. Then perhaps one day the whole back-end is polars and then we pull the trigger on the front end. If you like this approach, then I think I could have something fairly seamless to merge pretty soon!

[Bartdoekemeijer]

Yes, even better, keeping Pandas on the front-end! If it's not too computationally intensive to convert Pandas to Polars, let's go ahead with this!

[paulf81]

Sounds good!

[paulf81]

Hi @Bartdoekemeijer and @misi9170 what do you think of something. In this polars port of energy ratio, I'm noticing I'm not now supporting overlapping wd bins. But I also notice I don't use these anyway since I think the smoothing effect finally can be a little misleading. Ok to no longer support this?

[Bartdoekemeijer]

I still use this functionality, but could be implemented in a variety of ways later on. OK for me to ignore for now. For API continuality, maybe we can keep the input in the function but just raise an error when the users specifies a bin width and bin step size that are not equal?

[paulf81]

Ok Bart that sounds good, I did a little thinking about this this morning and believe it could be added later on. I'm happy also to take a shot at it, I think the main complication is that if the bins don't overlap, then basically every data point is assigned to a group, and grouping functions work in a straightforward way. If they overlap, points get assigned to multiple groups, but I think this is workable with some thought. Here's a solution from pandas where you replace groupby with rolling to get the effect you want:

https://stackoverflow.com/questions/69194187/pandas-group-by-with-overlapping-bins

Which I think can also be made to work for us, but maybe its ok if we tackle this a little later?

[paulf81]

Accomplished in pull request #107