Replace use of pandas in pyrealm.demography

[davidorme]

Description

In #277, I used pandas.Dataframe within Flora to provide an array-like view onto the plant functional type data. That then led to using it for Community.cohort_data (#282) and also typing the inputs to the T Model functions as pandas.Series. That turns out to be awkward for a number of core use cases:

• See comment Add helper functions for PFT geometry and canopy shape. #290 (comment) about the awkwardness of using this structure in creating allometry predictions for a Flora
• See comment Implementation of the canopy model #288 (comment) about the repeated casting of Series to np.ndarray when broadcasting needed.
• See issue Look at replacing pandas in pyrealm.demography #291 which summarises the issues and describes what this PR should do.

This PR starts to replace the usage with a pure numpy alternative. There is code in the currently open PR #288 that will also need updating, but this seems like the right way to go now.

The code:

• Replaces the use of pandas structures throughout.
• Adds a prototype validation function and unit test for that function to validate array inputs to T model functions.
• I have not yet added that validation to the functions - wanted to wait for feedback before moving on.

Fixes #291 (issue)

Type of change

• New feature (non-breaking change which adds functionality)
• Optimization (back-end change that speeds up the code)
• Bug fix (non-breaking change which fixes an issue)

Key checklist

• Make sure you've run the pre-commit checks: $ pre-commit run -a
• All tests pass: $ poetry run pytest

Further checks

• Code is commented, particularly in hard-to-understand areas
• Tests added that prove fix is effective or that feature works

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 95.22%. Comparing base (1f315ba) to head (97ad0ed).
Report is 88 commits behind head on develop.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop     #292      +/-   ##
===========================================
- Coverage    95.29%   95.22%   -0.07%     
===========================================
  Files           28       32       +4     
  Lines         1720     2115     +395     
===========================================
+ Hits          1639     2014     +375     
- Misses          81      101      +20     

Flag	Coverage Δ
	95.22% <100.00%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[j-emberton]

The implementation looks fine to me overall.

One issue to fix:
There's a straggler issue in that the community.py module docstring still makes reference to using pandas as the internal data vehicle.

Internally, the cohort data in the Community class is represented as a pandas dataframe, which makes it possible to update cohort attributes in parallel across all cohorts but also provide a clean interface for adding and removing cohorts to a Community.

And one query:
I see the type hints for the new NDArrays are now float32. Is there a specific reason for this? Will Python type promotion not mean that this doesn't survive contact with any float64 data?

[davidorme]

One issue to fix: There's a straggler issue in that the community.py module docstring still makes reference to using pandas as the internal data vehicle.

Fixed

And one query: I see the type hints for the new NDArrays are now float32. Is there a specific reason for this? Will Python type promotion not mean that this doesn't survive contact with any float64 data?

That's a good point - the typing of arrays has got cleaner and I think the earlier code was written when this was not such a transparent thing to do. I don't think we need the precision of float64 so we use less RAM by doing this, but equally I don't know if that has speed implications (good or bad) with 64 bit architecture.

So - not sure. I guess I'd like to leave this PR as is and tackle the array typing more widely as another issue.

[j-emberton]

Thanks for sorting the docstring. Happy for the float32 types hints to be left in place for the time being. My gut feel is that any data read in from csv/toml etc will default to float64, and would need to be explicitly converted to float32.