introduction of comprehensive test suite

[fleimgruber]

refs #87.

Right now, 4/12 tests are failing:

FAILED tests/test_clients.py::EntsoeRawClientTest::test_basic_queries - Asser...
FAILED tests/test_clients.py::EntsoeRawClientTest::test_query_withdrawn_unavailability_of_generation_units
FAILED tests/test_clients.py::EntsoePandasClientTest::test_query_withdrawn_unavailability_of_generation_units
FAILED tests/test_domains.py::test_czech_unavailability_of_production_units

I have not looked into those yet. Would appreciate a quick look by someone who can judge whether these tests are based on wrong assumptions regarding API results, are testing too deep into the implementation or are simply not relevant any more.

Additionally, I think a natural next step would be to have a Github Action for unit tests. I wanted to get familiar with Github Actions in any case so would be up for working on it as well. Please let me know if you want to have the Action also developed in this PR, a new PR or if you prefer to implement Actions yourself.

[fboerman]

hi @fleimgruber thanks for the work! I am going through the failings (using your branch):

test_basic_queries for raw client: the one failing is the last one which is correct that it returns a zipfile. That test function only checks if response is string
test_query_withdrawn_unavailability_of_generation_units for raw client: this test is kinda funny, when written by the original author (energyID is from belgium hence the country code) this was apparently not available for belgium. but now it is so the client gives a normal response which is not expected by the test.
test_query_withdrawn_unavailability_of_generation_units for pandasclient: same as above
the last one I am unfamiliar with so I am not sure ...

tbh I think it is good if we remove a lot of weird tests and strive to put all queries both pandas and raw in it and check if there is an appropriate response. checking actual data is a bit difficult a lot of work and not worth it I think. I will leave it up to you how you want to design that but that is my recommendation.

I want to focus on some bug fixing so if you can set this up further it would be great.

I have some experience with github actions so can set that up myself no problem. Thanks for your work so far!

[fleimgruber]

I rebased and pushed the latest changes. The following notes reference the state before the rebase. Note that after the rebase we get new NoMatchingDataErrors, probably related to 34daa30, but maybe the README is just not updated yet...

I tried to follow the README API examples as closely as possible. For some, I had to explicitly use keyword arguments, see also notes below.

Would appreciate a review as this is not a small change. I also tried to give examples of how things are extensible using the pytest fixture mechanisms here and there. If you have any questions on the pytest concepts and usage (e.g. only run subsets of tests), let me know.

• EntsoeRawClient.query_procured_balancing_capacity:
  returns a byte string, not a str
• EntsoeRawClient.query_generation_per_plant:
  throws a Pagination exception
• EntsoePandasClient.query_contracted_reserve_amount:
  call signature from README errors with: TypeError: year_wrapper() missing 2 required keyword-only arguments: 'start' and 'end'
  change to call signature (implemented in tests):

  client.query_contracted_reserve_amount(
        country_code, start=start, end=end, type_marketagreement_type=type_marketagreement_type
  )
  

  errors with: ValueError: No objects to concatenate
• EntsoePandasClient.query_contracted_reserve_prices:
  same problem as with EntsoePandasClient.query_contracted_reserve_amount
• EntsoePandasClient.query_unavailability_transmission:
  Error zipfile.BadZipFile: File is not a zip file
• EntsoePandasClient.query_unavailability_transmission:
  AssertionError: Cannot concat indices that do not have the same number of levels
• EntsoePandasClient.query_procured_balancing_capacity:
  call signature from README errors with: TypeError: year_wrapper() missing 2 required keyword-only arguments: 'start' and 'end'
  change to call signature (implemented in tests):

  process_type = "A47"
  result = client.query_procured_balancing_capacity(
      country_code, start=start, end=end, process_type=process_type
  )
  

  errors with: ValueError: No objects to concatenate

[fboerman]

hi @fleimgruber first of all apologies for not seeing the work you did! I was kind of misguided by the more simpler title of your PR and also missed the notification. I noticed now because I get new notifications for other PR's.

big thank you for trying your hand at this. Having a test suite would be great. I also got somebody else trying its hand at it this week, but yours is more comprehensive right now. I will try to run your fork locally to look at the tests and what goes wrong. I am not the original author of the library so some failures I may not know either but we shall see.

I hope to have some time in the coming days for this.

ps: i will update your title to better cover what the PR does

[fboerman]

hi @fleimgruber im going through the code, your setup looks very nice! I dont have a lot of experience with pytest but it works quite nice. Could you perhaps give me write access to your branch? I think that I can fix most things that are currently erroring. I also would like to check with the latest version of the library from the master branch just to be sure.

an important thing I saw is that is very crucial which days we query in the test. some endpoints are not available for those days, but its a but clunky to change the day all over the place. I have for now disabled that test in my local copy as I dont see that as a true failure

[fleimgruber]

Could you perhaps give me write access to your branch?

done.

but its a but clunky to change the day all over the place

Where does it need to be changed?

I have for now disabled that test in my local copy as I dont see that as a true failure

We could write tests that test the failure in these cases?

[fleimgruber]

FYI, I also just rebased refactor_tests on latest upstream master.

[fleimgruber]

FYI, just rebased again on master. Additional commits for

• API key usage via python-dotenv
• Method renaming for net position
• Test coverage due to README changes on master

This is the pytest summary:

================================================================================= short test summary info =================================================================================
FAILED tests/test_pandas.py::test_basic_queries_series[BE-start2-end2-query_aggregate_water_reservoirs_and_hydro_storage] - entsoe.exceptions.NoMatchingDataError
...
FAILED tests/test_pandas.py::test_query_offered_capacity - Exception: Please specify the start and end date explicity with start=<date> when calling this function
FAILED tests/test_pandas.py::test_basic_queries_dataframe[BE-start7-end7-query_installed_generation_capacity_per_unit] - requests.exceptions.HTTPError: 500 Server Error: Internal Server...
FAILED tests/test_raw.py::test_basic_queries[BE-start7-end7-query_generation_per_plant] - entsoe.exceptions.PaginationError: The API is limited to 1 elements per request. This query req...
FAILED tests/test_raw.py::test_basic_queries[BE-start9-end9-query_installed_generation_capacity_per_unit] - requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for ur...
...
================================================================= 31 failed, 27 passed, 101 warnings in 223.59s (0:03:43) =================================================================

where I omitted all other cases of entsoe.exceptions.NoMatchingDataError with ..., leaving these errors to address:

• requests.exceptions.HTTPError: 500 Server Error: Internal Server Error
• entsoe.exceptions.PaginationError: The API is limited to 1 elements per request.
• Exception: Please specify the start and end date explicity with start= when calling this function

[fboerman]

hi @fleimgruber wow thanks a lot thats quite some work! I am sorry this got left on my todo list. so if I understand correctly it mostly works? Could you please share the exact calls you do for those three remaining errors?

I will try to look into the pagination and general exceptions. for the http 500 ill send an email to the transparancy team itself, thats a server issue. Again thanks for your efforts and sorry for my delay!

[fleimgruber]

No worries, with holidays and all...

One nice thing about pytest is that you can use the outputs of the test report to select and re-run the interesting failing cases (including their respective parametrization) individually; in this case e.g. like this:

pytest -k test_basic_queries_dataframe[BE-start7-end7-query_installed_generation_capacity_per_unit]
pytest -k test_basic_queries[BE-start7-end7-query_generation_per_plant]
pytest -k test_basic_queries[BE-start9-end9-query_installed_generation_capacity_per_unit]
pytest -k test_query_offered_capacity

from the repo root. You can then also debug those tests to see the actual API call that they use.

[pee-po]

@fboerman In light of my bug factory (#287 which spawned #292 #296 and #301) I think that maybe it's a good idea to get back to the idea of serious automated tests. Cached requests (as discussed in #83 ) and well maintained tests will not only allow for quicker iterations and fewer bugs but will also take some weight off your shoulders.

I'd love to contribute but there are already 2 PRs and this is a big undertaking. Maybe you could chart the course?

[fleimgruber]

@fboerman I rebased and set up Github Actions for pytest to better talk about the remaining issues. I can split the CI related commits into a separate PR if you want to once the tests (this PR) are all green. Results on my repo are here: https://github.com/fleimgruber/entsoe-py/actions/runs/9085525870.

To be clear, the test suite is composed of all example usages from the README. Remaining errors are:

• entsoe.exceptions.NoMatchingDataError
• entsoe.exceptions.PaginationError

For this to work on https://github.com/EnergieID/entsoe-py we need to set a secret for the API Key referenced here: https://github.com/fleimgruber/entsoe-py/blob/eb1e321c6bc5e3eefc1c605d3a9f308b6d365d4a/.github/workflows/ci.yaml#L10.

[fboerman]

hi @fleimgruber sorry for the delay, I am rather busy need to find some time to go through this more thoroughly

[fleimgruber]

@fboerman What about a pair-review via video call, as an efficient quickstart for looking into it yourself more thoroughly afterwards?

[fboerman]

@fleimgruber I did not find enough time to go through thoroughly this myself so perhaps that is a good idea indeed. please shoot me an email at [email protected] to discuss how to set it up!

[fleimgruber]

@fboerman rebased on latest master.