ENH : improvement csv processing

[kalounis]

Pull request type

• Code changes (bugfix, features)

Checklist

• Tests for the changes have been added (if needed)
• Docs have been reviewed and added / updated
• Lint (black rocketpy/ tests/) has passed locally
• All tests (pytest --runslow) have passed locally
• CHANGELOG.md has been updated (if relevant)

Current behavior

The current code isn't able to deal with CSV files that contain a header.

New behavior

Now, when we create an instance of the Function Class, we can give for argument a CSV file with a header : the code will process this CSV file (drop NaN values) and will be able to deal with the header.

Breaking change

• No

Additional information

Enter text here...

[Gui-FernandesBR]

Good work! I liked the way you separated the file handler function into different functions within the tools.py module

This is my first review, I will later check documentation and tests, but you can already work on the suggestions.

Btw I think this function can be used in other parts of the code as well.

[Gui-FernandesBR]

did you commit this file by mistake?

[kalounis]

Yes I forgot to delete it, sorry!

[phmbressan]

Could you elaborate a bit more on why is this implementation an advantage over the current one, forgive me if I am mistaken, but my interpretation is that:

• The current way the CSVs are read follows the numpy.loadtxt in a try-except clause: numpy will try to convert the CSV to an array of floats, if there is an error, we skip the first header line. If there is still an error, the input is likely bad formatted.
• This new implementation creates aditional functions to process the data: the headers will be checked by trying a float() cast.

This seems to the same thing to me, but in the first case we can delegate to numpy the trouble of reading and not maintain these new functions.

We had recently a PR (#485 and #495) to handle CSV headers and there is even a test for it: test_func_from_csv_with_header.

[kalounis]

Hello @phmbressan, this code is not only able to skip the first row but also to drop NaN values which it is not the case in the current code. Moreover, the current code is not very efficient dealing with these cases (from my perspective). Try with the following CSV file that @Gui-FernandesBR gave me. The current code throws an error whereas mine deals perfectly with it. Let me know if you want a complete report on this code!
test_dataset_with_header.csv

[phmbressan]

Thanks to your clarifications @kalounis , now I understand the reasoning behind this PR. In fact, processing NaN values can be quite useful.

As I commented, we just need to make sure to inform if there were any problems on reading the source. After that, we are likely good to go.

[phmbressan]

Personally, I don't think writing a new file with the processed data is really needed here.

More important, I think, is that we must make it clear to the user that not everything of their file was used (because there were NaN). So, could you trigger a warning that informs if any lines were skipped on source processing.

The implementation of this warning is up to you, but I believe that a simple boolean that is set to True in the for loop above if there were any skipped lines is enough. Then, if this boolean is True raise the warning.

There are other places in rocketpy that we raise warnings if you want to base the implementation on that. Of course, should you have any doubts don't hesitate in commenting.

[Gui-FernandesBR]

tests not passing kinda worries me a bit but I couldn't find enough time to investigate it.

[Gui-FernandesBR]

If the problem we are trying to solve is to read .csv files that have NaN values, I think we should consider Changning the np.loadtxt to the np.genfromtxt (see https://numpy.org/doc/stable/reference/generated/numpy.genfromtxt.html).

This function is a bit more complex and can

Relying in numpy may be more beneficial than using the full pythonic approach, both in terms of maintenance and speed. But this is something that can be discussed.

https://stackoverflow.com/questions/20245593/difference-between-numpy-genfromtxt-and-numpy-loadtxt-and-unpack

[Gui-FernandesBR]

Closed as not finished.
The branch is still available in the repo tho.