Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow integer type for L0A parquet files #114

Open
ghiggi opened this issue Nov 11, 2022 · 0 comments
Open

Allow integer type for L0A parquet files #114

ghiggi opened this issue Nov 11, 2022 · 0 comments
Assignees
Labels
enhancement New feature or request performance

Comments

@ghiggi
Copy link
Collaborator

ghiggi commented Nov 11, 2022

Description

In this commit I had to remove all integer types from the L0A_encodings.yml files because if there is a nan value in an integer column, an error will be raised during column casting.

This change increases the size of the L0A Parquet files.

To reintroduce the integer type, it would be required:

  • to define a dictionary with the nan flag value for each variable which is expected to be cast to an integer format.
  • during L0A processing replace the nan values using the nan_flag dictionary
  • during L0B processing, replace the nan_flag with np.nan (by casting the variable to float) or ensure that the nan_flag value corresponds to the _FillValue of the L0B encoding.
@ghiggi ghiggi self-assigned this Nov 11, 2022
@ghiggi ghiggi changed the title Reintroduce integer type for L0A parquet files [ENHANCEMENT] Allow integer type for L0A parquet files Feb 13, 2023
@ghiggi ghiggi added enhancement New feature or request performance labels Nov 5, 2023
@ghiggi ghiggi changed the title [ENHANCEMENT] Allow integer type for L0A parquet files Allow integer type for L0A parquet files Nov 5, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request performance
Projects
None yet
Development

No branches or pull requests

1 participant