Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve from_json_to_raw_map #2562

Draft
wants to merge 37 commits into
base: branch-24.12
Choose a base branch
from

Conversation

ttnghia
Copy link
Collaborator

@ttnghia ttnghia commented Nov 1, 2024

This adds a couple of missing functionalities to from_json_to_raw_map, which is the backend of from_json with map<string, string> schema:

  • Correctly nullify the output rows corresponding to the null/empty/invalid input JSON strings.
  • Support parameters normalize_single_quotes, allow_leading_zeros, allow_nonnumeric_numbers, and allow_unquoted_control.
  • Recovery from failure: invalid JSON strings will result in nulls for these specific rows while the rest can be parsed correctly.

Depends on:

Closes:

Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia added bug Something isn't working enhancement New feature or request labels Nov 1, 2024
@ttnghia ttnghia self-assigned this Nov 1, 2024
Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia requested a review from revans2 November 1, 2024 21:44
Signed-off-by: Nghia Truong <[email protected]>
revans2
revans2 previously approved these changes Nov 5, 2024
# Conflicts:
#	src/main/cpp/src/json_utils.hpp
@ttnghia ttnghia marked this pull request as ready for review November 5, 2024 22:00
@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 5, 2024

build

@ttnghia
Copy link
Collaborator Author

ttnghia commented Nov 5, 2024

Need to add more integration tests to spark-rapids to verify before merging this.

Signed-off-by: Nghia Truong <[email protected]>
@ttnghia ttnghia marked this pull request as draft November 6, 2024 22:49
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Signed-off-by: Nghia Truong <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
2 participants