Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parsing leniency differs between EnceladusDateTimeParser and Spark #707

Closed
benedeki opened this issue Jul 27, 2019 · 0 comments
Closed

Parsing leniency differs between EnceladusDateTimeParser and Spark #707

benedeki opened this issue Jul 27, 2019 · 0 comments
Assignees
@benedeki
Labels
bug Something isn't working priority: medium Important but not urgent Standardization Standardization Job affected

Comments

@benedeki
Copy link
Collaborator

Describe the bug

During #466/ #679 implementation I noticed some skipped test were actually failing. Upon investigation it turns out Spark and EnceladusDateTimeParser have different setup for the pattern parsing leniency.
That causes different behavior of both, end can lead to unexpected Standardization failures

To Reproduce

Struct field with pattern = "dd-MM-YYYY" and default = "2015-01-01" passes validation but fails in Spark/Standardization

Expected behaviour

Spark and EnceladusDateTimeParser behave similarly in string parsing, validation is reliable.
@benedeki benedeki added bug Something isn't working Standardization Standardization Job affected priority: undecided Undecided priority to be assigned after discussion labels Jul 27, 2019
@benedeki benedeki self-assigned this Jul 27, 2019
@benedeki benedeki mentioned this issue Jul 27, 2019
Merged
benedeki added a commit that referenced this issue Jul 27, 2019
@benedeki
#707: Parsing leniency differs between EnceladusDateTimeParser and Spark
79097e0 
* Match EnceladusDateTimeParser lenient setting to Spark's
* Extra test on leniency
* Fixed SchemaValidationSuite tests
* Ensuring Maven runs tests in UTF-8
@GeorgiChochov GeorgiChochov added priority: medium Important but not urgent and removed priority: undecided Undecided priority to be assigned after discussion labels Jul 29, 2019
benedeki added a commit that referenced this issue Jul 29, 2019
@benedeki
Merge pull request #708 from AbsaOSS/bugfix/707-parsing-leniency-diff…
76085fd 
…ers-between-enceladusdatetimeparser-and-spark

#707: Parsing leniency differs between EnceladusDateTimeParser and Spark
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@benedeki benedeki

Labels
bug Something isn't working priority: medium Important but not urgent Standardization Standardization Job affected
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@benedeki @GeorgiChochov