Skip to content
This repository has been archived by the owner on Sep 28, 2022. It is now read-only.

GenericParser should be resilient to bad fields #53

Open
jaffee opened this issue Apr 4, 2018 · 0 comments
Open

GenericParser should be resilient to bad fields #53

jaffee opened this issue Apr 4, 2018 · 0 comments

Comments

@jaffee
Copy link
Member

jaffee commented Apr 4, 2018

The generic parser currently returns an error any time it fails to parse a field's value, and stops processing for that record. In reality, there are lots of innocuous reasons why a certain field might fail to parse, and it doesn't indicate that the entire record is suspect.

We should have stat/log options for notifying and counting which fields are failing, why, and how often, but we should make a best effort to parse any record and return some data.

I don't think we need to go crazy with configurability, some count stats that use the field path and encapsulate what the error is (e.g. null value, unsupported type, etc.). pdk/ingest.go has an example of a simple stats interface, that we should probably extend throughout the codebase so that it can be configured at the top level.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant