[Do not merg] TMP

[sfc-gh-alhuang]

No description provided.

[sfc-gh-alhuang]

We need this for extra field validation.

[sfc-gh-alhuang]

Encode all "_" to its hex, same idea as escape.

[sfc-gh-hmadan]

Doing this stringbuilder business on the hotpath will make performance even worse.

Have two suggestions:

Option 1:

• When parsing the schema, in IcebergDataTypeParser, set the struct's field's name to the field id.
• When validating the input row, in IcebergParquetValueParser, (over here next to this comment) : (i) switch the logic to loop over the structVal.Keys collection instead of type.Fields, (ii) for each key, do a lookup to get the field id (you'll have to pipe through a lookup map and maintain it somewhere too), (iii) use this fieldId as the "field name" when doing type.getType. Note that type.getType has an overload that takes in field names.

Option 2:

• When parsing the schema, in IcebergDataTypeParser, set the struct's field name to getEscapedString(fieldName) and let AvroSchemaUtil do its thing on whatever we pass in.
• Whan validating the input row, in IcebergParquetValueParser, (i) switch logic to loop over structVal.Keys instead of type.Fields, (ii) for each key, call type.getFieldName(getEscapedString(key))
• getEscapedString(String str) should be implemented as: return str.replace("_", "_x" + Integer.toHexString("_").toUpperCase())
  This combined with AvroSchemaUtil's internal behavior to escape every non-digit, non-letter, non-underscore character, will make sure we are able to generate safe non-colliding names. tldr with this fix we're making sure everything except digits and numbers is getting escaped properly, and effectively fixing AvroSchemaUtil's bug
  In both cases, switching what we iterate on will need to be accompanied by fixes in how extraFields collection's bookkeeping is done.

I prefer option 1 because it completely sidesteps AvroSchemaUtil's buggy behavior. But i'm not sure if there is an easy way to maintain this per-struct-type map somewhere and handle all nested struct situations where there's a series of structs embedded inside each other. Still putting it out here in case you can come up with something.

[sfc-gh-alhuang]

Sounds good, will try option 1