SNOW-1258064 Remove/Relax the limitations in order to generate bigger files

[sfc-gh-alhuang]

Relax file size limitation to deal with longer client flush lag which might produce larger file.

JIRA

[sfc-gh-tjones]

Should this still be here but with higher limits?

[sfc-gh-gdoci]

+1

[sfc-gh-tzhang]

1. This change was added before to avoid OOM issue in the scanner, given that the Parquert scanner is much better now and the iceberg doc mentions that it works well for files upto 1GB compressed, I don't think this is necessary now
2. We still have a warning on this, see checkBatchSizeRecommendedMaximum right below this function
3. We have valid use cases where the batch size could be big, imagine the case where the Kafka Connector has a big flush interval and everything will be inserted as one batch

[sfc-gh-lsembera]

Sorry, I somehow missed this PR until it got merged. I have concerns about this change because there was another reason why these limits were introduced - to prevent long GC pauses. We observed JVM freezes for customers who were passing hundreds of thousands of rows into insertRows(), see more details here.

I would prefer to keep some limit there, customers should not pass hundreds of megabytes of data in a single batch.

cc @sfc-gh-wfateem

[sfc-gh-tzhang]

@sfc-gh-alhuang Do you want to run some local tests to see if things are working with a few GBs of a batch and 1 minutes of flush interval? If not, then we can add the exception back but with a much higher limit, WDYT?

[sfc-gh-alhuang]

Got GCLocker timeout error message after building a 600+MB of blobs. I'll add the assertion back with higher limit.

[sfc-gh-tzhang]

1. This change was added before to avoid OOM issue in the scanner, given that the Parquert scanner is much better now and the iceberg doc mentions that it works well for files upto 1GB compressed, I don't think this is necessary now
2. We still have a warning on this, see checkBatchSizeRecommendedMaximum right below this function
3. We have valid use cases where the batch size could be big, imagine the case where the Kafka Connector has a big flush interval and everything will be inserted as one batch

[sfc-gh-tzhang]

Thanks!

[jdcaperon]

Keen to see this released, any idea on how long til this is cut?