[FEATURE] Spark streaming job rate limit

[dai-chen]

Is your feature request related to a problem?

Spark structured streaming API supports rate limit by maxFilesPerTrigger. Currently, Flint user can configure this when create index like example here: . However, there is no guidance on this value or similar support like maxBytesPerTrigger in Delta or Iceberg table.

Without such limit, Flint index like covering index may cause heavy ingestion pressure on OpenSearch cluster until back pressure mechanism triggered (end up with connect exception, socket timeout exception, circuit break or cluster write block exception).

What solution would you like?

Options:

1. Guide user recommended maxFilesPerTrigger value
2. Implement similar maxBytesPerTrigger in File source (maybe helpful to provide ALTER index statement so user can change this)

What alternatives have you considered?

Implement adaptive rate limiter on OpenSearch write latency

[penghuo]

apache/spark#44636

[dai-chen]

Confirmed that the PR above was merged to Spark master branch (Spark 4.0.0).