You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem?
Currently, both interactive and streaming queries share configurations for Spark executors. When streaming jobs are running, executors are dynamically allocated with an upper limit set by spark.dynamicAllocation.maxExecutors (the default value is 30). Spark's dynamic resource allocation (DRA) system aggressively allocates more executors than necessary to achieve low latency, leading to inefficiencies, because the volume of data processed in each interval is relatively small (100GB per 15 minutes), and the latency requirement is not stringent (15 minutes).
What solution would you like?
add spark.dynamicAllocation.streaming.maxExecutors configuration and default value is 10
What alternatives have you considered?
in openserach-sql, submit interactive and streaming query separately and configure spark.dynamicAllocation.maxExecutors.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem?
Currently, both interactive and streaming queries share configurations for Spark executors. When streaming jobs are running, executors are dynamically allocated with an upper limit set by spark.dynamicAllocation.maxExecutors (the default value is 30). Spark's dynamic resource allocation (DRA) system aggressively allocates more executors than necessary to achieve low latency, leading to inefficiencies, because the volume of data processed in each interval is relatively small (100GB per 15 minutes), and the latency requirement is not stringent (15 minutes).
What solution would you like?
add
spark.dynamicAllocation.streaming.maxExecutors
configuration and default value is 10What alternatives have you considered?
in openserach-sql, submit interactive and streaming query separately and configure spark.dynamicAllocation.maxExecutors.
The text was updated successfully, but these errors were encountered: