[FEATURE] Performance and Scalability Enhancements for Flint Index

[dai-chen]

Is your feature request related to a problem?

While the current Flint index solution, as of version 0.4 (Issue #3), significantly enhances query acceleration, it still falls short in terms of scalability and performance when handling large-scale datasets. This limitation often leads to user frustration due to inefficient processing and delayed responses.

This meta issue aims to address these user pain points by consolidating all related enhancements and improvements for better tracking and resolution. Here are linked issues for use case support:

• [EPIC] Zero-ETL - AWS ALB Logs Integration #186
• [EPIC] Zero-ETL - AWS WAF logs Integration #198

What solution would you like?

To effectively address the performance and scalability issues identified, a comprehensive and systematic approach is proposed. This strategy is organized into tasks, each targeting specific components of the Flint index system, and categorized as follows:

• Direct Query

  • Approximate Query Processing

• Source

  • [FEATURE] Push mode streaming support #29
  • [FEATURE] Spark streaming job rate limit #187
  • [FEATURE] Non hive partition discovery and update. #212
  • [FEATURE] Avoid expensive S3 listing by using file list in skipping index #218
  • [FEATURE] Avoid S3 throttle when fetching object metadata #607

• Sink

  • [FEATURE] Avoid throttling when writing data to index #640

• Skipping Index

  • [FEATURE] Support partial indexing for skipping and covering index #89
  • [FEATURE] Skipping index and materialized view refresh synchronization #93
  • [RFC] Proposal: introduce new Flint Table concept #217
  • [RFC] Flint serverless skipping index #118
  • [FEATURE] Enrich skipping index data structure #193

• Covering Index

  • [FEATURE] Support COPY operation #129
  • [RFC] Accelerate Spark SQL queries by covering index #298
  • [FEATURE] Enhance Flint data source reader performance in OpenSearch integration #334
  • [BUG] Flint index cannot handle large dataset due to OpenSearch doc limit #339

• Materialized View

  • [FEATURE] Large backlog processing when materialized view cold start #90

• Advanced OpenSearch Optimization

  • [EPIC] Zero-ETL - OpenSearch Table #185
  • [FEATURE] Create skipping index on IP type field #203
  • [FEATURE] Improve filtering condition push down in Flint data source #148

• Performance Test

  • Derived dataset auto refresh benchmark #38
  • Performance test on large dataset sql#1399
  • Estimate the cost reduction by Maximus solution sql#1401
  • Figure out recommendation on Spark cluster config and sizing sql#1402

What alternatives have you considered?

N/A

Do you have any additional context?

N/A