[Refactor] Decouple Flint index monitor from Flint Spark API layer

[dai-chen]

Is your feature request related to a problem?

Currently, FlintSparkIndexMonitor is placed within the Flint Spark integration module and is started automatically by the Flint refresh and recover API. This causes the following problems:

1. Monitoring logic is coupled with the Flint Spark API.
2. The Flint Spark application code has to bypass Flint SQL and call the awaitMonitor API directly: https://github.com/opensearch-project/opensearch-spark/blob/main/spark-sql-application/src/main/scala/org/apache/spark/sql/JobOperator.scala#L96
3. Index monitor is unable to reuse error handling code in application: Store error message for streaming job execution in Flint metadata log #433 (comment)

What solution would you like?

I propose moving the index monitor to the Flint Spark application layer. The application code should control the start and stop of the index monitor. This would:

1. Decouple the monitoring logic from the Flint Spark API.
2. Allow for more flexible and explicit control of index monitoring within the application code.

Key considerations include evaluating if the benefits are worth the effort and any associated risks, as well as determining how the application code should decide to start monitoring after the Flint index creation statement completes.

What alternatives have you considered?

• Keeping the FlintSparkIndexMonitor in the integration module but refactoring the awaitMonitor call mechanism to reduce complexity.
• Creating a separate monitoring service that can be invoked independently by the Flint Spark application and Flint SQL.

Do you have any additional context?

N/A