[FEATURE] Redefine drop index semantic as logical deletion

[dai-chen]

Is your feature request related to a problem?

Currently the semantic of DROP statement for all Flint index is:

1. Terminate incremental refreshing job in current SparkContext
2. Delete physical Flint index data in OpenSearch

This is inconvenient for user who wants to do step #1 only (there is no STOP refresh job command in the meanwhile).

What solution would you like?

Redefine drop index statement as logical deletion with the following changes:

1. Only stop refreshing job without deleting Flint index data
2. [TBD] Provide new VACCUM index statement which deletes both index data and checkpoint data (if any)
3. [TBD] Provide new a way (REFRESH or a new command) to revive the logical deleted index back to ACTIVE state

What alternatives have you considered?

N/A

[dai-chen]

Planning the several PRs as follows:

1. Add vacuumIndex API and SQL support
2. Change deleteIndex API and DROP index statement to logical delete
3. Later add cancelIndex API and CANCEL INDEX JOB statement.
   a. deleteIndex API will do nothing than state transition
   b. DROP statement semantic stay the same but will call both cancelIndex and deleteIndex API instead.