Add more info in Flint metadata log

[dai-chen]

Description

The PR incorporates the follow-up minor changes below around recently added Flint metadata log and optimistic transaction in PR #110

New Changes

1. Added new jobStartTime: this won't be update until auto refresh restart by RECOVER command

Bug Fixes

2. Added missing latestId: this fixes the drop index failure
3. Changed lastUpdateTime to Long type: earlier it was double quoted when create doc by mistake

Improvements

4. Add transient log rollback capability in optimistic transaction
5. Singleton scheduler executor service, add it to SparkContext shutdown hook and use scheduleWithFixedDelay() API instead
6. Changed createIndex() to allow transition from DELETED state: in case that index is logical deleted and user wants to recreate after they delete index data manually

Cleanup

7. Remove old EMR info stored in Flint metadata: it's currently moved to metadata log and no PPL plugin is dependent on it any more

Testing

Creating index with latest Flint metadata format (cover 1, 2, 3)

spark.sql("""
CREATE SKIPPING INDEX ON stream.lineitem_tiny
(l_shipdate VALUE_SET)
WITH (
  auto_refresh = true,
  checkpoint_location = "s3://checkpoints/job-1"
)
""")

      {
        "_index": ".query_execution_request_myglue",
        "_id": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
        "_score": 1,
        "_source": {
          "version": "1.0",
          "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
          "type": "flintindexstate",
          "state": "refreshing",
          "applicationId": "unknown",
          "jobId": "unknown",
          "dataSourceName": "myglue",
          "jobStartTime": 1699313001843,
          "lastUpdateTime": 1699313001847,
          "error": ""
        }
      }

jobStartTime stay the same and lastUpdateTime will be updated constantly:

      {
        "_index": ".query_execution_request_myglue",
        "_id": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
        "_score": 1,
        "_source": {
          "version": "1.0",
          "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
          "type": "flintindexstate",
          "state": "refreshing",
          "applicationId": "unknown",
          "jobId": "unknown",
          "dataSourceName": "myglue",
          "jobStartTime": 1699313001843,
          "lastUpdateTime": 1699313212525,
          "error": ""
        }
      }

Both jobStartTime and lastUpdateTime changes after recoveer:

scala> spark.sql("RECOVER INDEX JOB flint_myglue_stream_lineitem_tiny_skipping_index")

      {
        "_index": ".query_execution_request_myglue",
        "_id": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
        "_score": 1,
        "_source": {
          "version": "1.0",
          "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
          "type": "flintindexstate",
          "state": "refreshing",
          "applicationId": "unknown",
          "jobId": "unknown",
          "dataSourceName": "myglue",
          "jobStartTime": 1699313076529,
          "lastUpdateTime": 1699313081649,
          "error": ""
        }

Logical deleted index can be recreated if user deletes index data and checkpoint manually (cover 6)

Simulate logical delete by drop index statement. Previous index state doc should be reused by create statement.

scala> spark.sql("DROP SKIPPING INDEX ON stream.lineitem_tiny")

{
  "version": "1.0",
  "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
  "type": "flintindexstate",
  "state": "deleted",
  "applicationId": "unknown",
  "jobId": "unknown",
  "dataSourceName": "myglue",
  "jobStartTime": 1699314176713,
  "lastUpdateTime": 1699314256497,
  "error": ""
}

spark.sql("""
CREATE SKIPPING INDEX ON stream.lineitem_tiny
(l_shipdate VALUE_SET)
WITH (
  auto_refresh = true,
  refresh_interval = '10 seconds',
  checkpoint_location = "s3://checkpoint/job-2",
  extra_options = '{"myglue.ds_tables.http_logs": {"maxFilesPerTrigger": "1"}}'
)
""")

      {
        "_index": ".query_execution_request_myglue",
        "_id": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
        "_score": 1,
        "_source": {
          "version": "1.0",
          "latestId": "ZmxpbnRfbXlnbHVlX3N0cmVhbV9saW5laXRlbV90aW55X3NraXBwaW5nX2luZGV4",
          "type": "flintindexstate",
          "state": "refreshing",
          "applicationId": "unknown",
          "jobId": "unknown",
          "dataSourceName": "myglue",
          "jobStartTime": 1699314720847,
          "lastUpdateTime": 1699314723438,
          "error": ""
        }
      }

Issues Resolved

#123

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.