[BUG] Materialized view creation showing no error, but no view is found

[Swiddis]

What is the bug?
When creating a materialized view connected to Parquet data, no discoverable view is created, but creation cannot be re-attempted due to a collision.

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Create a Parquet table on EMR with some data in it, including a timestamp field.

CREATE EXTERNAL TABLE IF NOT EXISTS mys3.default.elb_logs_parquet (
  type string,
  time timestamp,
  elb string
)
USING parquet
LOCATION 's3://[sample_location]/data/elb-spark-parquet/';

2. Attempt to create a materialized view on this table, aggregating count over time:

CREATE MATERIALIZED VIEW mys3.default.elb_logs_metrics_3
AS
SELECT
  window.start AS `start.time`,
  COUNT(*) AS count
FROM mys3.default.elb_logs_parquet
GROUP BY TUMBLE(`time`, '1 Minutes')
WITH (
  auto_refresh = true,
  checkpoint_location = "s3://[sample_location]/data/checkpoint/job-3",
  watermark_delay = '30 Minutes'
);

3. Observe that the view creation is reported as successful
4. Attempt to query the view:

SELECT * FROM mys3.default.elb_logs_metrics_3;

The query fails with the error:

-- Fail to analyze query. Cause: Table or view not found: mys3.default.elb_logs_metrics_3; line 1 pos 14; 'Project [*] +- 'UnresolvedRelation [mys3, default, elb_logs_metrics_3], [], false 

5. Observe that if you try to recreate the view with the same query as above, it fails with the error:

Fail to write result, cause: Flint index elb_logs_metrics_3 already exists

6. However, if you try to drop the view:

DROP TABLE mys3.default.elb_logs_metrics_3;

You get:

Fail to analyze query. Cause: Table or view not found: mys3.default.elb_logs_metrics_3; line 1 pos 11; 'DropTable false, false +- 'UnresolvedTableOrView [mys3, default, elb_logs_metrics_3], DROP TABLE, true 

After this, I'm not aware of any way to break out of the loop. You need to restart with a new view name and try again. (Incidentally, that's why we're on elb_logs_metrics_3.)

What is the expected behavior?
Either MV creation should fail gracefully, or DROP TABLE should work on the faulty created MV.

What is your host/environment?

• EMR

Do you have any screenshots?
N/A

Do you have any additional context?
N/A

[dai-chen]

@Swiddis This is because currently all Flint index are not registered to Spark catalog and thus no accessible by SparkSQL query. For now by design, it can be only accessed by OpenSearch (DSL, PPL/SQL).