You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
What is the bug?
Metadata log doesn't sanitize index name before generating log entry id. This will cause SHOW FLINT INDEX and FlintSpark.describeIndexes not able to find some index with special characters.
This will mostly happen when the custom datasource table is allowed to have special characters. For example, if a custom datasource has a table test/special, then its skipping index and covering index will both contain the special character /. #215 allows for such index to be created in OpenSearch, by percent encoding the special characters.
In FlintSpark.describeIndexes, when trying to getAllIndexMetadata, we get a Map of (sanitized index name -> metadata). Then we fetch metadata log entry using the returned index name. However, when creating metadata log entry for an index, the entry id is generated using unsanitized index name. Therefore it couldn't find the corresponding log entry.
How can one reproduce the bug?
Spark doesn't allow us to create table name with special characters so we can't directly test it. We can simulate above behavior by creating covering index or materialized view with special characters in their name, then try to SHOW FLINT INDEX. The returned index status will be unavailable
CREATE INDEX `test/special` ON mys3.default.http_logs (status)
SHOW FLINT INDEX IN mys3
[
"flint_mys3_default_http_logs_test/special_index",
"covering",
"default",
"http_logs",
"test/special",
false,
"unavailable" // should be "active"
],
The text was updated successfully, but these errors were encountered:
seankao-az
changed the title
[BUG] Metadata log doesn't sanitize index name before generating log entry id
[BUG] OpenSearch metadata log doesn't sanitize index name before generating log entry id
Aug 9, 2024
What is the bug?
Metadata log doesn't sanitize index name before generating log entry id. This will cause
SHOW FLINT INDEX
andFlintSpark.describeIndexes
not able to find some index with special characters.This will mostly happen when the custom datasource table is allowed to have special characters. For example, if a custom datasource has a table
test/special
, then its skipping index and covering index will both contain the special character/
. #215 allows for such index to be created in OpenSearch, by percent encoding the special characters.In
FlintSpark.describeIndexes
, when trying togetAllIndexMetadata
, we get a Map of (sanitized index name -> metadata). Then we fetch metadata log entry using the returned index name. However, when creating metadata log entry for an index, the entry id is generated using unsanitized index name. Therefore it couldn't find the corresponding log entry.How can one reproduce the bug?
Spark doesn't allow us to create table name with special characters so we can't directly test it. We can simulate above behavior by creating covering index or materialized view with special characters in their name, then try to
SHOW FLINT INDEX
. The returned index status will beunavailable
CREATE INDEX `test/special` ON mys3.default.http_logs (status)
SHOW FLINT INDEX IN mys3
The text was updated successfully, but these errors were encountered: