Add scheduler_mode index option #415
Conversation
noCharger
force-pushed
the
Add-scheduler-mode
branch
3 times, most recently
from
f2c4a4a to
9f6a0de
Compare
noCharger
requested review from
dai-chen,
rupal-bq,
vamsimanohar,
penghuo,
seankao-az,
anirudha,
kaituo and
YANG-DB
as code owners
noCharger
force-pushed
the
Add-scheduler-mode
branch
4 times, most recently
from
abebc9e to
4358e83
Compare
| @@ -43,9 +44,14 @@ class AutoIndexRefresh(indexName: String, index: FlintSparkIndex) | |||
| !isTableProviderSupported(spark, index), | |||
| "Index auto refresh doesn't support Hive table") | |||
|
|
|||
| // Checkpoint location is required if mandatory option set | |||
| // Checkpoint location is required if mandatory option set or external scheduler is used | |||
There was a problem hiding this comment.
Is External Auto Refresh == Incremental Refresh? I thought we can just use IncrementalIndexRefresh and decide this in factory method in FlintSparkIndexRefresh?
There was a problem hiding this comment.
Is External Auto Refresh == Incremental Refresh?
Physically, yes. But, logically, should we consider it the same as IncrementalIndexRefresh?
I thought we can just use
IncrementalIndexRefreshand decide this in factory method inFlintSparkIndexRefresh?
In the current implementation, the index option remains the source of truth for every operation, therefore it is fine. If RefreshMode is used directly to assign index options in the future, it may result in a mismatch.
One example, if the checkpoint is missing, the output would be
"requirement failed: Checkpoint location is required by incremental refresh"
instead of
"requirement failed: Checkpoint location is required for external scheduler"
There was a problem hiding this comment.
As discussed, the confusion is refreshMode in FlintSparkIndexRefresh makes use of the same enum as the code manipulates index option. This actually should be separated enum which describes the refresh execution mode instead. We can open an refactoring issue for tracking. Thanks!
Signed-off-by: Louis Chu <[email protected]>
Signed-off-by: Louis Chu <[email protected]>
…ark/skipping/worksheet.sc Signed-off-by: Louis Chu <[email protected]>
Signed-off-by: Louis Chu <[email protected]>
Signed-off-by: Louis Chu <[email protected]>
noCharger
force-pushed
the
Add-scheduler-mode
branch
from
05b1d49 to
ed23902
Compare
Signed-off-by: Louis Chu <[email protected]>
Description
Add scheduler_mode index option
Issues Resolved
#417
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.