filter presets and processors based on backend version #537

kamahuan · 2024-12-15T07:05:30Z

Filter presets and processors based on backend version

Description

This PR consists of 3 parts:

Filter presets based on backend version. If backend version is 2.17 or 2.18, show presets which not use ML processors. Keep all the presets for version 2.19.
Remove default ML processor for the enrich data section for both ingest and search pipeline if version is 2.17 or 2.18.
Add ML processor and remove neural processors for version 2.19. Remove ML processors and keep neural processors for version 2.17 and 2.18.

If no datasource is configured during the first render, the create work flow page will have a loading state.

Issues Resolved

NONE

Check List

Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

ohltyler

Getting close! Few comments.

Only remaining piece I think we can add is integrating with the quick-configure fields for the newly-added presets. This will help set defaults for the text embedding & text/image embedding processors, such as the model IDs, fields, etc.

You can look at the quick configure components here for some reference: https://github.com/opensearch-project/dashboards-flow-framework/tree/main/public/pages/workflows/new_workflow

I wouldn't consider a blocker for this PR though. I'll follow up by pulling these changes and testing locally for another sanity test as well, once you've addressed comments. Thanks!

common/constants.ts

common/interfaces.ts

opensearch_dashboards.json

public/pages/workflow_detail/workflow_detail.test.tsx

public/pages/workflows/new_workflow/new_workflow.tsx

ohltyler · 2024-12-17T23:32:34Z

@kamahuan Btw, I'm ok to close the previous PRs, quickly went through comments, nothing else to carry over.

ohltyler

Few remaining things, getting closer. Some more items from previous review I've unresolved as well, please have a look.

In general, it is much easier from my end to add comments / details into the changes made for each comment instead of just resolving, otherwise I have to go digging for each individually. please consider for the future :)

common/constants.ts

opensearch_dashboards.json

public/pages/workflow_detail/workflow_inputs/processors_list.tsx

public/pages/workflow_detail/components/header.tsx

ohltyler · 2024-12-20T18:32:32Z

Cloned locally, and have a 2.19 cluster running. New workflow page is stuck in loading state. No network failures I'm seeing

kamahuan · 2024-12-20T18:34:59Z

Cloned locally, and have a 2.19 cluster running. New workflow page is stuck in loading state. No network failures I'm seeing

Hey Tyler, I think the loading state might due to that we do not have any data source configured. Once we configured a datasource, it will show the presets. The loading state is added to indicate when the user first render the page and not data source is configured.

ohltyler · 2024-12-20T18:38:01Z

Hey Tyler, I think the loading state might due to that we do not have any data source configured. Once we configured a datasource, it will show the presets. The loading state is added to indicate when the user first render the page and not data source is configured.

There should be a check if multiple data source (MDS) is disabled, which is the default / existing behavior, and if so, not do such checks. If it is disabled, we can assume it is 2.19+.

kamahuan · 2024-12-20T18:40:54Z

Hey Tyler, I think the loading state might due to that we do not have any data source configured. Once we configured a datasource, it will show the presets. The loading state is added to indicate when the user first render the page and not data source is configured.

There should be a check if multiple data source (MDS) is disabled, which is the default / existing behavior, and if so, not do such checks. If it is disabled, we can assume it is 2.19+.

Gotcha, will add this check

kamahuan · 2024-12-26T01:14:42Z

Cloned locally, and have a 2.19 cluster running. New workflow page is stuck in loading state. No network failures I'm seeing

Hey Tyler, I add logic to filter when MDS are disabled. These are the screenshots showing:

MDS disabled and version set default to 2.19. Hence all presets should show
MDS not disabled and for version217, it should show a loading state when no data source is configured, and show 3 presets when data source is configured.
MDS not disabled and for version218, it should show a loading state when no data source is configured, and show 3 presets when data source is configured.
MDS not disabled and for version219, it should show a loading state when no data source is configured, and show all presets when data source is configured.

public/pages/workflow_detail/workflow_inputs/processors_list.tsx

public/pages/workflows/new_workflow/new_workflow.tsx

public/pages/workflows/new_workflow/utils.ts

public/pages/workflow_detail/workflow_detail.test.tsx

public/pages/workflows/new_workflow/utils.ts

public/pages/workflows/new_workflow/new_workflow.tsx

ohltyler

Tested locally on a 2.19 cluster, built semantic search and RAG workflows, checked the processor lists etc. Noticed a few regressions on the existing preset values, should be one line changes a few places.

Other than that LGTM! Will approve after those changes are made.

public/pages/workflows/new_workflow/utils.ts

and change enrich data section processors based on the version. Signed-off-by: Kama Huang <[email protected]>

Signed-off-by: Kama Huang <[email protected]>

…ading the page. If disabled, the version should be set to 2.19+ Signed-off-by: Kama Huang <[email protected]>

…ersion to 2.19 Signed-off-by: Kama Huang <[email protected]>

Signed-off-by: Kama Huang <[email protected]>

…workflows file and modify processors for each preset in the fetch function Signed-off-by: Kama Huang <[email protected]>

Signed-off-by: Kama Huang <[email protected]>

ohltyler

LGTM!

ohltyler · 2024-12-31T22:54:16Z

Please follow up by front-porting to main (3.0) branch.

Signed-off-by: Kama Huang <[email protected]> Co-authored-by: Kama Huang <[email protected]> (cherry picked from commit ea08b7d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

(cherry picked from commit ea08b7d) Signed-off-by: Kama Huang <[email protected]> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Kama Huang <[email protected]>

kamahuan marked this pull request as ready for review December 15, 2024 07:06

kamahuan requested review from dbwiddis, owaiskazi19, joshpalis, ohltyler, amitgalitz, jackiehanyang, minalsha and saimedhi as code owners December 15, 2024 07:06

ohltyler reviewed Dec 17, 2024

View reviewed changes

kamahuan force-pushed the backup branch from 7657dfe to d66d230 Compare December 19, 2024 06:48

ohltyler reviewed Dec 20, 2024

View reviewed changes

public/pages/workflow_detail/components/header.tsx Show resolved Hide resolved

kamahuan force-pushed the backup branch from 9999d01 to d25422a Compare December 26, 2024 01:30

ohltyler reviewed Dec 26, 2024

View reviewed changes

This was referenced Dec 26, 2024

filter preset and enrich data section processors based on version #522

Closed

filter processors for the add processor section #524

Closed

Support feature filtering based on datasource #410

Closed

kamahuan mentioned this pull request Dec 27, 2024

[ENHANCEMENT] hide ProcessorList component for SEARCH_REQUEST when version < 2.19.0 #550

Open

kamahuan force-pushed the backup branch 2 times, most recently from 5e853ab to af7ed8c Compare December 28, 2024 08:05

ohltyler reviewed Dec 30, 2024

View reviewed changes

ohltyler reviewed Dec 31, 2024

View reviewed changes

public/pages/workflows/new_workflow/utils.ts Outdated Show resolved Hide resolved

public/pages/workflows/new_workflow/utils.ts Outdated Show resolved Hide resolved

filter preset based on version get from backend

40be91b

and change enrich data section processors based on the version. Signed-off-by: Kama Huang <[email protected]>

Kama Huang added 8 commits December 31, 2024 14:50

address comments made on 12/17

119b38f

Signed-off-by: Kama Huang <[email protected]>

addressing comments made on 12/19. Check if MDS is configured when lo…

d612c1f

…ading the page. If disabled, the version should be set to 2.19+ Signed-off-by: Kama Huang <[email protected]>

add filtering logic. If MDS is configured but disabled, set default v…

61a4c6f

…ersion to 2.19 Signed-off-by: Kama Huang <[email protected]>

clean up logs and format code

4e1c00f

Signed-off-by: Kama Huang <[email protected]>

address comments on 12/26

200b2d9

Signed-off-by: Kama Huang <[email protected]>

fix unit tests after address comments on 12/27

71bf71f

Signed-off-by: Kama Huang <[email protected]>

address comments on 12/30. remove unnecessary try-catch blocks in new…

b57af43

…workflows file and modify processors for each preset in the fetch function Signed-off-by: Kama Huang <[email protected]>

address comments on 12/31

c2cf94b

Signed-off-by: Kama Huang <[email protected]>

kamahuan force-pushed the backup branch from f36a922 to c2cf94b Compare December 31, 2024 22:50

ohltyler approved these changes Dec 31, 2024

View reviewed changes

ohltyler merged commit ea08b7d into opensearch-project:2.x Dec 31, 2024
5 of 6 checks passed

ohltyler added the backport main label Dec 31, 2024

opensearch-trigger-bot bot mentioned this pull request Dec 31, 2024

[Backport main] filter presets and processors based on backend version #556

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

filter presets and processors based on backend version #537

filter presets and processors based on backend version #537

kamahuan commented Dec 15, 2024

ohltyler left a comment

ohltyler commented Dec 17, 2024

ohltyler left a comment

ohltyler commented Dec 20, 2024

kamahuan commented Dec 20, 2024

ohltyler commented Dec 20, 2024

kamahuan commented Dec 20, 2024

kamahuan commented Dec 26, 2024

ohltyler left a comment

ohltyler left a comment

ohltyler commented Dec 31, 2024

filter presets and processors based on backend version #537

filter presets and processors based on backend version #537

Conversation

kamahuan commented Dec 15, 2024

Description

Issues Resolved

Check List

ohltyler left a comment

Choose a reason for hiding this comment

ohltyler commented Dec 17, 2024

ohltyler left a comment

Choose a reason for hiding this comment

ohltyler commented Dec 20, 2024

kamahuan commented Dec 20, 2024

ohltyler commented Dec 20, 2024

kamahuan commented Dec 20, 2024

kamahuan commented Dec 26, 2024

ohltyler left a comment

Choose a reason for hiding this comment

ohltyler left a comment

Choose a reason for hiding this comment

ohltyler commented Dec 31, 2024