Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support different input transform types #495

Merged

Conversation

ohltyler
Copy link
Member

@ohltyler ohltyler commented Nov 20, 2024

Description

Continuation of #493. Opening to a feature branch so main stays stable. This PR breaks the advanced transform modals entirely.

This PR completes the new UX flows for configuring input transforms. At a high level, this adds support for:

  • toggling between different transform types (static value vs. field vs. JSONPath transform vs. template)
  • configuring JSONPath expressions / transforms via new modal
  • configuring templates (and any sub-transform mappings) via new modal

Implementation details:

  • updates the schema for the input map - this means changes in the UI config, Formik form, and yup schema to be compatible. Basically, we introduce a new field for each input map entry, called transformType, which allows dynamic rendering of different form components based on the type selected. For example, selecting 'Field' will offer a dropdown of the available document or query fields. Selecting 'String' will show a freeform textbox.
  • refactoring and re-creating a lot of the existing MapField component logic inside of the newly-refactored ModelInputs component, but entirely customized for the input map, since it requires too much custom logic to remain a generic component, and was getting overloaded with props anyways. Note: later on, we will likely clean up all of these props, as the generic form is still used for other processors, such as the text chunking processor's field_map field. Also, copies over some of the logic from MapArrayField to dynamically render a 'Configure' button if the map is empty.
  • adds new ConfigureExpressionModal to configure the JSONPath expression in a dedicated modal. Copies over some of the data fetching & transform logic from the existing InputTransformModal, but now it is just at a per-field level.
  • adds new ConfigureTemplateModal to configure a template, as well as any sub-transform variables that can be injected in the template. Copies over some of the data fetching and transform logic from the InputTransformModal, and ConfigurePromptModal.
  • updates the QuickConfigureModal to support the new config such that quick-configure values are populated in the input map appropriately
  • changes in the conversion util fns (config -> form, config -> schema, etc.) to account for the schema changes. Adds logic to parse out some of the values and bucket them into model_config fields (if static), or input_map values (if dynamic). Takes any transforms defined under templates and adds them to the underlying input_map.
  • additions to constants and interfaces to account for the schema changes

Testing

  • confirmed underlying ML processor configs handle all new transform types successfully
  • confirmed configs can build and produce working ingest / search pipelines

Demo video:

screen-capture.7.webm
screen-capture.7.webm

Issues Resolved

Makes progress on #491

Check List

  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@ohltyler ohltyler changed the base branch from main to feature/new-transform November 20, 2024 23:27
@ohltyler ohltyler marked this pull request as ready for review November 20, 2024 23:44
@ohltyler ohltyler changed the title Support different types of input / output maps Update input map schema and reflect changes on UI Nov 20, 2024
@ohltyler ohltyler marked this pull request as draft November 20, 2024 23:48
@ohltyler ohltyler marked this pull request as ready for review November 25, 2024 21:00
@ohltyler ohltyler changed the title Update input map schema and reflect changes on UI Support different input transform types Nov 25, 2024
@ohltyler
Copy link
Member Author

Tests will be updated later; this is all just going to a feature branch for faster iteration.

@saimedhi
Copy link
Collaborator

@ohltyler, I am not able to play the Demo video. Could you add it again please?

@ohltyler
Copy link
Member Author

@ohltyler, I am not able to play the Demo video. Could you add it again please?

Sure, added again. Working fine from chrome for me.

@saimedhi saimedhi merged commit 55521e3 into opensearch-project:feature/new-transform Nov 26, 2024
2 of 3 checks passed
opensearch-trigger-bot bot pushed a commit that referenced this pull request Nov 26, 2024
* Set up the config -> form -> schema -> template transformations for input_map

Signed-off-by: Tyler Ohlsen <[email protected]>

* Handle empty casse

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add new input type selector

Signed-off-by: Tyler Ohlsen <[email protected]>

* Fix bug of model interface not autopopulating in map

Signed-off-by: Tyler Ohlsen <[email protected]>

* Support preset templates

Signed-off-by: Tyler Ohlsen <[email protected]>

* Dynamically render forms; edge case handling of empty docs

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add bucketing logic and add to model config if applicable

Signed-off-by: Tyler Ohlsen <[email protected]>

* Set up new configure_template_modal; add config button when template type selected

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add nested form state in new template modal

Signed-off-by: Tyler Ohlsen <[email protected]>

* Update layout

Signed-off-by: Tyler Ohlsen <[email protected]>

* Update template val form config to support subvars

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add multi input var component & form integration

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add var copying into input var list

Signed-off-by: Tyler Ohlsen <[email protected]>

* Update spacing

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add fetching source data functionality

Signed-off-by: Tyler Ohlsen <[email protected]>

* Fix some spacing; add output jsonfield

Signed-off-by: Tyler Ohlsen <[email protected]>

* committing but broken

Signed-off-by: Tyler Ohlsen <[email protected]>

* Show transformed prompt

Signed-off-by: Tyler Ohlsen <[email protected]>

* add spacer

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add nested template vars to input map

Signed-off-by: Tyler Ohlsen <[email protected]>

* Set up base expression modal

Signed-off-by: Tyler Ohlsen <[email protected]>

* Get expression modal working

Signed-off-by: Tyler Ohlsen <[email protected]>

* Add dynamic rendering if template or expression is populated

Signed-off-by: Tyler Ohlsen <[email protected]>

---------

Signed-off-by: Tyler Ohlsen <[email protected]>
(cherry picked from commit 55521e3)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants