Skip to content

Commit

Permalink
[Ingest pipelines] Update readmes (#78350)
Browse files Browse the repository at this point in the history
  • Loading branch information
alisonelizabeth authored Sep 30, 2020
1 parent caa2605 commit 272e850
Show file tree
Hide file tree
Showing 4 changed files with 110 additions and 26 deletions.
81 changes: 79 additions & 2 deletions x-pack/plugins/ingest_pipelines/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,14 +11,91 @@ It requires a Basic license and the following cluster privileges: `manage_pipeli

## Development

A new app called Ingest Node Pipelines is registered in the Management section and follows a typical CRUD UI pattern. The client-side portion of this app lives in [public/application](public/application) and uses endpoints registered in [server/routes/api](server/routes/api).
A new app called Ingest Node Pipelines is registered in the Management section and follows a typical CRUD UI pattern. The client-side portion of this app lives in [public/application](public/application) and uses endpoints registered in [server/routes/api](server/routes/api). For more information on the pipeline processors editor component, check out the [component readme](public/application/components/pipeline_processors_editor/README.md).

See the [kibana contributing guide](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md) for instructions on setting up your development environment.

### Test coverage

The app has the following test coverage:

- Complete API integration tests
- API integration tests
- Smoke-level functional test
- Client-integration tests

### Quick steps for manual testing

You can run the following request in Console to create an ingest node pipeline:

```
PUT _ingest/pipeline/test_pipeline
{
"description": "_description",
"processors": [
{
"set": {
"field": "field1",
"value": "value1"
}
},
{
"rename": {
"field": "dont_exist",
"target_field": "field1",
"ignore_failure": true
}
},
{
"rename": {
"field": "foofield",
"target_field": "new_field",
"on_failure": [
{
"set": {
"field": "field2",
"value": "value2"
}
}
]
}
},
{
"drop": {
"if": "false"
}
},
{
"drop": {
"if": "true"
}
}
]
}
```

Then, go to the Ingest Node Pipelines UI to edit, delete, clone, or view details of the pipeline.

To simulate a pipeline, go to the "Edit" page of your pipeline. Click the "Add documents" link under the "Processors" section. You may add the following sample documents to test the pipeline:

```
// The first document in this example should trigger the on_failure processor in the pipeline, while the second one should succeed.
[
{
"_index": "my_index",
"_id": "id1",
"_source": {
"foo": "bar"
}
},
{
"_index": "my_index",
"_id": "id2",
"_source": {
"foo": "baz",
"foofield": "bar"
}
}
]
```

Alternatively, you can add a document from an existing index, or create some sample data of your own. Afterward, click the "Run the pipeline" button to view the output.
Original file line number Diff line number Diff line change
Expand Up @@ -7,18 +7,37 @@ pipeline.

## Editor components

The top-level API consists of 3 pieces that enable the maximum amount
of flexibility for consuming code to determine overall layout.
The top-level API consists of two pieces:

- PipelineProcessorsEditorContext
- ProcessorsEditor
- GlobalOnFailureProcessorsEditor
- ProcessorsEditorContextProvider
- PipelineProcessorsEditor

The editor components must be wrapped inside of the context component
The editor component must be wrapped inside of the context component
as this is where the shared processors state is contained.

## Load JSON button
Example usage from the [PipelineFormFields](../pipeline_form/pipeline_form_fields.tsx) component:

This component is totally standalone. It gives users a button that
```
<ProcessorsEditorContextProvider
onFlyoutOpen={onEditorFlyoutOpen}
onUpdate={onProcessorsUpdate}
value={{ processors, onFailure }}
>
<PipelineProcessorsEditor onLoadJson={onLoadJson} />
</ProcessorsEditorContextProvider>
```

The editor has a dependency on `KibanaContextProvider`, which is defined in the main app's `index.tsx` file. Note that the editor also relies on imports from `public/shared_imports.ts` and `common/types.ts`.

### ProcessorsEditorContextProvider
This component manages state for the processors, as well as state for the test pipeline functionality.

### PipelineProcessorsEditor
This component is responsible for building the layout of the processors editor.

It contains the processor and on-failure processor editors. It also includes the following capabilities that are rendered within the processors header:

- **Load JSON button:** This component gives users a button that
presents a modal for loading a pipeline. It does some basic
validation on the JSON to ensure that it is correct.
- **Test pipeline actions:** This component presents users with a toolbar to test a pipeline. It includes a flyout where users can add sample documents. It issues a request to simulate the pipeline and displays the output. Once the request is successful, a user can use the documents dropdown to view the results for a particular document.
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,7 @@ import {
apiService,
} from '../../../services';

import {
ProcessorsEditorContextProvider,
Props,
GlobalOnFailureProcessorsEditor,
ProcessorsEditor,
} from '../';
import { TestPipelineActions } from '../';
import { ProcessorsEditorContextProvider, Props, PipelineProcessorsEditor } from '../';

import { initHttpRequests } from './http_requests.helpers';

Expand Down Expand Up @@ -105,9 +99,7 @@ const testBedSetup = registerTestBed<TestSubject>(
(props: Props) => (
<KibanaContextProvider services={appServices}>
<ProcessorsEditorContextProvider {...props}>
<TestPipelineActions />
<ProcessorsEditor />
<GlobalOnFailureProcessorsEditor />
<PipelineProcessorsEditor onLoadJson={jest.fn()} />
</ProcessorsEditorContextProvider>
</KibanaContextProvider>
),
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,16 +4,12 @@
* you may not use this file except in compliance with the Elastic License.
*/

export { PipelineProcessorsContextProvider, Props } from './context';

export { ProcessorsEditorContextProvider } from './context';

export { ProcessorsEditor, GlobalOnFailureProcessorsEditor } from './editors';
export { Props, ProcessorsEditorContextProvider } from './context';

export { OnUpdateHandlerArg, OnUpdateHandler } from './types';

export { SerializeResult } from './serialize';

export { LoadFromJsonButton, OnDoneLoadJsonHandler, TestPipelineActions } from './components';
export { OnDoneLoadJsonHandler } from './components';

export { PipelineProcessorsEditor } from './pipeline_processors_editor';

0 comments on commit 272e850

Please sign in to comment.