Skip to content

Commit

Permalink
Add Index Template APIs (#7635)
Browse files Browse the repository at this point in the history 
* Add Index Template APIs

Signed-off-by: Archer <[email protected]>

* Add Index Template APIs

Signed-off-by: Archer <[email protected]>

* Add Delete template API

Signed-off-by: Archer <[email protected]>

* Add delete API

Signed-off-by: Archer <[email protected]>

* Add component template.

Signed-off-by: Archer <[email protected]>

* Add missing parameter

Signed-off-by: Archer <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _api-reference/index-apis/create-index-template.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update _api-reference/index-apis/create-index-template.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update _api-reference/index-apis/component-template.md

Signed-off-by: Heather Halter <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Sarthak Aggarwal <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Update component-template.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update create-index-template.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Update delete-index-template.md

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Heather Halter <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>

---------

Signed-off-by: Archer <[email protected]>
Signed-off-by: Naarcha-AWS <[email protected]>
Signed-off-by: Heather Halter <[email protected]>
Co-authored-by: Heather Halter <[email protected]>
Co-authored-by: Sarthak Aggarwal <[email protected]>
Co-authored-by: Nathan Bower <[email protected]>
(cherry picked from commit 47d64f8)
Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
  • Loading branch information
@github-actions @hdhalter
@sarthakaggarwal97 @natebower
4 people committed Jul 31, 2024
1 parent 37b8c72 commit 77e4e5a
Show file tree
Hide file tree
Showing 5 changed files with 470 additions and 3 deletions.
154 changes: 154 additions & 0 deletions _api-reference/index-apis/component-template.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
layout: default
title: Create or update component template
parent: Index APIs
nav_order: 29
---

# Create or update component template

You can use the Component Template API to create or update a component template. A component template is a reusable building block that defines settings, mappings, and aliases that can be shared across multiple index templates.

An index template can be constructed using multiple component templates. To incorporate a component template into an index template, you need to list it in the `composed_of` section of the index template. Component templates are only applied to newly created data streams and indexes that match the criteria specified in the index template.

If any settings or mappings are directly defined in the index template or the index creation request, those settings will take precedence over the settings or mappings specified in a component template.

Component templates are used solely during the process of index creation. For data streams, this includes the creation of the data stream itself and the creation of the backing indexes that support the stream. Modifications made to component templates will not affect existing indexes, including the backing indexes of a data stream.

## Path and HTTP methods

The PUT method adds a component template and accepts both query parameters and a request body. The GET method retrieves information about an existing component template and accepts only query parameters:

```json
PUT _component_template/<component-template-name>
GET _component_template/<component-template-name>
```

## Path parameters

Parameter | Data type | Description
:--- | :--- | :---
`component-template-name` | String | The name of the component template.

## Query parameters

The following optional query parameters are supported.

Parameter | Data type | Description
:--- | :--- | :---
`create` | Boolean | When true, the API cannot replace or update any existing index templates. Default is `false`.
`cluster_manager_timeout` | Time | The amount of time to wait for a connection to the cluster manager node. Default is `30s`.
`timeout` | Time | The amount of time for the operation to wait for a response. Default is `30s`.

## Request fields

The following options can be used in the request body to customize the index template.


Parameter | Data type | Description
:--- | :--- | :---
`template` | Object | The template that includes the `aliases`, `mappings`, or `settings` for the index. For more information, see [#template]. Required.
`version` | Integer | The version number used to manage index templates. Version numbers are not automatically set by OpenSearch. Optional.
`_meta` | Object | The metadata that provides details about the index template. Optional.
`allow_auto_create` | Boolean | When `true`, indexes can be automatically created with this template even if the `actions.auto_create_index` is disabled. When `false`, indexes and data streams matching the template cannot be automatically created. Optional.
`deprecated` | Boolean | When `true`, the component template is deprecated. If deprecated, OpenSearch will output a warning whenever the template is referenced.


### Template

You can use the following objects with the `template` option in the request body.

#### `alias`

The name of the alias to associate with the template as a key. Required when the `template` option exists in the request body. This option supports multiple aliases.

The object body contains the following optional alias parameters.

Parameter | Data type | Description
:--- | :--- | :---
`filter` | Query DSL object | The query that limits the number of documents that the alias can access.
`index_routing` | String | The value that routes the indexing operations to a specific shard. When specified, overwrites the `routing` value for the indexing operations.
`is_hidden` | Boolean | When `true`, the alias is hidden. Default is false. All alias indexes must have matching values for this setting.
`is_write_index` | Boolean | When `true`, the index is the write index for the alias. Default is `false`.
`routing` | String | The value used to route index and search operations to a specific shard.
`search_routing` | String | The value used to write search operations to a specific shard. When specified, this option overwrites the `routing` value for the search operations.

#### `mappings`

The field mappings that exist in the index. For more information, see [Mappings and field types](https://opensearch.org/docs/latest/field-types/). Optional.

#### `settings`

Any configuration options for the index. For more information, see [Index settings](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/).

## Example requests

The following example requests show how to use the Component Template API.

### Create with index aliases

The following example request creates a component template including index aliases:

```json
PUT _component_template/alias_template
{
"template": {
"settings" : {
"number_of_shards" : 1
},
"aliases" : {
"alias1" : {},
"alias2" : {
"filter" : {
"term" : {"user.id" : "hamlet" }
},
"routing" : "shard-1"
},
"{index}-alias" : {}
}
}
}
```

### Adding component versioning


The following example adds a `version` number to a component template which simplifies template management for external systems:

```json
PUT /_component_template/version_template
{
"template": {
"settings" : {
"number_of_shards" : 1
}
},
"version": 3
}
```
{% include copy-curl.html %}

## Adding template metadata

The following example request uses the `meta` parameter to add metadata to the index template. All metadata is stored in the cluster state.

```json
PUT /_component_template/meta_template
{
"template": {
"settings" : {
"number_of_shards" : 1
}
},
"_meta": {
"description": "Where art thou",
"serialization": {
"class": "MyIndexTemplate",
"id": 12
}
}
}
```



240 changes: 240 additions & 0 deletions _api-reference/index-apis/create-index-template.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,240 @@
---
layout: default
title: Create or update index template
parent: Index APIs
nav_order: 26
---

# Create or update index template

You can use the Create or Update Index Template API to create indexes with predefined mappings and settings as well as update existing index templates.

## Path and HTTP methods

```json
PUT _index_template/<template-name>
POST _index_template/<template-name>
```

## Path parameters

Parameter | Data type | Description
:--- | :--- | :---
`template-name` | String | The name of the index template.

## Query parameters

The following optional query parameters are supported.

Parameter | Data type | Description
:--- | :--- | :---
`create` | Boolean | When true, the API cannot replace or update any existing index templates. Default is `false`.
`cluster_manager_timeout` | Time | The amount of time to wait for a connection to the cluster manager node. Default is `30s`.

## Request body options

The following options can be used in the request body to customize the index template.


Parameter | Type | Description
:--- | :--- | :---
`index_patterns` | String array | An array of wildcard expressions that match the names of data streams and indexes created during template creation. Required.
`composed_of` | String array | An ordered list of component template names. These templates are merged using the specified order. For more information, see [Using multiple component templates](#using-multiple-component-templates). Optional.
`data_stream` | Object | When used, the request creates data streams and any backing indexes based on the template. This setting requires a matching index template. It can also be used with the `hidden` setting, which, when set to `true`, hides the data stream backing indexes. Optional.
`_meta` | Object | Optional metadata that provides details about the index template. Optional.
`priority` | Integer | A number that determines which index templates take precedence during the creation of a new index or data stream. OpenSearch chooses the template with the highest priority. When no priority is given, the template is assigned a `0`, signifying the lowest priority. Optional.
`template` | Object | The template that includes the `aliases`, `mappings`, or `settings` for the index. For more information, see [#template]. Optional.
`version` | Integer | The version number used to manage index templates. Version numbers are not automatically set by OpenSearch. Optional.


### Template

You can use the following objects with the `template` option in the request body.

#### `alias`

The name of the alias to associate with the template as a key. Required when the `template` option exists in the request body. This option supports multiple aliases.

The object body contains the following optional alias parameters.

Parameter | Data type | Description
:--- | :--- | :---
`filter` | Query DSL object | The query that limits the number of documents that the alias can access.
`index_routing` | String | The value that routes indexing operations to a specific shard. When specified, overwrites the `routing` value for indexing operations.
`is_hidden` | Boolean | When `true`, the alias is hidden. Default is `false`. All alias indexes must have matching values for this setting.
`is_write_index` | Boolean | When `true`, the index is the write index for the alias. Default is `false`.
`routing` | String | The value used to route index and search operations to a specific shard.
`search_routing` | String | The value used to write specific search operations to a specific shard. When specified, this option overwrites the `routing` value for search operations.

#### `mappings`

The field mappings that exist in the index. For more information, see [Mappings and field types](https://opensearch.org/docs/latest/field-types/). Optional.

#### `settings`

Any configuration options for the index. For more information, see [Index settings](https://opensearch.org/docs/latest/install-and-configure/configuring-opensearch/index-settings/).

## Example requests

The following examples show how to use the Create or Update Index Template API.

### Index template with index aliases

The following example request includes index aliases in the template:

```json
PUT _index_template/alias-template
{
"index_patterns" : ["sh*"],
"template": {
"settings" : {
"number_of_shards" : 1
},
"aliases" : {
"alias1" : {},
"alias2" : {
"filter" : {
"term" : {"user.id" : "hamlet" }
},
"routing" : "shard-1"
},
"{index}-alias" : {}
}
}
}
```
{% include copy-curl.html %}

### Using multiple matching templates

When multiple index templates match the name of a new index or data stream, the template with the highest priority is used. For example, the following two requests create index templates with different priorities:

```json
PUT /_index_template/template_one
{
"index_patterns" : ["h*"],
"priority" : 0,
"template": {
"settings" : {
"number_of_shards" : 1,
"number_of_replicas": 0
},
"mappings" : {
"_source" : { "enabled" : false }
}
}
}

PUT /_index_template/template_two
{
"index_patterns" : ["ha*"],
"priority" : 1,
"template": {
"settings" : {
"number_of_shards" : 2
},
"mappings" : {
"_source" : { "enabled" : true }
}
}
}
```
{% include copy-curl.html %}

For indexes that start with `ha`, the `_source` is enabled. Because only `template_two` is applied, the index will have two primary shards and one replica.

Overlapping index patterns given the same priority are not allowed. An error will occur when attempting to create a template matching an existing index template with identical priorities.
{: .note}

### Adding template versioning

The following example request adds a `version` number to an index template, which simplifies template management for external systems:

```json
PUT /_index_template/template_one
{
"index_patterns" : ["mac", "cheese"],
"priority" : 0,
"template": {
"settings" : {
"number_of_shards" : 1
}
},
"version": 1
}
```
{% include copy-curl.html %}


### Adding template metadata

The following example request uses the `meta` parameter to add metadata to the index template. All metadata is stored in the cluster state:

```json
PUT /_index_template/template_one
{
"index_patterns": ["rom", "juliet"],
"template": {
"settings" : {
"number_of_shards" : 2
}
},
"_meta": {
"description": "Where art thou",
"serialization": {
"class": "MyIndexTemplate",
"id": 12
}
}
}
```

### Data stream definition

Include a `data_stream` object to use an index template for data streams, as shown in the following example request:

```json
PUT /_index_template/template_1
{
"index_patterns": ["logs-*"],
"data_stream": { }
}
```

## Using multiple component templates

When using multiple component templates with the `composed_of` field, the component templates are merged in the specified order. Next, all mappings, settings, and aliases from the parent index template of the component are merged. Lastly, any configuration options added to the index requests are merged.

In the following example request, an index with `h*` has two merged primary shards. If the order in the request body were reversed, then the index would have one primary shard:

```json
PUT /_component_template/template_with_1_shard
{
"template": {
"settings": {
"index.number_of_shards": 1
}
}
}

PUT /_component_template/template_with_2_shards
{
"template": {
"settings": {
"index.number_of_shards": 2
}
}
}

PUT /_index_template/template_1
{
"index_patterns": ["h*"],
"composed_of": ["template_with_1_shard", "template_with_2_shards"]
}
```
{% include copy-curl.html %}


Recursive merging is used for mapping definition and root options such as `dynamic_templates` and `meta`, meaning that when an earlier component contains a `meta` block, new `meta` entries are added to the end of the metadata in the index. Any entries containing a preexisting key are overwritten.


Loading

0 comments on commit 77e4e5a

Please sign in to comment.