Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update and edit logsdb docs for logsdb / synthetic source GA #118303

Merged
merged 27 commits into from
Dec 11, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
5f5db01
Update licensing; fix screenshots; edit generally
marciw Dec 9, 2024
b908878
Small edit for clarity and style
marciw Dec 9, 2024
4d8736e
Merge branch 'main' into mw-logsdb
marciw Dec 9, 2024
4a01131
Update docs/reference/index-modules.asciidoc
marciw Dec 10, 2024
ce7a558
Apply changes from review
marciw Dec 10, 2024
737ea70
Address review comments
marciw Dec 10, 2024
b2a34e2
Merge branch 'main' into mw-logsdb
marciw Dec 10, 2024
c5cbf91
Match similar change from review
marciw Dec 10, 2024
d0cf775
Merge branch 'mw-logsdb' of github.com:marciw/elasticsearch into mw-l…
marciw Dec 10, 2024
f01ce45
More changes from review
marciw Dec 10, 2024
0ddf0e3
Apply suggestions from review
marciw Dec 10, 2024
0765a05
Apply suggestions from review
marciw Dec 10, 2024
c200598
Update docs/reference/data-streams/logs.asciidoc
marciw Dec 10, 2024
718aaf8
Apply suggestions from review
marciw Dec 10, 2024
80f9dfa
Apply suggestions from review
marciw Dec 10, 2024
adc0941
Merge branch 'mw-logsdb' of github.com:marciw/elasticsearch into mw-l…
marciw Dec 10, 2024
468c776
Merge branch 'main' into mw-logsdb
marciw Dec 10, 2024
d9dc4da
Change to general subscription note
marciw Dec 10, 2024
773cb92
Merge branch 'main' into mw-logsdb
marciw Dec 10, 2024
140c3f8
Apply suggestions from review
marciw Dec 10, 2024
7c3fb05
Apply suggestions from review
marciw Dec 10, 2024
e9bfa0e
Apply suggestions from review; additional edits
marciw Dec 10, 2024
4ebd4fa
Merge branch 'main' into mw-logsdb
marciw Dec 10, 2024
c9419d1
Apply suggestions from review; clarity tweaks
marciw Dec 11, 2024
61da7ab
Merge branch 'main' into mw-logsdb
marciw Dec 11, 2024
4620a37
Restore previous paragraph structure and context
marciw Dec 11, 2024
fccda9c
Merge branch 'main' into mw-logsdb
marciw Dec 11, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
197 changes: 103 additions & 94 deletions docs/reference/data-streams/logs.asciidoc

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion docs/reference/data-streams/tsds.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ metrics data. Only use a TSDS if you typically add metrics data to {es} in near
real-time and `@timestamp` order.

A TSDS is only intended for metrics data. For other timestamped data, such as
logs or traces, use a regular data stream.
logs or traces, use a <<logs-data-stream,logs data stream>> or regular data stream.

[discrete]
[[differences-from-regular-data-stream]]
Expand Down
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file not shown.
Binary file modified docs/reference/images/index-mgmt/management-index-templates.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
5 changes: 2 additions & 3 deletions docs/reference/index-modules.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -113,10 +113,9 @@ Index mode supports the following values:

`standard`::: Standard indexing with default settings.

`time_series`::: Index mode optimized for storage of metrics documented in <<tsds-index-settings,TSDS Settings>>.
`tsds`::: _(data streams only)_ Index mode optimized for storage of metrics. For more information, see <<tsds-index-settings>>.

`logsdb`::: Index mode optimized for storage of logs. It applies default sort settings on the `hostname` and `timestamp` fields and uses <<synthetic-source,synthetic `_source`>>. <<index-modules-index-sorting,Index sorting>> on different fields is still allowed.
preview:[]
`logsdb`::: _(data streams only)_ Index mode optimized for <<logs-data-stream,logs>>.

[[routing-partition-size]] `index.routing_partition_size`::

Expand Down
2 changes: 1 addition & 1 deletion docs/reference/indices/index-mgmt.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ This value is the time period for which your data is guaranteed to be stored. Da
Elasticsearch at a later time.

[role="screenshot"]
image::images/index-mgmt/management-data-stream.png[Data stream details]
image::images/index-mgmt/management-data-stream-fields.png[Data stream details]

* To view more information about a data stream, such as its generation or its
current index lifecycle policy, click the stream's name. From this view, you can navigate to *Discover* to
Expand Down
6 changes: 3 additions & 3 deletions docs/reference/indices/put-index-template.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -115,10 +115,10 @@ See <<create-index-template,create an index template>>.

`index_mode`::
(Optional, string) Type of data stream to create. Valid values are `null`
(regular data stream) and `time_series` (<<tsds,time series data stream>>).
(standard data stream), `time_series` (<<tsds,time series data stream>>) and `logsdb`
(<<logs-data-stream,logs data stream>>).
+
If `time_series`, each backing index has an `index.mode` index setting of
`time_series`.
The template's `index_mode` sets the `index.mode` of the backing index.
=====

`index_patterns`::
Expand Down
11 changes: 2 additions & 9 deletions docs/reference/mapping/fields/synthetic-source.asciidoc
Original file line number Diff line number Diff line change
@@ -1,17 +1,10 @@
[[synthetic-source]]
==== Synthetic `_source`

IMPORTANT: Synthetic `_source` is Generally Available only for TSDB indices
(indices that have `index.mode` set to `time_series`). For other indices,
synthetic `_source` is in technical preview. Features in technical preview may
be changed or removed in a future release. Elastic will work to fix
any issues, but features in technical preview are not subject to the support SLA
of official GA features.

Though very handy to have around, the source field takes up a significant amount
of space on disk. Instead of storing source documents on disk exactly as you
send them, Elasticsearch can reconstruct source content on the fly upon retrieval.
Enable this by using the value `synthetic` for the index setting `index.mapping.source.mode`:
To enable this https://www.elastic.co/subscriptions[subscription] feature, use the value `synthetic` for the index setting `index.mapping.source.mode`:

[source,console,id=enable-synthetic-source-example]
----
Expand All @@ -30,7 +23,7 @@ PUT idx
----
// TESTSETUP

While this on the fly reconstruction is *generally* slower than saving the source
While this on-the-fly reconstruction is _generally_ slower than saving the source
documents verbatim and loading them at query time, it saves a lot of storage
space. Additional latency can be avoided by not loading `_source` field in queries when it is not needed.

Expand Down