Skip to content

Commit

Permalink
[DOCS] Move snapshot repo types to separate pages (elastic#82826)
Browse files Browse the repository at this point in the history
With elastic#81870, the Azure, GCS, and S3 repository types have separate, dedicated pages in the Elasticsearch guide. For consistency, this PR creates separate pages for the shared file system, read-only URL, and source-only repository types.

Related changes:

- Adds redirects to the plugins docs
- Fixes a few breaking changes that refer to the Azure, GCS, and S3 repositories as plugins.

Co-authored-by: Adam Locke <[email protected]>
  • Loading branch information
jrodewig and Adam Locke authored Jan 26, 2022
1 parent 86174d6 commit cb6265f
Show file tree
Hide file tree
Showing 10 changed files with 358 additions and 296 deletions.
74 changes: 74 additions & 0 deletions docs/plugins/redirects.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -71,3 +71,77 @@ See {ref}/monitor-elasticsearch-cluster.html[{stack} monitoring].
=== Security plugins

See {ref}/secure-cluster.html[{stack} security].

[role="exclude",id="repository-azure"]
=== Azure repository plugin

// tag::azure-repo-migration[]
The Azure repository plugin is now included in {es}.
See {ref}/repository-azure.html[Azure repository].
// end::azure-repo-migration[]

[role="exclude",id="repository-azure-usage"]
=== Azure repository plugin

include::redirects.asciidoc[tag=azure-repo-migration]

[role="exclude",id="repository-azure-client-settings"]
=== Azure repository plugin

include::redirects.asciidoc[tag=azure-repo-migration]

[role="exclude",id="repository-azure-repository-settings"]
=== Azure repository plugin

include::redirects.asciidoc[tag=azure-repo-migration]

[role="exclude",id="repository-azure-validation"]
=== Azure repository plugin

include::redirects.asciidoc[tag=azure-repo-migration]

[role="exclude",id="repository-s3"]
=== S3 repository plugin

// tag::s3-repo-migration[]
The S3 repository plugin is now included in {es}.
See {ref}/repository-s3.html[S3 repository].
// end::s3-repo-migration[]

[role="exclude",id="repository-s3-usage"]
=== S3 repository plugin

include::redirects.asciidoc[tag=s3-repo-migration]

[role="exclude",id="repository-s3-client"]
=== S3 repository plugin

include::redirects.asciidoc[tag=s3-repo-migration]

[role="exclude",id="repository-s3-repository"]
=== S3 repository plugin

include::redirects.asciidoc[tag=s3-repo-migration]

[role="exclude",id="repository-gcs"]
=== Google Cloud Storage repository plugin

// tag::gcs-repo-migration[]
The Google Cloud Storage repository plugin is now included in {es}.
See {ref}/repository-gcs.html[Google Cloud Storage repository].
// end::gcs-repo-migration[]

[role="exclude",id="repository-gcs-usage"]
=== Google Cloud Storage repository plugin

include::redirects.asciidoc[tag=gcs-repo-migration]

[role="exclude",id="repository-gcs-client"]
=== Google Cloud Storage repository plugin

include::redirects.asciidoc[tag=gcs-repo-migration]

[role="exclude",id="repository-gcs-repository"]
=== Google Cloud Storage repository plugin

include::redirects.asciidoc[tag=gcs-repo-migration]
Original file line number Diff line number Diff line change
Expand Up @@ -875,11 +875,10 @@ previously be used to enable compression for all shared file system repositories
The `repositories.fs.compress` setting has been removed.
*Impact* +
Use the repository specific `compress` setting to enable compression. See
{ref}/snapshots-register-repository.html[Register a snapshot repository] for
information on the `compress` setting.
Discontinue use of the `repositories.fs.compress` node-level setting.
Discontinue use of the `repositories.fs.compress` node-level setting. Use the
repository-specific `compress` setting to enable compression instead. Refer to
{ref}/snapshots-filesystem-repository.html#filesystem-repository-settings[Shared
file system repository settings].
====
//end::notable-breaking-changes[]

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -991,21 +991,18 @@ Previously, the default value for `compress` was `false`. The default has been c
This change will affect both newly created repositories and existing repositories where `compress=false` has not been
explicitly specified.
For more information on the compress option, see
{ref}/snapshots-register-repository.html[Register a snapshot repository].
*Impact* +
Update your workflow and applications to assume a default value of `true` for
the `compress` parameter.
====

.The S3 repository plugin now uses a DNS-style access pattern by default.
.S3 snapshot repositories now use a DNS-style access pattern by default.
[%collapsible]
====
*Details* +
Starting in version 7.4 the `repository-s3` plugin does not use the
Starting in version 7.4, `s3` snapshot repositories no longer use the
now-deprecated path-style access pattern by default. In versions 7.0, 7.1, 7.2
and 7.3 the `repository-s3` plugin always used the path-style access pattern.
and 7.3 `s3` snapshot repositories always used the path-style access pattern.
This is a breaking change for deployments that only support path-style access
but which are recognized as supporting DNS-style access by the AWS SDK. This
breaking change was made necessary by
Expand Down
183 changes: 21 additions & 162 deletions docs/reference/snapshot-restore/apis/put-repo-api.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -75,183 +75,42 @@ You can manually perform this verification using the

[[put-snapshot-repo-api-request-type]]
`type`::
+
--
(Required, string)
Repository type.

+
.Valid values for `type`
[%collapsible%open]
====
`fs`::
Shared file system repository. Repositories of this type use a shared file
system to store snapshots. This file system must accessible to all master and
data nodes in the cluster.
+
IMPORTANT: To register a shared file system repository, you must mount the same
shared filesystem to the same location on all master and data nodes. This
location must be registered in the `path.repo` setting on all master and data
nodes in the cluster.
+
See <<snapshots-filesystem-repository>>.
[xpack]#`source`#::
Source-only repository. You can use source-only repositories to create minimal,
source-only snapshots that take up to 50% less space on disk.
+
Source-only snapshots are only supported if the <<mapping-source-field,`_source`
field>> is enabled and no
<<source-filtering,source-filtering>> is applied.
+
WARNING: Source-only snapshots contain stored fields and index metadata. They do
not include index or doc values structures and are not searchable when restored.
After restoring a source-only snapshot, you must <<docs-reindex,reindex>> the
data into a new index.
+
See <<snapshots-source-only-repository>>.
`url`::
URL repository. Repositories of this type are read-only
for the cluster. This means the cluster can retrieve or restore snapshots from
the repository but cannot write or create snapshots in it.
+
You can use URL repositories as an alternative way to give a cluster read-only
access to a shared file system (`fs`) repository.
+
See <<snapshots-read-only-repository>>.
====
`azure`:: <<repository-azure>>
`gcs`:: <<repository-gcs>>
`s3`:: <<repository-s3>>
`fs`:: <<snapshots-filesystem-repository>>
`source`:: <<snapshots-source-only-repository>>
`url`:: <<snapshots-read-only-repository>>
More repository types are available through these official
plugins:
Other repository types are available through official plugins:
* <<repository-s3,repository-s3>> for S3 repository support
* {plugins}/repository-hdfs.html[repository-hdfs] for HDFS repository support in
Hadoop environments
* <<repository-azure,repository-azure>> for Azure storage repositories
* <<repository-gcs,repository-gcs>> for Google Cloud Storage repositories
--
`hfds`:: {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS) repository]
====

[[put-snapshot-repo-api-settings-param]]
`settings`::
+
--
(Required, object)
Contains settings for the repository.

The following `settings` properties are valid for all repository types:

.Properties of `settings`
[%collapsible%open]
====
`chunk_size`::
(Optional, <<byte-units,byte value>>)
Maximum size of files in snapshots. In snapshots, files larger than this are
broken down into chunks of this size or smaller. Defaults to `null` (unlimited
file size).
`compress`::
(Optional, Boolean)
If `true`, metadata files, such as index mappings and settings, are compressed
in snapshots. Data files are not compressed. Defaults to `true`.
`max_number_of_snapshots`::
(Optional, integer)
Maximum number of snapshots the repository can contain.
Defaults to `Integer.MAX_VALUE`, which is `2^31-1` or `2147483647`.
`max_restore_bytes_per_sec`::
(Optional, <<byte-units,byte value>>)
Maximum snapshot restore rate per node. Defaults to unlimited. Note
that restores are also throttled through <<recovery,recovery settings>>.
`max_snapshot_bytes_per_sec`::
(Optional, <<byte-units,byte value>>)
Maximum snapshot creation rate per node. Defaults to `40mb` per second.
`readonly`::
(Optional, Boolean)
If `true`, the repository is read-only. The cluster can retrieve and restore
snapshots from the repository but not write to the repository or create
snapshots in it.
+
If `false`, the cluster can write to the repository and create snapshots in it.
Defaults to `false`.
+
[TIP]
=====
If you register the same snapshot repository with multiple clusters, only
one cluster should have write access to the repository. Having multiple clusters
write to the repository at the same time risks corrupting the contents of the
repository.

Only a cluster with write access can create snapshots in the repository. All
other clusters connected to the repository should have the `readonly` parameter
set to `true`. This means those clusters can retrieve or restore snapshots from
the repository but not create snapshots in it.
=====
====

Other accepted `settings` properties depend on the repository type, set using the
<<put-snapshot-repo-api-request-type,`type`>> parameter.

.Valid `settings` properties for `fs` repositories
[%collapsible%open]
====
`location`::
(Required, string)
Location of the shared filesystem used to store and retrieve snapshots. This
location must be registered in the `path.repo` setting on all master and data
nodes in the cluster.
====

.Valid `settings` properties for `source` repositories
[%collapsible%open]
====
`delegate_type`::
(Optional, string)
Delegated repository type. For valid values, see the
<<put-snapshot-repo-api-request-type,`type` parameter>>.
+
`source` repositories can use `settings` properties for its delegated repository
type. See <<snapshots-source-only-repository>>.
====

.Valid `settings` properties for `url` repositories
[%collapsible%open]
====
`url`::
Settings for the repository. Supported settings vary based on the repository
type:
+
---
(Required, string)
URL location of the root of the shared filesystem repository. The following
protocols are supported:
* `file`
* `ftp`
* `http`
* `https`
* `jar`
URLs using the `http`, `https`, or `ftp` protocols must be explicitly allowed
with the <<repositories-url-allowed,`repositories.url.allowed_urls`>> cluster
setting. This setting supports wildcards in the place of a host, path, query, or
fragment in the URL.
URLs using the `file` protocol must point to the location of a shared filesystem
accessible to all master and data nodes in the cluster. This location must be
registered in the `path.repo` setting. You don't need to register URLs using the
`ftp`, `http`, `https`, or `jar` protocols in the `path.repo` setting.
---
--
* <<repository-azure>>
* <<repository-gcs>>
* <<repository-s3>>
* <<snapshots-filesystem-repository>>
* <<snapshots-read-only-repository>>
* <<snapshots-source-only-repository>>

`http_max_retries`::
(Optional, integer) Maximum number of retries for `http` and `https` URLs.
Defaults to `5`.
Other repository types are available through official plugins:

`http_socket_timeout`::
(Optional, <<time-units,time value>>) Maximum wait time for data transfers over
a connection. Defaults to `50s`.
====
* {plugins}/repository-hdfs.html[Hadoop Distributed File System (HDFS) repository]
--

`verify`::
Expand Down
2 changes: 2 additions & 0 deletions docs/reference/snapshot-restore/on-prem-repo-type.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
NOTE: This repository type is only available if you run {es} on your own
hardware. If you use {ess}, see <<ess-repo-types>>.
Loading

0 comments on commit cb6265f

Please sign in to comment.