Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[CCI][GUIDE] Added Guide for Snapshot API(snapshot.md) #486

Closed
wants to merge 81 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
5638fd1
Added Guide for Snapshot API(snapshot.md)
roma2023 Sep 5, 2023
1603ff9
Updated CHANGELOG.md and applied working sample on snapshot.md
roma2023 Sep 5, 2023
589bc1d
fixed integration test fail
roma2023 Sep 5, 2023
14cca93
added snapshot sample and updated snapshot.md
roma2023 Sep 7, 2023
d63c358
Added temporary directory to snapshot sample
roma2023 Sep 13, 2023
199aa5d
add Dockerfile, change temp_repo_location
Eager-coder Dec 14, 2023
31aa5f8
update USER_GUIDE.md
roma2023 Dec 27, 2023
4ad0c5f
[CCI][GUIDE] Minor fixes to poetry docs (#494)
roma2023 Sep 15, 2023
ab2eaf0
Fix: typo. (#497)
dblock Sep 15, 2023
a4036a7
Updated APIs to match other clients and opensearch openapi spec (#502)
saimedhi Sep 26, 2023
55c1790
Integrated generated APIs into the existing module, ensuring alignme…
saimedhi Sep 26, 2023
a29be4a
Integrated generated ingest client APIs into the existing module, ens…
saimedhi Sep 29, 2023
335111b
Integrated generated dangling_indices client APIs into the existing m…
saimedhi Oct 3, 2023
1241968
Bumps from >=1.21.1 to >=1.26.9 (#518)
saimedhi Oct 3, 2023
aa4b6d6
Modified generator to generate 'options' and 'default value' for para…
saimedhi Oct 3, 2023
626659b
fixes: #512 Wrong return type hint in async_scan (#520)
bhavaniravi Oct 3, 2023
c261f88
Integrated generated 'nodes' client APIs into the existing module, en…
saimedhi Oct 3, 2023
40f62fb
updated changelog (#522)
saimedhi Oct 3, 2023
c13dd31
Bump version to 2.3.2 (#524)
saimedhi Oct 6, 2023
36df6e8
Fix: typos. (#526)
dblock Oct 9, 2023
2f42f68
Modified generator to generate api deprecation warnings (#527)
saimedhi Oct 9, 2023
5e5a4dd
Generate cat client from API specs (#529)
saimedhi Oct 9, 2023
fce2249
Generate cluster client from API specs (#530)
saimedhi Oct 9, 2023
781c0ff
Added new guide & sample module for using index templates. (#531)
Djcarrillo6 Oct 11, 2023
ef898dc
Removed EOL Python3.5 & bumped urllib3 version to patch security vuln…
Djcarrillo6 Oct 12, 2023
ea219a3
Align pool_maxsize for different connection pool implementations. (#535)
dblock Oct 12, 2023
af0ae87
Add micro benchmarks. (#537)
dblock Oct 13, 2023
9d7cd43
Added advanced index actioins guide & sample code file. (#541)
Djcarrillo6 Oct 16, 2023
48b22d3
Added a guide on making raw JSON REST requests. (#542)
dblock Oct 16, 2023
93b4698
Added support for AWS Sigv4 for UrlLib3. (#547)
dblock Oct 23, 2023
2b164de
Remove support for Python 2.x. (#548)
dblock Oct 24, 2023
b0f1136
Remove PY2. (#550)
dblock Oct 25, 2023
2ddac11
Generate opensearch-py client from API specs (#551)
saimedhi Oct 26, 2023
d6e0b5f
added remote store client (#552)
saimedhi Oct 26, 2023
c14f8ac
Override put_mapping definition to make index name optional. (#553)
dblock Oct 26, 2023
5cea1b5
Add a nox session for client generator. (#554)
dblock Oct 26, 2023
ed32d8e
Fix: don't hardcode client class name. (#555)
dblock Oct 26, 2023
4f51d9c
Added a utf-8 header to all .py files (#557)
dblock Oct 26, 2023
4357685
Added samples, benchmarks and docs for nox format. (#556)
dblock Oct 27, 2023
9dbf689
Applied document lifecycle guide & sample. (#559)
Djcarrillo6 Oct 31, 2023
7b36528
Merge `.pyi` type stubs inline (#563)
dblock Nov 6, 2023
3de1a8f
Expanded type coverage to benchmarks, samples and tests. (#566)
dblock Nov 9, 2023
09c7394
Set enable_cleanup_closed=True to drop TLS connections without a shut…
snemes Nov 9, 2023
e60a6ca
Added Windows CI. (#569)
dblock Nov 10, 2023
2dac852
Expanded `nox -rs docs` to generate docs. (#568)
dblock Nov 10, 2023
663e20d
Fix asyncio warnings/markers. (#574)
dblock Nov 10, 2023
39da24e
Fix: assert deprecation. (#572)
dblock Nov 10, 2023
12f9db5
Fix: build and deploy docs. (#575)
dblock Nov 11, 2023
e5f5e97
Removed EOL Python 3.6. Bumped urllib3 to 1.26.18 to patch sec vulner…
Djcarrillo6 Nov 12, 2023
438593c
Remove support for local client in testing. (#573)
dblock Nov 12, 2023
b1198d7
Fixed typo in CHANGELOG from merged PR #576 (#577)
Djcarrillo6 Nov 12, 2023
5ad65ce
Avoid decoding request body unless it needs to be logged. (#571)
dblock Nov 12, 2023
7af7b5f
Update coverage requirement from <7.0.0 to <8.0.0 (#578)
dependabot[bot] Nov 13, 2023
6aaa52c
Added client-level REST helpers. (#544)
dblock Nov 13, 2023
ebe0b4b
Preparing for release, 2.4.0 (#581)
dblock Nov 14, 2023
75f0e5f
Remove 2.11. (#583)
dblock Nov 15, 2023
866a8b7
Add 2.11.0 to the test matrix. (#587)
dblock Nov 15, 2023
0531c8d
Fix: version number in filename for distribution. (#588)
dblock Nov 15, 2023
6fc07df
Fix for 592 (#594)
dblock Nov 16, 2023
ad036ac
Preparing for next development iteration, 2.4.2. (#597)
dblock Nov 16, 2023
a1ed1ab
Fix: TypeError on calling parallel_bulk. (#601)
dblock Nov 17, 2023
7b4d93b
Fix Amazon OpenSearch Serverless integration with LangChain. (#603)
dblock Nov 17, 2023
f956d91
Fixes a suspicious setattr that should be taking an Any (#604)
dblock Nov 17, 2023
8e94496
Preparing for next developer iteration, 2.4.3. (#606)
dblock Nov 19, 2023
7fc5a01
Added pylint, enforce naming. (#590)
dblock Nov 21, 2023
09d4dba
Enabled pylint:pointless-statement. (#611)
dblock Nov 21, 2023
46025a2
Added a guide & sample for a custom logger client implementation. (#579)
Djcarrillo6 Nov 22, 2023
51909bb
Update the GHA release workflow with trusted publisher enabled (#614)
zelinh Nov 23, 2023
f7a018f
remove unnecessary utf-8 header in .py files (#615)
samuelorji Nov 24, 2023
56c40d2
Remove unnecessary utf-8 header in license_headers.py (follow-up to #…
jayaddison Nov 28, 2023
546da5f
Add OpenSearch 2.11.1 integration tests. (#584)
dblock Dec 1, 2023
1aeeef5
Added fix for key error because of missing 'hits' key. (#616)
Djcarrillo6 Dec 4, 2023
ac16a10
Update pytest-asyncio requirement from <=0.21.1 to <=0.23.2 (#625)
dependabot[bot] Dec 11, 2023
3ce7584
Bumps urllib3 from >=1.26.18 to >=1.26.18, <2 (#632)
saimedhi Dec 15, 2023
e7f9831
samples directory now passes the missing-function-docstring linter (#…
macohen Dec 22, 2023
674f14e
format snapshot_sample.py signed-off-by: roma2023 <romasaparhan19@gma…
roma2023 Dec 28, 2023
1448208
format snapshot_sample.py signed-off-by: roma2023 <romasaparhan19@gma…
roma2023 Dec 28, 2023
46ee9c1
format snapshot_sample.py signed-off-by: roma2023 <romasaparhan19@gma…
roma2023 Dec 28, 2023
9656c27
lint snapshot_sample.py
roma2023 Dec 28, 2023
b22a37b
added license header
roma2023 Dec 28, 2023
dc0cf68
Merge branch 'main' into snapshot-md-guide
roma2023 Dec 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,7 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Improved CI performance of integration with unreleased OpenSearch ([#318](https://github.com/opensearch-project/opensearch-py/pull/318))
- Added k-NN guide and samples ([#449](https://github.com/opensearch-project/opensearch-py/pull/449))
- Added the ability to run tests matching a pattern to `.ci/run-tests` ([#454](https://github.com/opensearch-project/opensearch-py/pull/454))
- Added a guide for taking snapshots ([#486](https://github.com/opensearch-project/opensearch-py/pull/429))
### Changed
- Moved security from `plugins` to `clients` ([#442](https://github.com/opensearch-project/opensearch-py/pull/442))
- Updated Security Client APIs ([#450](https://github.com/opensearch-project/opensearch-py/pull/450))
Expand Down
1 change: 1 addition & 0 deletions USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,6 +153,7 @@ print(response)
- [Search](guides/search.md)
- [Point in Time](guides/point_in_time.md)
- [Using a Proxy](guides/proxy.md)
- [Working with Snapshots](guides/snapshot.md)
- [Index Templates](guides/index_template.md)
- [Advanced Index Actions](guides/advanced_index_actions.md)
- [Making Raw JSON REST Requests](guides/json.md)
Expand Down
144 changes: 144 additions & 0 deletions guides/snapshot.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# Table of Contents
- [Snapshot Actions](#snapshot-actions)
- [Setup](#setup)
- [API Actions](#api-actions)
- [Create Snapshot Repository](#create-snapshot-repository)
- [Create Snapshot](#create-snapshot)
- [Verify Snapshot Repository](#verify-snapshot-repository)
- [Delete Snapshot](#delete-snapshot)
- [Restore Snapshot](#restore-snapshot)
- [Get Snapshot Status](#get-snapshot-status)
- [Clone Snapshot](#clone-snapshot)
- [Get Snapshot](#get-snapshot)
- [Get Repository](#get-repository)
- [Repository Analyze](#repository-analyze)
- [Cleanup](#cleanup)

# Snapshot Actions
dblock marked this conversation as resolved.
Show resolved Hide resolved
In this guide, we will look at some snapshot actions that allow you to manage and work with snapshots of your indices.


## Setup
Let's create a client instance, and an index named `movies`:
```python
from opensearchpy import OpenSearch

host = 'localhost'
port = 9200
auth = ('admin', 'admin') # For testing only. Don't store credentials in code.

client = OpenSearch(
hosts = [{'host': host, 'port': port}],
http_auth = auth,
use_ssl = True,
verify_certs = False,
ssl_show_warn = False
)

print(client.info()) # Check server info and make sure the client is connected
client.indices.create(index='movies')
```
## API Actions
### Create Snapshot Repository
Before taking a snapshot, you need to create a snapshot repository to store the snapshots. You can use the `create_repository` API action for this purpose. The following example creates a snapshot repository named `my_repository`:

```python
repo_body = {
"type": "fs", # Replace 'fs' with the appropriate repository type
"settings": {
"location": "/path/to/repo",
}
}

# Create the snapshot repository and capture the response
response = client.snapshot.create_repository(repository='my_repository', body=repo_body)

# Print the response to see the result
print(response)
```

### Create Snapshot
To create a snapshot of an index, you can use the `create` method from the `snapshot` API. The following example creates a snapshot named `my_snapshot` for the movies index:

```python
client.snapshot.create(repository='my_repository', snapshot='my_snapshot', body={"indices": "movies"})
```

### Verify Snapshot Repository
The `verify_repository` API action allows you to verify a snapshot repository. Verifying a repository ensures that it is accessible and operational, but it does not validate the integrity of the snapshots stored within the repository. The following example verifies `my_repository`:

```python
response = client.snapshot.verify_repository(repository='my_repository')
dblock marked this conversation as resolved.
Show resolved Hide resolved

# Print the HTTP status code
print("HTTP Status Code:", response.status_code)

# Print the response content
print("Response Content:", response.content)
```

### Delete Snapshot
To delete a specific snapshot, use the `delete` API action:

```python
client.snapshot.delete(repository='my_repository', snapshot='my_snapshot')
```
### Restore Snapshot
To restore a snapshot and recreate the indices, mappings, and data, you can use the `restore` method. The following example restores the `my_snapshot` snapshot:

```python
response = client.snapshot.restore(repository='my_repository', snapshot='my_snapshot')
```

### Get Snapshot Status
To check the status of a snapshot, you can use the `status` method.

```python
response = client.snapshot.status(repository='my_repository', snapshot='my_snapshot')
```

### Clone Snapshot
You can clone an existing snapshot to create a new snapshot with the same contents. The `clone` operation allows you to create multiple copies of a snapshot, which can be useful for backup retention or creating snapshots for different purposes. The following example clones a snapshot named `my_snapshot` to create a new snapshot named `my_snapshot_clone`:

```python
client.snapshot.clone(
repository='my_repository',
snapshot='my_snapshot',
target_snapshot='my_snapshot_clone'
)
```
## Get Snapshot
To retrieve information about a specific snapshot, you can use the `get` API action. It provides metadata such as the snapshot's status, indices included in the snapshot, and the timestamp when the snapshot was taken. The following example retrieves information about the `my_snapshot`:

```python
response = client.snapshot.get(
repository='my_repository',
snapshot='my_snapshot'
)

# Print the response to see the result
print(response)
```

## Get Repository
To retrieve information about a snapshot repository, you can use the `get_repository` API action. It provides details about the configured repository, including its type and settings. The following example retrieves information about the `my_repository`:

```python
response = client.snapshot.get_repository(repository='my_repository')
```

## Repository Analyze
The `repository_analyze` API action allows you to analyze a snapshot repository for correctness and performance. It checks for any inconsistencies or corruption in the repository. The following example performs a repository analysis on `my_repository`:

```python
response = client.snapshot.repository_analyze(repository='my_repository')
```

## Cleanup

Finally, let's delete the `movies` index and clean up all the snapshots and the repository:
```python
client.indices.delete(index='movies')
client.snapshot.delete(repository='my_repository', snapshot='my_snapshot')
client.snapshot.delete_repository(repository='my_repository')
```
9 changes: 9 additions & 0 deletions samples/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
FROM opensearchproject/opensearch:2.11.0

ARG OPENSEARCH_HOME=/usr/share/opensearch
ARG UID=1000
ARG GID=1000

RUN echo 'path.repo: ["/usr/share/opensearch/backups"]' >> $OPENSEARCH_HOME/config/opensearch.yml
RUN mkdir -p $OPENSEARCH_HOME/backups
RUN chown -Rv $UID:$GID $OPENSEARCH_HOME/backups
74 changes: 74 additions & 0 deletions samples/snapshot/snapshot_sample.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
#!/usr/bin/env python

# SPDX-License-Identifier: Apache-2.0
#
# The OpenSearch Contributors require contributions made to
# this file be licensed under the Apache-2.0 license or a
# compatible open source license.
#
# Modifications Copyright OpenSearch Contributors. See
# GitHub history for details.

import tempfile

from opensearchpy import OpenSearch

# connect to OpenSearch

HOST = "localhost"
PORT = 9200
auth = ("admin", "admin") # For testing only. Don't store credentials in code.

client = OpenSearch(
hosts=[{"host": HOST, "port": PORT}],
http_auth=auth,
use_ssl=True,
verify_certs=False,
ssl_show_warn=False,
)

# Create an index

INDEX_NAME = "test-snapshot"
client.indices.create(index=INDEX_NAME)

# Create a temporary directory for the snapshot repository
temp_repo = tempfile.TemporaryDirectory()
TEMP_REPO_LOCATION = "/usr/share/opensearch/backups"

# Define the repository body with the temporary location
repo_body = {
"type": "fs", # Replace 'fs' with the appropriate repository type
"settings": {
"location": TEMP_REPO_LOCATION, # Replace with the desired repository location
},
}

REPOSITORY_NAME = "my_repository"
response = client.snapshot.create_repository(repository=REPOSITORY_NAME, body=repo_body)

print(response)

# Create a snapshot

SNAPSHOT_NAME = "my_snapshot"
response = client.snapshot.create(
repository=REPOSITORY_NAME, snapshot=SNAPSHOT_NAME, body={"indices": INDEX_NAME}
)

print(response)

# Get Snapshot Information

snapshot_info = client.snapshot.get(repository=REPOSITORY_NAME, snapshot=SNAPSHOT_NAME)

print(snapshot_info)

# Clean up - Delete Snapshot and Repository

client.snapshot.delete(repository=REPOSITORY_NAME, snapshot=SNAPSHOT_NAME)
client.snapshot.delete_repository(repository=REPOSITORY_NAME)

# Clean up - Delete Index

client.indices.delete(index=INDEX_NAME)
Loading