Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Blog Post: OpenSearch plugin zips now in Maven repo #930

Merged
merged 14 commits into from
Aug 25, 2022
Merged
Show file tree
Hide file tree
Changes from 10 commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions Gemfile
Original file line number Diff line number Diff line change
Expand Up @@ -33,3 +33,5 @@ end
gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin]

gem "rouge"

gem "webrick", "~> 1.7"
4 changes: 3 additions & 1 deletion Gemfile.lock
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,7 @@ GEM
terminal-table (2.0.0)
unicode-display_width (~> 1.1, >= 1.1.1)
unicode-display_width (1.7.0)
webrick (1.7.0)

PLATFORMS
ruby
Expand All @@ -91,6 +92,7 @@ DEPENDENCIES
tzinfo (~> 1.2)
tzinfo-data
wdm (~> 0.1.1)
webrick (~> 1.7)

BUNDLED WITH
2.3.5
2.3.18
8 changes: 8 additions & 0 deletions _authors/prudhvigodithi.markdown
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
---
name: Prudhvi Godithi
short_name: prudhvigodithi
linkedin: 'prudhvi-godithi'
photo: '/assets/media/authors/prudhvigodithi.jpg'
---

**Prudhvi Godithi** is an AWS System Engineer working on the OpenSearch project.
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
---
layout: post
title: "OpenSearch plugin zips now in Maven repo"
authors:
- prudhvigodithi
date: 2022-07-24 01:01:01 -0700
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@prudhvigodithi since this has been in edit for a while - please update this date

categories:
- technical-post
twittercard:
description: "Details on how to consume OpenSearch plugin zips from a Maven repo and the process involved in shipping them to a Maven repo."
---

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I deleted the Summary section and used that description as the Intro.

Suggested change
Starting with the release of OpenSearch `2.1.0`, OpenSearch plugin zips are now signed and published to a central Apache Maven [repo](https://repo1.maven.org/maven2/org/opensearch/plugin/). Using the [Release zips Maven Repo](https://repo1.maven.org/maven2/org/opensearch/plugin/) and [Snapshot zips Maven Repo](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/) URL’s, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sentence Reformation:

Starting with the release of OpenSearch 2.1.0, OpenSearch plugin zips are now signed and published to a central Apache Maven repo. Using the Release zips and Snapshot zips Maven Repo URL’s, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In "Maven Repo URL's", change to "Maven repo URLs".

Starting with the release of OpenSearch `2.1.0`, OpenSearch plugin zips are now signed and published to a central Apache Maven [repo](https://repo1.maven.org/maven2/org/opensearch/plugin/). Using the [Release zips](https://repo1.maven.org/maven2/org/opensearch/plugin/) and [Snapshot zips](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/) Maven Repo URLs, OpenSearch plugin zips can now be consumed as a dependency to build other plugins or fetched as standalone components for your OpenSearch cluster.

## Motivation

Before OpenSearch 2.1, plugin zips used as dependencies could not be downloaded dynamically during runtime because plugin zips were not a part of the version-controlled Maven system. The only mechanism for plugin downloads was each plugins' respective Java jars through Maven coordinates. This system forced users who wanted more control over their OpenSearch plugin configuration to use a dependency plugin zip built on a developer desktop instead of a more reliable version-controlled plugin.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change "plugins'" to "plugin's".


Furthermore, to facilitate the plugin availability as a dependency, tests executed against as part the OpenSearch build process from zip were not accurate, as each local build zip had to find the plugin repo in order to ensure that plugin's availability. These restrictions were challenging to our community since using plugin zips as separate isolated components via a cached mechanism proved to be impossible.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Currently reads "...tests executed against as part the...". Please correct. Change "since" to "because".


## Benefits of Maven

With Maven, plugin zips can now be retrieved by:

- Downloading each plugin directly using their respective Maven coordinates .
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the space after the period.

- Using clickstream from the central Maven repo, which can be cached later to a local Maven repo.
- Fetching the development `SNAPSHOT` version with same Maven `groupID` as [org.opensearch.plugin](https://aws.oss.sonatype.org/content/repositories/snapshots/org/opensearch/plugin/).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add "the" before "same".


Using OpenSearch plugin zips through Maven offers the following benefits:

- Plugins zip in the central Maven repo are already signed with `.asc`, `.md5`, `.sha1`, `.sha256`, and `.sha512` extensions.
- Users are no longer required to to check in zips to any `src/` files since zips can be fetched with the right `groupID`, `artifactID`, and `version`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change "since" to "because".

- Tests and CI workflows can directly run against zips from the Maven repo instead of requiring a manual download.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define "CI" as "continuous integration (CI)".


## Maven zip publication with Gradle

OpenSearch publishes plugin zips using a custom Gradle plugin, `opensearch.pluginzip`. With OpenSearch 2.1, all OpenSearch Gradle-supported plugins create a new task, `publishPluginZipPublicationToZipStagingRepository`. The task performs all the heavy lifting for users by:

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
You can fetch and use OpenSearch plugin zips from Maven in three ways:

- Identifying the distribution plugin zip.
- Setting the Maven coordinates.
- Generating the POM file.
- Updating with the user-generated POM fields.
- Publishing the zip to your Maven repo.

You can find more details about the inner workings of OpenSearch plugins in the [opensearch-plugins repo](https://github.com/opensearch-project/opensearch-plugins/blob/main/BUILDING.md#opensearchpluginzip).

**Figure 1**: Workflow that ships generated plugin zips to Maven

![Figure 1: Workflow that ship zips to maven]({{ site.baseurl }}/assets/media/blog-images/2022-07-24-opensearch-plugin-zips-now-in-maven-repo/figure1.png){: .img-fluid }**Figure 1**: Workflow that ships generated plugin zips to Maven

## Consume plugin in zips

You can fetch plugin in zips in three different ways:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should "plugin" be "plugins" in both instances, or should there be an article in front of "plugin" in both instances? Change the colon to a period.


**Using the Maven CLI**

Consume from central Maven repo:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add "the" before "central".


```
mvn dependency:get -DgroupId=org.opensearch.plugin -DartifactId=opensearch-job-scheduler -Dversion=2.1.0.0 -Dpackaging=zip -DremoteRepositories=central::default::https://repo.maven.apache.org/maven2,myrepo::::http://myrepo.com/maven2
prudhvigodithi marked this conversation as resolved.
Show resolved Hide resolved
```

Consume from the Snapshot Maven repo:

```
Snapshot Maven repo
mvn dependency:get -DgroupId=org.opensearch.plugin -DartifactId=opensearch-job-scheduler -Dversion=2.1.0.0-SNAPSHOT -Dpackaging=zip -DremoteRepositories=https://aws.oss.sonatype.org/content/repositories/snapshots/
```

**Gradle Project: Using the build.gradle file**

```
dependencies {
classpath "org.opensearch.plugin:opensearch-job-scheduler:2.1.0.0@zip"
classpath "org.opensearch.plugin:opensearch-knn:2.1.0.0@zip"
}
```

**Maven Project: Using the pom.xml file**

```
<dependencies>
<dependency>
<groupId>org.opensearch.plugin</groupId>
<artifactId>opensearch-job-scheduler</artifactId>
<version>2.1.0.0</version>
<packaging>zip</packaging>
</dependency>
</dependencies>
```





Binary file added assets/media/authors/prudhvigodithi.jpg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.