From 723e9f3c42287596394bc302c1fd0253f8ce317f Mon Sep 17 00:00:00 2001 From: Joshua Palis Date: Thu, 16 Jun 2022 18:41:09 +0000 Subject: [PATCH 1/6] updated developer guide Signed-off-by: Joshua Palis --- DEVELOPER_GUIDE.md | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index f069b70c..a3bfbffc 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -2,8 +2,10 @@ # OpenSearch SDK Developer Guide - [Introduction](#introduction) - [Getting Started](#getting-started) - - [Git Clone IndependentPlugin Repo](#git-clone-independentplugin-repo) - - [Run Independent Plugin](#run-independent-plugin) + - [Git Clone OpenSearch-SDK Repo](#git-clone-OpenSearch-SDK-repo) + - [Git Clone OpenSearch Repo](#git-clone-opensearch-repo) + - [Publish OpenSearch Feature/Extensions branch to Maven local](#publish-opensearch-feature/extensions-branch-to-maven-local) + - [Run Extension](#run-extension) - [Run Tests](#run-tests) - [Send Message using Telnet](#send-message-using-telnet) @@ -14,12 +16,18 @@ Read more about extensibility [here](https://github.com/opensearch-project/OpenS ## Getting Started -### Git Clone IndependentPlugin Repo -Fork [IndependentPlugin](https://github.com/owaiskazi19/IndependentPlugin) and clone locally, e.g. `git clone https://github.com/[your username]/IndependentPlugin.git`. +### Git Clone OpenSearch SDK Repo +Fork [OpenSearch SDK](https://github.com/opensearch-project/opensearch-sdk) and clone locally, e.g. `git clone https://github.com/[your username]/opensearch-sdk.git`. -### Run Independent Plugin +### Git Clone OpenSearch Repo +Fork [OpenSearch](https://github.com/opensearch-project/OpenSearch) and clone locally, e.g. `git clone https://github.com/[your username]/OpenSearch.git`. -Run main script using `gradlew run`. +## Publish OpenSearch feature/extensions Branch to Maven local +The work done to support the extensions framework is located on the `Feature/Extensions` branch of the OpenSearch project. It is necessary to publish the dependencies of this branch to your local maven repository prior to running the extension on a seperate process. In order to do this, first navigate to the directory that OpenSearch has been cloned to, checkout the correct branch, e.g. `git checkout feature/extensions`. Then run `./gradlew publishToMavenLocal`. It is necessary to publish dependencies to a local maven repository until this branch is merged to `main`, at which point all dependencies will be published to Maven central. + +### Run Extension + +Navigate to the directory that OpenSearch-SDK has been cloned to and run main script using `gradlew run`. ``` ./gradlew run @@ -46,7 +54,7 @@ Run tests : ### Send Message using Telnet -To send a message, first run the IndependentPlugin : +To send a message, first run the OpenSearch-SDK : ``` ./gradlew run @@ -61,7 +69,7 @@ Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. ``` -The original terminal used to run the independent plugin will log the connection request : +The original terminal used to run the OpenSearch-SDK will log the connection request : ``` [opensearch[NettySizeHeaderFrameDecoderTests][transport_worker][T#5]] TRACE transportservice.TcpTransport - Tcp transport channel accepted: Netty4TcpChannel{localAddress=/127.0.0.1:5555, remoteAddress=/127.0.0.1:57302} [opensearch[NettySizeHeaderFrameDecoderTests][transport_worker][T#5]] TRACE transportservice.netty4.OpenSearchLoggingHandler - [id: 0x8c1cc239, L:/127.0.0.1:5555 - R:/127.0.0.1:57302] REGISTERED From 32d34044faf36440fc1cf223418a26e0d43b3f5e Mon Sep 17 00:00:00 2001 From: Joshua Palis Date: Thu, 16 Jun 2022 18:48:56 +0000 Subject: [PATCH 2/6] added more larification on how to run both the SDK and OpenSearch together Signed-off-by: Joshua Palis --- DEVELOPER_GUIDE.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index a3bfbffc..f83bf85a 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -45,6 +45,9 @@ Bound addresses will then be logged to the terminal : [main] INFO transportservice.TransportService - profile [test]: publish_address {127.0.0.1:5555}, bound_addresses {[::1]:5555}, {127.0.0.1:5555} ``` +Once the bound addresses are logged to the terminal, navigate to the directory that OpenSearch has been cloned to and start OpenSearch feature/extensions branch using `./gradlew run`. +It is important that the extension is already up and running prior to starting OpenSearch, since extension discovery occurs only if the extension is already listening on a pre-defined port. Once discovery is complete and the data transfer connection between both nodes has been established, OpenSearch and the extension will now be able to comminicate. + ### Run Tests Run tests : From c93ba19e402beb1eec93071f610058a77aa2e5b9 Mon Sep 17 00:00:00 2001 From: Joshua Palis Date: Thu, 16 Jun 2022 19:01:08 +0000 Subject: [PATCH 3/6] resolved PR comments Signed-off-by: Joshua Palis --- DEVELOPER_GUIDE.md | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index f83bf85a..447fd6c7 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -23,9 +23,15 @@ Fork [OpenSearch SDK](https://github.com/opensearch-project/opensearch-sdk) and Fork [OpenSearch](https://github.com/opensearch-project/OpenSearch) and clone locally, e.g. `git clone https://github.com/[your username]/OpenSearch.git`. ## Publish OpenSearch feature/extensions Branch to Maven local -The work done to support the extensions framework is located on the `Feature/Extensions` branch of the OpenSearch project. It is necessary to publish the dependencies of this branch to your local maven repository prior to running the extension on a seperate process. In order to do this, first navigate to the directory that OpenSearch has been cloned to, checkout the correct branch, e.g. `git checkout feature/extensions`. Then run `./gradlew publishToMavenLocal`. It is necessary to publish dependencies to a local maven repository until this branch is merged to `main`, at which point all dependencies will be published to Maven central. +The work done to support the extensions framework is located on the `feature/extensions` branch of the OpenSearch project. It is necessary to publish the dependencies of this branch to your local maven repository prior to running the extension on a seperate process. -### Run Extension +- First navigate to the directory that OpenSearch has been cloned to +- Checkout the correct branch, e.g. `git checkout feature/extensions`. +- Run `./gradlew publishToMavenLocal`. + +It is necessary to publish dependencies to a local maven repository until this branch is merged to `main`, at which point all dependencies will be published to Maven central. + +### Run OpenSearch SDK Navigate to the directory that OpenSearch-SDK has been cloned to and run main script using `gradlew run`. From 3e8d9d62ecae974b3b614416aa829bf1ee46ac23 Mon Sep 17 00:00:00 2001 From: Joshua Palis Date: Thu, 16 Jun 2022 19:06:14 +0000 Subject: [PATCH 4/6] renaming extensions to OpenSearch SDK Signed-off-by: Joshua Palis --- DEVELOPER_GUIDE.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index 447fd6c7..f0fa5b5f 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -5,7 +5,7 @@ - [Git Clone OpenSearch-SDK Repo](#git-clone-OpenSearch-SDK-repo) - [Git Clone OpenSearch Repo](#git-clone-opensearch-repo) - [Publish OpenSearch Feature/Extensions branch to Maven local](#publish-opensearch-feature/extensions-branch-to-maven-local) - - [Run Extension](#run-extension) + - [Run OpenSearch-SDK](#run-opensearch-sdk) - [Run Tests](#run-tests) - [Send Message using Telnet](#send-message-using-telnet) @@ -23,7 +23,7 @@ Fork [OpenSearch SDK](https://github.com/opensearch-project/opensearch-sdk) and Fork [OpenSearch](https://github.com/opensearch-project/OpenSearch) and clone locally, e.g. `git clone https://github.com/[your username]/OpenSearch.git`. ## Publish OpenSearch feature/extensions Branch to Maven local -The work done to support the extensions framework is located on the `feature/extensions` branch of the OpenSearch project. It is necessary to publish the dependencies of this branch to your local maven repository prior to running the extension on a seperate process. +The work done to support the extensions framework is located on the `feature/extensions` branch of the OpenSearch project. It is necessary to publish the dependencies of this branch to your local maven repository prior to running the OpenSearch SDK on a seperate process. - First navigate to the directory that OpenSearch has been cloned to - Checkout the correct branch, e.g. `git checkout feature/extensions`. @@ -33,7 +33,7 @@ It is necessary to publish dependencies to a local maven repository until this b ### Run OpenSearch SDK -Navigate to the directory that OpenSearch-SDK has been cloned to and run main script using `gradlew run`. +Navigate to the directory that OpenSearch-SDK has been cloned to and run main script using `./gradlew run`. ``` ./gradlew run @@ -52,7 +52,7 @@ Bound addresses will then be logged to the terminal : ``` Once the bound addresses are logged to the terminal, navigate to the directory that OpenSearch has been cloned to and start OpenSearch feature/extensions branch using `./gradlew run`. -It is important that the extension is already up and running prior to starting OpenSearch, since extension discovery occurs only if the extension is already listening on a pre-defined port. Once discovery is complete and the data transfer connection between both nodes has been established, OpenSearch and the extension will now be able to comminicate. +It is important that the OpenSearch SDK is already up and running prior to starting OpenSearch, since extension discovery occurs only if the OpenSearch SDK is already listening on a pre-defined port. Once discovery is complete and the data transfer connection between both nodes has been established, OpenSearch and the OpenSearch SDK will now be able to comminicate. ### Run Tests From e38d470cffb3adfd93030230a334e7bd0cada47d Mon Sep 17 00:00:00 2001 From: Joshua Palis Date: Thu, 16 Jun 2022 19:32:58 +0000 Subject: [PATCH 5/6] added additional step to run gradle check Signed-off-by: Joshua Palis --- DEVELOPER_GUIDE.md | 1 + 1 file changed, 1 insertion(+) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index f0fa5b5f..c736ef6c 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -28,6 +28,7 @@ The work done to support the extensions framework is located on the `feature/ext - First navigate to the directory that OpenSearch has been cloned to - Checkout the correct branch, e.g. `git checkout feature/extensions`. - Run `./gradlew publishToMavenLocal`. +- Run `./gradlew check` to make sure the build is successful. It is necessary to publish dependencies to a local maven repository until this branch is merged to `main`, at which point all dependencies will be published to Maven central. From b95ef4c6f54f4d7de76cb380a6539047263c3f2b Mon Sep 17 00:00:00 2001 From: Joshua Palis Date: Thu, 16 Jun 2022 21:40:41 +0000 Subject: [PATCH 6/6] added additional comments o running opensearch on a seperate terminal/process. Added example logs of TCP handshake and plugin name request responses Signed-off-by: Joshua Palis --- DEVELOPER_GUIDE.md | 97 +++++++++++++++++++++++++++++----------------- 1 file changed, 62 insertions(+), 35 deletions(-) diff --git a/DEVELOPER_GUIDE.md b/DEVELOPER_GUIDE.md index c736ef6c..7e5f036c 100644 --- a/DEVELOPER_GUIDE.md +++ b/DEVELOPER_GUIDE.md @@ -6,8 +6,8 @@ - [Git Clone OpenSearch Repo](#git-clone-opensearch-repo) - [Publish OpenSearch Feature/Extensions branch to Maven local](#publish-opensearch-feature/extensions-branch-to-maven-local) - [Run OpenSearch-SDK](#run-opensearch-sdk) + - [Run OpenSearch](#run-opensearch) - [Run Tests](#run-tests) - - [Send Message using Telnet](#send-message-using-telnet) ## Introduction Opensearch plugins have allowed the extension and ehancements of various core features however, current plugin architecture carries the risk of fatally impacting clusters should they fail. In order to ensure that plugins may run safely without impacting the system, our goal is to effectively isolate plugin interactions with OpenSearch by modularizing the [extension points](https://opensearch.org/blog/technical-post/2021/12/plugins-intro/) to which they hook onto. @@ -52,53 +52,80 @@ Bound addresses will then be logged to the terminal : [main] INFO transportservice.TransportService - profile [test]: publish_address {127.0.0.1:5555}, bound_addresses {[::1]:5555}, {127.0.0.1:5555} ``` -Once the bound addresses are logged to the terminal, navigate to the directory that OpenSearch has been cloned to and start OpenSearch feature/extensions branch using `./gradlew run`. -It is important that the OpenSearch SDK is already up and running prior to starting OpenSearch, since extension discovery occurs only if the OpenSearch SDK is already listening on a pre-defined port. Once discovery is complete and the data transfer connection between both nodes has been established, OpenSearch and the OpenSearch SDK will now be able to comminicate. +### Run OpenSearch -### Run Tests +- Start a seperate terminal and navigate to the directory that OpenSearch has been cloned to using `cd OpenSearch` +- Start OpenSearch feature/extensions branch using `./gradlew run`. + +During OpenSearch bootstrap, `ExtensionsOrchestrator` will then discover the extension listenening on a pre-defined port and execute the TCP handshake protocol to establish a data transfer connection. A request will be sent to the OpenSearch SDK and upon acknowledgment, the extension will respond with its name which will be logged onto terminal that OpenSearch is running on. -Run tests : ``` -./gradlew clean build test +[2022-06-16T21:30:18,857][INFO ][o.o.t.TransportService ] [runTask-0] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} +[2022-06-16T21:30:18,978][INFO ][o.o.t.TransportService ] [runTask-0] Action: internal:transport/handshake +[2022-06-16T21:30:18,989][INFO ][o.o.t.TransportService ] [runTask-0] TransportService:sendRequest action=internal:discovery/extensions +[2022-06-16T21:30:18,989][INFO ][o.o.t.TransportService ] [runTask-0] Action: internal:discovery/extensions +[2022-06-16T21:30:19,000][INFO ][o.o.e.ExtensionsOrchestrator] [runTask-0] received PluginResponse{examplepluginname} ``` -### Send Message using Telnet +OpenSearch SDK terminal will also log all requests and responses it recieves from OpenSearch : -To send a message, first run the OpenSearch-SDK : - -``` -./gradlew run -``` -In another terminal, run : +TCP HandShake Request : ``` -telnet localhost 5555 -``` -Once Telnet Client is connected, the terminal will print out : -``` -Trying 127.0.0.1... -Connected to localhost. -Escape character is '^]'. -``` -The original terminal used to run the OpenSearch-SDK will log the connection request : -``` -[opensearch[NettySizeHeaderFrameDecoderTests][transport_worker][T#5]] TRACE transportservice.TcpTransport - Tcp transport channel accepted: Netty4TcpChannel{localAddress=/127.0.0.1:5555, remoteAddress=/127.0.0.1:57302} -[opensearch[NettySizeHeaderFrameDecoderTests][transport_worker][T#5]] TRACE transportservice.netty4.OpenSearchLoggingHandler - [id: 0x8c1cc239, L:/127.0.0.1:5555 - R:/127.0.0.1:57302] REGISTERED -[opensearch[NettySizeHeaderFrameDecoderTests][transport_worker][T#5]] TRACE transportservice.netty4.OpenSearchLoggingHandler - [id: 0x8c1cc239, L:/127.0.0.1:5555 - R:/127.0.0.1:57302] ACTIVE -``` -Messages sent through the Telnet Client must begin with an 'ES', for example : -``` -ES1234SHDDF +21:30:18.943 [opensearch[extension][transport_worker][T#7]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0x37b22600, L:/127.0.0.1:4532 - R:/127.0.0.1:47766] READ: 55B + +-------------------------------------------------+ + | 0 1 2 3 4 5 6 7 8 9 a b c d e f | ++--------+-------------------------------------------------+----------------+ +|00000000| 45 53 00 00 00 31 00 00 00 00 00 00 00 01 08 08 |ES...1..........| +|00000010| 1e ab f3 00 00 00 1a 00 00 00 16 69 6e 74 65 72 |...........inter| +|00000020| 6e 61 6c 3a 74 63 70 2f 68 61 6e 64 73 68 61 6b |nal:tcp/handshak| +|00000030| 65 00 04 a3 8e b7 41 |e.....A | ++--------+-------------------------------------------------+----------------+ +MESSAGE RECEIVED:E«󀀀internal:tcp/handshake£·A ``` -The original terminal will then log the recieved message if format is validated : + +Extension Name Request / Response : ``` +21:30:18.992 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] READ: 204B +-------------------------------------------------+ | 0 1 2 3 4 5 6 7 8 9 a b c d e f | +--------+-------------------------------------------------+----------------+ -|00000000| 45 53 31 32 33 34 53 48 44 44 46 0d 0a |ES1234SHDDF.. | +|00000000| 45 53 00 00 00 c6 00 00 00 00 00 00 00 03 00 08 |ES..............| +|00000010| 2d c7 23 00 00 00 21 00 00 00 1d 69 6e 74 65 72 |-.#...!....inter| +|00000020| 6e 61 6c 3a 64 69 73 63 6f 76 65 72 79 2f 65 78 |nal:discovery/ex| +|00000030| 74 65 6e 73 69 6f 6e 73 00 00 0e 6e 6f 64 65 5f |tensions...node_| +|00000040| 65 78 74 65 6e 73 69 6f 6e 16 51 53 74 39 6f 4b |extension.QSt9oK| +|00000050| 58 46 54 53 57 71 67 58 34 62 6b 56 6a 47 2d 51 |XFTSWqgX4bkVjG-Q| +|00000060| 09 31 32 37 2e 30 2e 30 2e 31 09 31 32 37 2e 30 |.127.0.0.1.127.0| +|00000070| 2e 30 2e 31 04 7f 00 00 01 09 31 32 37 2e 30 2e |.0.1......127.0.| +|00000080| 30 2e 31 00 00 11 b4 00 04 0f 63 6c 75 73 74 65 |0.1.......cluste| +|00000090| 72 5f 6d 61 6e 61 67 65 72 01 6d 00 04 64 61 74 |r_manager.m..dat| +|000000a0| 61 01 64 01 06 69 6e 67 65 73 74 01 69 00 15 72 |a.d..ingest.i..r| +|000000b0| 65 6d 6f 74 65 5f 63 6c 75 73 74 65 72 5f 63 6c |emote_cluster_cl| +|000000c0| 69 65 6e 74 01 72 00 a3 8e b7 41 00 |ient.r....A. | +--------+-------------------------------------------------+----------------+ -MESSAGE RECEIVED:ES1234SHDDF - -REFERENCE LENGTH 13 ES1234SHDDF +MESSAGE RECEIVED:ES-ǣ!internal:discovery/extensionsnode_extensionQSt9oKXFTSWqgX4bkVjG-Q 127.0.0.1 127.0.0.1 127.0.0.1´cluster_managermdatadingestiremote_cluster_clientr£·A +21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 204, request id: 3, type: request, version: 3.0.0, action: internal:discovery/extensions] READ: 204B +21:30:18.993 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] received request +21:30:18.996 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - register 2 [transport] [internal:discovery/extensions] [] +21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.tasks.TaskManager - unregister task for id: 2 +21:30:18.997 [opensearch[extension][generic][T#1]] TRACE org.opensearch.transport.TransportLogger - Netty4TcpChannel{localAddress=/127.0.0.1:4532, remoteAddress=/127.0.0.1:47782} [length: 48, request id: 3, type: response, version: 3.0.0, header size: 2B] WRITE: 48B +21:30:18.998 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] WRITE: 48B + +-------------------------------------------------+ + | 0 1 2 3 4 5 6 7 8 9 a b c d e f | ++--------+-------------------------------------------------+----------------+ +|00000000| 45 53 00 00 00 2a 00 00 00 00 00 00 00 03 01 08 |ES...*..........| +|00000010| 2d c7 23 00 00 00 02 00 00 16 4e 61 6d 65 64 57 |-.#.......exampl| +|00000020| 72 69 74 65 61 62 6c 65 52 65 67 69 73 74 72 79 |epluginname | ++--------+-------------------------------------------------+----------------+ +21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.latencytester.transportservice.netty4.OpenSearchLoggingHandler - [id: 0xb2be651b, L:/127.0.0.1:4532 - R:/127.0.0.1:47782] FLUSH +21:30:18.999 [opensearch[extension][transport_worker][T#6]] TRACE org.opensearch.transport.TransportService.tracer - [3][internal:discovery/extensions] sent response ``` +It is important that the OpenSearch SDK is already up and running on a seperate process prior to starting OpenSearch, since extension discovery occurs only if the OpenSearch SDK is already listening on a pre-defined port. Once discovery is complete and the data transfer connection between both nodes has been established, OpenSearch and the OpenSearch SDK will now be able to comminicate. + +### Run Tests +Run tests : +``` +./gradlew clean build test +``` \ No newline at end of file