Make NamedXContent available to extensions (opensearch-project#244)

* Make NamedXContent available to extensions Signed-off-by: Daniel Widdis <[email protected]> * Add tests Signed-off-by: Daniel Widdis <[email protected]> * No need to save custom list in registry as it's on ExtensionsRunner Signed-off-by: Daniel Widdis <[email protected]> * Add sequence diagram to DESIGN.md Signed-off-by: Daniel Widdis <[email protected]> Signed-off-by: Daniel Widdis <[email protected]>
kokibas · Mar 17, 2023 · a9c1063 · a9c1063
1 parent f6b60a1
commit a9c1063
Show file tree

Hide file tree

Showing 9 changed files with 380 additions and 19 deletions.
diff --git a/DESIGN.md b/DESIGN.md
@@ -74,10 +74,10 @@ The sequence diagram below shows the process of initializing an Extension, regis
 
 The `org.opensearch.sdk.sample` package contains a sample `HelloWorldExtension` implementing the below steps. It is executed following the steps in the [`DEVELOPER_GUIDE`](DEVELOPER_GUIDE.md).
 
-![](Docs/ExtensionRestActions.svg)
-
 #### Extension REST Actions Walk Through
 
+![](Docs/ExtensionRestActions.svg)
+
 ##### Extension Startup
 
 (1) Extensions must implement the `Extension` interface which requires them to define their settings (name, host address and port) and a list of `ExtensionRestHandler` implementations they will handle.  They are started up using a `main()` method which passes an instance of the extension to the `ExtensionsRunner` using `ExtensionsRunner.run(this)`.
@@ -116,9 +116,35 @@ The `ExtensionsOrchestrator` reads a list of extensions present in `extensions.y
 
 (27) The User receives the response.
 
+#### Extension Point Implementation Walk Through
+
+An example of a more complex extension point, `getNamedXContent()` is shown below. A similar pattern can be followed for most extension points.
+
+![](Docs/NamedXContent.svg)
+
+##### Extension Startup
+
+(1, 2) Extensions initialize by passing an instance of themselves to the `ExtensionsRunner`. The first step in the constructor is for the `ExtensionsRunner` to pass its own instance back to the Extension via setter.
+
+(3, 4) The `Extension` interface includes extensions points such as `getNamedXContent()` (returning a default empty list). If overridden, the Extension will return a list of `NamedXContentRegistry.Entry` which will be saved as `customNamedXContent`. Other extension points operate in a similar manner.
+
+(5) The `ExtensionsRunner` registers an `ExtensionInitRequestHandler` which will complete the initialization process on OpenSearch startup.
+
+##### OpenSearch Startup, Extension Initialization, and NamedXContent Registration
+
+(6) Upon receipt of an `InitializeExtensionsRequest` (among other actions):
+
+(7, 8) Obtains Environment Settings from OpenSearch, necessary for some core XContent.
+
+(9, 10) Instantiates a new `ExtensionNamedXContentRegistry` which is set on the ExtensionsRunner.
+This uses the OpenSearch environment settings along with NamedXContent from several OpenSearch modules,
+and combines it the custom Extension NamedXContent.
+
+Since the Extension has an instance of the ExtensionsRunner, it can now access the registry via getter and pass it to Extension Rest Handlers as needed.
+
 ## FAQ
 
 - Will extensions replace plugins?
-  Plugins will continue to be supported and extensions are preferred as they will be easier to develop, deploy, and operate.
+  Plugins will continue to be supported in the near term but are on a path to deprecation. New development should consider using extensions, as they will be easier to develop, deploy, and operate.
 - How is the latency going to be for extensions?
   https://github.com/opensearch-project/OpenSearch/issues/3012#issuecomment-1122682444
diff --git a/Docs/NamedXContent.svg b/Docs/NamedXContent.svg
diff --git a/src/main/java/org/opensearch/sdk/BaseExtension.java b/src/main/java/org/opensearch/sdk/BaseExtension.java
@@ -20,6 +20,11 @@
  * An abstract class that provides sample methods required by extensions
  */
 public abstract class BaseExtension implements Extension {
+    /**
+     * The {@link ExtensionsRunner} instance running this extension
+     */
+    protected ExtensionsRunner extensionsRunner;
+
     /**
      * A client to make requests to the system
      */
@@ -66,14 +71,12 @@ public ExtensionSettings getExtensionSettings() {
         return this.settings;
     }
 
-    /**
-     * Returns components added by this extension.
-     *
-     * @param client A client to make requests to the system
-     * @param clusterService A service to allow watching and updating cluster state
-     * @param threadPool A service to allow retrieving an executor to run an async action
-     * @return A collection of objects
-     */
+    @Override
+    public void setExtensionsRunner(ExtensionsRunner extensionsRunner) {
+        this.extensionsRunner = extensionsRunner;
+    }
+
+    @Override
     public Collection<Object> createComponents(SDKClient client, ClusterService clusterService, ThreadPool threadPool) {
         this.client = client;
         this.clusterService = clusterService;

diff --git a/src/main/java/org/opensearch/sdk/Extension.java b/src/main/java/org/opensearch/sdk/Extension.java
@@ -21,6 +21,7 @@
 import org.opensearch.action.ActionResponse;
 import org.opensearch.action.support.TransportAction;
 import org.opensearch.common.settings.Setting;
+import org.opensearch.common.xcontent.NamedXContentRegistry;
 
 /**
  * This interface defines methods which an extension must provide. Extensions
@@ -52,6 +53,22 @@ default List<Setting<?>> getSettings() {
         return Collections.emptyList();
     }
 
+    /**
+     * Gets an optional list of custom {@link NamedXContentRegistry.Entry} for the extension to combine with OpenSearch NamedXConent.
+     *
+     * @return a list of custom NamedXConent this extension uses.
+     */
+    default List<NamedXContentRegistry.Entry> getNamedXContent() {
+        return Collections.emptyList();
+    }
+
+    /**
+     * Set the Extension's instance of its corresponding {@link ExtensionsRunner}.
+     *
+     * @param extensionsRunner The ExtensionsRunner running this extension.
+     */
+    void setExtensionsRunner(ExtensionsRunner extensionsRunner);
+
     /**
      * Returns components added by this extension.
      *
@@ -60,7 +77,7 @@ default List<Setting<?>> getSettings() {
      * @param threadPool A service to allow retrieving an executor to run an async action
      * @return A collection of objects
      */
-    public Collection<Object> createComponents(SDKClient client, ClusterService clusterService, ThreadPool threadPool);
+    Collection<Object> createComponents(SDKClient client, ClusterService clusterService, ThreadPool threadPool);
 
     /**
      * Gets an optional list of custom {@link TransportAction} for the extension to register with OpenSearch.

diff --git a/src/main/java/org/opensearch/sdk/ExtensionNamedXContentRegistry.java b/src/main/java/org/opensearch/sdk/ExtensionNamedXContentRegistry.java
@@ -0,0 +1,59 @@
+/*
+ * Copyright OpenSearch Contributors
+ * SPDX-License-Identifier: Apache-2.0
+ *
+ * The OpenSearch Contributors require contributions made to
+ * this file be licensed under the Apache-2.0 license or a
+ * compatible open source license.
+ */
+
+package org.opensearch.sdk;
+
+import static java.util.stream.Collectors.toList;
+
+import java.util.Collections;
+import java.util.List;
+import java.util.function.Function;
+import java.util.stream.Stream;
+
+import org.opensearch.cluster.ClusterModule;
+import org.opensearch.common.network.NetworkModule;
+import org.opensearch.common.settings.Settings;
+import org.opensearch.common.xcontent.NamedXContentRegistry;
+import org.opensearch.indices.IndicesModule;
+import org.opensearch.search.SearchModule;
+
+/**
+ * Combines Extension NamedXContent with core OpenSearch NamedXContent
+ */
+public class ExtensionNamedXContentRegistry {
+    private final NamedXContentRegistry namedXContentRegistry;
+
+    /**
+     * Creates and populates a NamedXContentRegistry with the given NamedXContentRegistry entries for this extension and
+     * locally defined content.
+     *
+     * @param settings OpenSearch environment settings
+     * @param extensionNamedXContent List of NamedXContentRegistry.Entry to be registered
+     */
+    public ExtensionNamedXContentRegistry(Settings settings, List<NamedXContentRegistry.Entry> extensionNamedXContent) {
+        this.namedXContentRegistry = new NamedXContentRegistry(
+            Stream.of(
+                extensionNamedXContent.stream(),
+                NetworkModule.getNamedXContents().stream(),
+                IndicesModule.getNamedXContents().stream(),
+                new SearchModule(settings, Collections.emptyList()).getNamedXContents().stream(),
+                ClusterModule.getNamedXWriteables().stream()
+            ).flatMap(Function.identity()).collect(toList())
+        );
+    }
+
+    /**
+     * Gets the NamedXContentRegistry.
+     *
+     * @return The NamedXContentRegistry. Includes both extension-defined XContent and core OpenSearch XContent.
+     */
+    public NamedXContentRegistry getRegistry() {
+        return this.namedXContentRegistry;
+    }
+}
diff --git a/src/main/java/org/opensearch/sdk/ExtensionsRunner.java b/src/main/java/org/opensearch/sdk/ExtensionsRunner.java
@@ -20,6 +20,7 @@
 import org.opensearch.extensions.settings.RegisterCustomSettingsRequest;
 import org.opensearch.common.settings.Setting;
 import org.opensearch.common.settings.Settings;
+import org.opensearch.common.xcontent.NamedXContentRegistry;
 import org.opensearch.discovery.InitializeExtensionsRequest;
 import org.opensearch.extensions.ExtensionActionListenerOnFailureRequest;
 import org.opensearch.extensions.DiscoveryExtension;
@@ -50,6 +51,7 @@
 
 import java.io.IOException;
 import java.util.ArrayList;
+import java.util.Collections;
 import java.util.List;
 import java.util.Map;
 import java.util.function.Consumer;
@@ -64,6 +66,9 @@ public class ExtensionsRunner {
     private static final Logger logger = LogManager.getLogger(ExtensionsRunner.class);
     private static final String NODE_NAME_SETTING = "node.name";
 
+    // Set when initialization is complete
+    private boolean initialized = false;
+
     private String uniqueId;
     /**
      * This field is initialized by a call from {@link ExtensionsInitRequestHandler}.
@@ -77,16 +82,27 @@ public class ExtensionsRunner {
 
     // The routes and classes which handle the REST requests
     private final ExtensionRestPathRegistry extensionRestPathRegistry = new ExtensionRestPathRegistry();
-    // Custom settings from the extension's getSettings
     /**
-     * This field is initialized by a call from {@link ExtensionsInitRequestHandler}.
+     * Custom namedXContent from the extension's getNamedXContent. This field is initialized in the constructor.
+     */
+    private final List<NamedXContentRegistry.Entry> customNamedXContent;
+    /**
+     * Custom settings from the extension's getSettings. This field is initialized in the constructor.
      */
     private final List<Setting<?>> customSettings;
-    // Node name, host, and port
     /**
-     * This field is initialized by a call from {@link ExtensionsInitRequestHandler}.
+     * Environment settings from OpenSearch. This field is initialized by a call from
+     * {@link ExtensionsInitRequestHandler}.
+     */
+    private Settings environmentSettings = Settings.EMPTY;
+    /**
+     * Node name, host, and port. This field is initialized by a call from {@link ExtensionsInitRequestHandler}.
      */
     public final Settings settings;
+    private ExtensionNamedXContentRegistry extensionNamedXContentRegistry = new ExtensionNamedXContentRegistry(
+        Settings.EMPTY,
+        Collections.emptyList()
+    );
     private ExtensionNamedWriteableRegistry namedWriteableRegistry = new ExtensionNamedWriteableRegistry();
     private ExtensionsInitRequestHandler extensionsInitRequestHandler = new ExtensionsInitRequestHandler(this);
     private OpensearchRequestHandler opensearchRequestHandler = new OpensearchRequestHandler(namedWriteableRegistry);
@@ -114,10 +130,11 @@ public class ExtensionsRunner {
     /**
      * Instantiates a new Extensions Runner using the specified extension.
      *
-     * @param extension  The settings with which to start the runner.
+     * @param extension The settings with which to start the runner.
      * @throws IOException if the runner failed to read settings or API.
      */
     protected ExtensionsRunner(Extension extension) throws IOException {
+        extension.setExtensionsRunner(this);
         ExtensionSettings extensionSettings = extension.getExtensionSettings();
         this.settings = Settings.builder()
             .put(NODE_NAME_SETTING, extensionSettings.getExtensionName())
@@ -132,6 +149,8 @@ protected ExtensionsRunner(Extension extension) throws IOException {
         }
         // save custom settings
         this.customSettings = extension.getSettings();
+        // save custom namedXContent
+        this.customNamedXContent = extension.getNamedXContent();
         // save custom transport actions
         this.transportActions = new TransportActions(extension.getActions());
 
@@ -142,15 +161,70 @@ protected ExtensionsRunner(Extension extension) throws IOException {
     }
 
     /**
-     * This method is call from {@link ExtensionsInitRequestHandler}.
+     * Marks the extension initialized.
+     */
+    public void setInitialized() {
+        this.initialized = true;
+        logger.info("Extension initialization is complete!");
+    }
+
+    /**
+     * Reports if the extension has finished initializing.
+     *
+     * @return true if the extension has been initialized
+     */
+    boolean isInitialized() {
+        return this.initialized;
+    }
+
+    /**
+     * Sets the TransportService. Called from {@link ExtensionsInitRequestHandler}.
+     *
      * @param extensionTransportService assign value for extensionTransportService
      */
     void setExtensionTransportService(TransportService extensionTransportService) {
         this.extensionTransportService = extensionTransportService;
     }
 
+    /**
+     * Sets the Environment Settings. Called from {@link ExtensionsInitRequestHandler}.
+     *
+     * @param settings assign value for environmentSettings
+     */
+    public void setEnvironmentSettings(Settings settings) {
+        this.environmentSettings = settings;
+    }
+
+    /**
+     * Gets the Environment Settings. Only valid if {@link #isInitialized()} returns true.
+     *
+     * @return the environment settings if initialized, an empty settings object otherwise.
+     */
+    public Settings getEnvironmentSettings() {
+        return this.environmentSettings;
+    }
+
+    /**
+     * Sets the NamedXContentRegistry. Called from {@link ExtensionsInitRequestHandler}.
+     *
+     * @param registry assign value for namedXContentRegistry
+     */
+    public void setNamedXContentRegistry(ExtensionNamedXContentRegistry registry) {
+        this.extensionNamedXContentRegistry = registry;
+    }
+
+    /**
+     * Gets the NamedXContentRegistry. Only valid if {@link #isInitialized()} returns true.
+     *
+     * @return the NamedXContentRegistry if initialized, an empty registry otherwise.
+     */
+    public ExtensionNamedXContentRegistry getNamedXContentRegistry() {
+        return this.extensionNamedXContentRegistry;
+    }
+
     /**
      * Sets the Unique ID, used in REST requests to uniquely identify this extension
+     *
      * @param id assign value for id
      */
     public void setUniqueId(String id) {
@@ -173,6 +247,10 @@ public DiscoveryNode getOpensearchNode() {
         return this.opensearchNode;
     }
 
+    public List<NamedXContentRegistry.Entry> getCustomNamedXContent() {
+        return this.customNamedXContent;
+    }
+
     /**
      * Starts a TransportService.
      *

diff --git a/src/main/java/org/opensearch/sdk/handlers/ExtensionsInitRequestHandler.java b/src/main/java/org/opensearch/sdk/handlers/ExtensionsInitRequestHandler.java
@@ -11,8 +11,10 @@
 
 import org.apache.logging.log4j.LogManager;
 import org.apache.logging.log4j.Logger;
+import org.opensearch.common.settings.Settings;
 import org.opensearch.discovery.InitializeExtensionsRequest;
 import org.opensearch.discovery.InitializeExtensionsResponse;
+import org.opensearch.sdk.ExtensionNamedXContentRegistry;
 import org.opensearch.sdk.ExtensionsRunner;
 import org.opensearch.transport.TransportService;
 
@@ -28,14 +30,15 @@ public class ExtensionsInitRequestHandler {
 
     /**
      * Instantiate this object with a reference to the ExtensionsRunner
+     *
      * @param extensionsRunner the ExtensionsRunner instance
      */
     public ExtensionsInitRequestHandler(ExtensionsRunner extensionsRunner) {
         this.extensionsRunner = extensionsRunner;
     }
 
     /**
-     * Handles a extension request from OpenSearch.  This is the first request for the transport communication and will initialize the extension and will be a part of OpenSearch bootstrap.
+     * Handles a extension request from OpenSearch. This is the first request for the transport communication and will initialize the extension and will be a part of OpenSearch bootstrap.
      *
      * @param extensionInitRequest  The request to handle.
      * @return A response to OpenSearch validating that this is an extension.
@@ -60,6 +63,15 @@ public InitializeExtensionsResponse handleExtensionInitRequest(InitializeExtensi
                 extensionsRunner.opensearchNode,
                 extensionsRunner.getUniqueId()
             );
+            // Get OpenSearch Settings and set values on ExtensionsRunner
+            Settings settings = extensionsRunner.sendEnvironmentSettingsRequest(extensionTransportService);
+            extensionsRunner.setEnvironmentSettings(settings);
+            extensionsRunner.setNamedXContentRegistry(
+                new ExtensionNamedXContentRegistry(settings, extensionsRunner.getCustomNamedXContent())
+            );
+
+            // Last step of initialization
+            extensionsRunner.setInitialized();
         }
     }
 }