Skip to content

Commit

Permalink
Merge branch 'main' into feat-sidecar
Browse files Browse the repository at this point in the history
Signed-off-by: tygao <[email protected]>
  • Loading branch information
raintygao authored Mar 2, 2024
2 parents bf341c6 + 2c8d9d3 commit c7ddbda
Show file tree
Hide file tree
Showing 71 changed files with 4,507 additions and 152 deletions.
2 changes: 1 addition & 1 deletion .github/CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -1 +1 @@
* @ananzh @kavilla @AMoo-Miki @ashwin-pc @joshuarrrr @abbyhu2000 @zengyan-amazon @kristenTian @zhongnansu @manasvinibs @ZilongX @Flyingliuhub @BSFishy @curq @bandinib-amzn @SuZhou-Joe
* @ananzh @kavilla @AMoo-Miki @ashwin-pc @joshuarrrr @abbyhu2000 @zengyan-amazon @kristenTian @zhongnansu @manasvinibs @ZilongX @Flyingliuhub @BSFishy @curq @bandinib-amzn @SuZhou-Joe @ruanyl @BionIT
2 changes: 1 addition & 1 deletion .github/workflows/build_and_test_workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ name: Build and test
# trigger on every commit push and PR for all branches except pushes for backport branches
on:
push:
branches: ['**', '!backport/**']
branches: ['main', '[0-9].x', '[0-9].[0=9]+'] # Run the functional test on push for only release branches
paths-ignore:
- '**/*.md'
- 'docs/**'
Expand Down
40 changes: 40 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,35 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)

### Deprecations

### 🛡 Security

### 📈 Features/Enhancements

### 🐛 Bug Fixes

- [BUG][Discover] Allow saved sort from search embeddable to load in Dashboard ([#5934](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5934))
- [BUG][Discover] Add key to index pattern options for support deplicate index pattern names([#5946](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5946))
- [Discover] Fix table cell content overflowing in Safari ([#5948](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5948))
- [BUG][MD]Fix schema for test connection to separate validation based on auth type([#5997](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5997))

### 🚞 Infrastructure

### 📝 Documentation

- Fix link to documentation for geoHash precision ([#5967](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5967))

### 🛠 Maintenance

### 🪛 Refactoring

### 🔩 Tests

## [2.12.0 - 2024-02-20](https://github.com/opensearch-project/OpenSearch-Dashboards/releases/tag/2.12.0)

### 💥 Breaking Changes

### Deprecations

- Rename `withLongNumerals` to `withLongNumeralsSupport` in `HttpFetchOptions` [#5592](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5592)

### 🛡 Security
Expand Down Expand Up @@ -41,6 +70,11 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- [Multiple Datasource] Add interfaces to register add-on authentication method from plug-in module ([#5851](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5851))
- [Multiple Datasource] Able to Hide "Local Cluster" option from datasource DropDown ([#5827](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5827))
- [Multiple Datasource] Add api registry and allow it to be added into client config in data source plugin ([#5895](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5895))
- [Multiple Datasource] Concatenate data source name with index pattern name and change delimiter to double colon ([#5907](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5907))
- [Multiple Datasource] Refactor client and legacy client to use authentication registry ([#5881](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5881))
- [Multiple Datasource] Improved error handling for the search API when a null value is passed for the dataSourceId ([#5882](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5882))
- [Multiple Datasource] Hide/Show authentication method in multi data source plugin based on configuration ([#5916](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5916))
- [[Dynamic Configurations] Add support for dynamic application configurations ([#5855](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5855))
- Add sidecar service ([#5920](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5920))

### 🐛 Bug Fixes
Expand Down Expand Up @@ -69,6 +103,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- [BUG] Remove duplicate sample data as id 90943e30-9a47-11e8-b64d-95841ca0b247 ([5668](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5668))
- [BUG][Multiple Datasource] Fix datasource testing connection unexpectedly passed with wrong endpoint [#5663](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5663)
- [Table Visualization] Fix filter action buttons for split table aggregations ([#5619](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5619))
- [osd/std] Add additional recovery from false-positives in handling of long numerals ([#5956](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5956))
- [BUG][Discover] Allow saved sort from search embeddable to load in Dashboard ([#5934](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5934))
- [BUG][Multiple Datasource] Fix missing customApiRegistryPromise param for test connection ([#5944](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5944))

### 🚞 Infrastructure

Expand Down Expand Up @@ -101,6 +138,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Add @SuZhou-Joe as a maintainer ([#5594](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5594))
- Move @seanneumann to emeritus maintainer ([#5634](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5634))
- Remove `ui-select` dev dependency ([#5660](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5660))
- Bump `chromedriver` dependency to `121.0.1"` ([#5926](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5926))
- Add @ruanyl as a maintainer ([#5982](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5982))
- Add @BionIT as a maintainer ([#5988](https://github.com/opensearch-project/OpenSearch-Dashboards/pull/5988))

### 🪛 Refactoring

Expand Down
2 changes: 2 additions & 0 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ This document contains a list of maintainers in this repo. See [opensearch-proje
| Sirazh Gabdullin | [curq](https://github.com/curq) | External contributor |
| Bandini Bhopi | [bandinib-amzn](https://github.com/bandinib-amzn) | Amazon |
| Su Zhou | [SuZhou-Joe](https://github.com/SuZhou-Joe) | Amazon |
| Yulong Ruan | [ruanyl](https://github.com/ruanyl) | Amazon |
| Lu Yu | [BionIT](https://github.com/BionIT) | Amazon |

## Emeritus

Expand Down
18 changes: 18 additions & 0 deletions config/opensearch_dashboards.yml
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,13 @@
# dashboards. OpenSearch Dashboards creates a new index if the index doesn't already exist.
#opensearchDashboards.index: ".opensearch_dashboards"

# OpenSearch Dashboards uses an index in OpenSearch to store dynamic configurations.
# This shall be a different index from opensearchDashboards.index.
# opensearchDashboards.configIndex: ".opensearch_dashboards_config"

# Set the value of this setting to true to enable plugin application config. By default it is disabled.
# application_config.enabled: false

# The default application to load.
#opensearchDashboards.defaultAppId: "home"

Expand Down Expand Up @@ -270,6 +277,17 @@
# 'ff00::/8',
# ]

# Set enabled false to hide authentication method in OpenSearch Dashboards.
# If this setting is commented then all 3 options will be available in OpenSearch Dashboards.
# Default value will be considered to True.
#data_source.authTypes:
# NoAuthentication:
# enabled: true
# UsernamePassword:
# enabled: true
# AWSSigV4:
# enabled: true

# Set the value of this setting to false to hide the help menu link to the OpenSearch Dashboards user survey
# opensearchDashboards.survey.url: "https://survey.opensearch.org"

Expand Down
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -357,7 +357,7 @@
"chai": "3.5.0",
"chance": "1.0.18",
"cheerio": "1.0.0-rc.1",
"chromedriver": "^119.0.1",
"chromedriver": "^121.0.1",
"classnames": "2.3.1",
"compare-versions": "3.5.1",
"cypress": "9.5.4",
Expand Down
32 changes: 27 additions & 5 deletions packages/osd-std/src/json.ts
Original file line number Diff line number Diff line change
Expand Up @@ -150,16 +150,38 @@ const parseStringWithLongNumerals = (
} catch (e) {
hadException = true;
/* There are two types of exception objects that can be raised:
* 1) a proper object with lineNumber and columnNumber which we can use
* 2) a textual message with the position that we need to parse
* 1) a textual message with the position that we need to parse
* i. Unexpected [token|string ...] at position ...
* ii. expected ',' or '}' after property value in object at line ... column ...
* 2) a proper object with lineNumber and columnNumber which we can use
* Note: this might refer to the part of the code that threw the exception but
* we will try it anyway; the regex is specific enough to not produce
* false-positives.
*/
let { lineNumber, columnNumber } = e;
if (!lineNumber || !columnNumber) {
const match = e?.message?.match?.(/^Unexpected token.*at position (\d+)$/);

if (typeof e?.message === 'string') {
/* Check for 1-i (seen in Node)
* Finding "..."෴1111"..." inside a string value, the extra quotes throw a syntax error
* and the position points to " that is assumed to be the begining of a value.
*/
let match = e.message.match(/^Unexpected .*at position (\d+)(\s|$)/);
if (match) {
lineNumber = 1;
// The position is zero-indexed; adding 1 to normalize it for the -2 that comes later
// Add 1 to reach the marker
columnNumber = parseInt(match[1], 10) + 1;
} else {
/* Check for 1-ii (seen in browsers)
* Finding "...,"෴1111"..." inside a string value, the extra quotes throw a syntax error
* and the column number points to the marker after the " that is assumed to be terminating the
* value.
*/
// ToDo: Add functional tests for this path
match = e.message.match(/expected .*at line (\d+) column (\d+)(\s|$)/);
if (match) {
lineNumber = parseInt(match[1], 10);
columnNumber = parseInt(match[2], 10);
}
}
}

Expand Down
1 change: 1 addition & 0 deletions src/core/server/mocks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ export function pluginInitializerContextConfigMock<T>(config: T) {
const globalConfig: SharedGlobalConfig = {
opensearchDashboards: {
index: '.opensearch_dashboards_tests',
configIndex: '.opensearch_dashboards_config_tests',
autocompleteTerminateAfter: duration(100000),
autocompleteTimeout: duration(1000),
},
Expand Down
1 change: 1 addition & 0 deletions src/core/server/opensearch_dashboards_config.ts
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ export const config = {
schema: schema.object({
enabled: schema.boolean({ defaultValue: true }),
index: schema.string({ defaultValue: '.kibana' }),
configIndex: schema.string({ defaultValue: '.opensearch_dashboards_config' }),
autocompleteTerminateAfter: schema.duration({ defaultValue: 100000 }),
autocompleteTimeout: schema.duration({ defaultValue: 1000 }),
branding: schema.object({
Expand Down
1 change: 1 addition & 0 deletions src/core/server/plugins/plugin_context.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,7 @@ describe('createPluginInitializerContext', () => {
expect(configObject).toStrictEqual({
opensearchDashboards: {
index: '.kibana',
configIndex: '.opensearch_dashboards_config',
autocompleteTerminateAfter: duration(100000),
autocompleteTimeout: duration(1000),
},
Expand Down
7 changes: 6 additions & 1 deletion src/core/server/plugins/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -287,7 +287,12 @@ export interface Plugin<

export const SharedGlobalConfigKeys = {
// We can add more if really needed
opensearchDashboards: ['index', 'autocompleteTerminateAfter', 'autocompleteTimeout'] as const,
opensearchDashboards: [
'index',
'configIndex',
'autocompleteTerminateAfter',
'autocompleteTimeout',
] as const,
opensearch: ['shardTimeout', 'requestTimeout', 'pingTimeout'] as const,
path: ['data'] as const,
savedObjects: ['maxImportPayloadBytes'] as const,
Expand Down
1 change: 1 addition & 0 deletions src/legacy/server/config/schema.js
Original file line number Diff line number Diff line change
Expand Up @@ -227,6 +227,7 @@ export default () =>
opensearchDashboards: Joi.object({
enabled: Joi.boolean().default(true),
index: Joi.string().default('.kibana'),
configIndex: Joi.string().default('.opensearch_dashboards_config'),
autocompleteTerminateAfter: Joi.number().integer().min(1).default(100000),
// TODO Also allow units here like in opensearch config once this is moved to the new platform
autocompleteTimeout: Joi.number().integer().min(1).default(1000),
Expand Down
112 changes: 112 additions & 0 deletions src/plugins/application_config/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,112 @@
# ApplicationConfig Plugin

An OpenSearch Dashboards plugin for application configuration and a default implementation based on OpenSearch as storage.

---

## Introduction

This plugin introduces the support of dynamic application configurations as opposed to the existing static configuration in OSD YAML file `opensearch_dashboards.yml`. It stores the configuration in an index whose default name is `.opensearch_dashboards_config` and could be customized through the key `opensearchDashboards.configIndex` in OSD YAML file. Initially the new index does not exist. Only OSD users who need dynamic configurations will create it.

It also provides an interface `ConfigurationClient` for future extensions of external configuration clients. A default implementation based on OpenSearch as database is used.

This plugin is disabled by default.

## Configuration

OSD users who want to set up application configurations will first need to enable this plugin by the following line in OSD YML.

```
application_config.enabled: true
```

Then they can perform configuration operations through CURL the OSD APIs.

(Note that the commands following could be first obtained from a copy as curl option from the network tab of a browser development tool and then replaced with the API names)

Below is the CURL command to view all configurations.

```
curl '{osd endpoint}/api/appconfig' -X GET
```

Below is the CURL command to view the configuration of an entity.

```
curl '{osd endpoint}/api/appconfig/{entity}' -X GET
```

Below is the CURL command to update the configuration of an entity.

```
curl '{osd endpoint}/api/appconfig/{entity}' -X POST -H 'Accept: application/json' -H 'Content-Type: application/json' -H 'osd-xsrf: osd-fetch' -H 'Sec-Fetch-Dest: empty' --data-raw '{"newValue":"{new value}"}'
```

Below is the CURL command to delete the configuration of an entity.

```
curl '{osd endpoint}/api/appconfig/{entity}' -X DELETE -H 'osd-xsrf: osd-fetch' -H 'Sec-Fetch-Dest: empty'
```


## External Configuration Clients

While a default OpenSearch based client is implemented, OSD users can use external configuration clients through an OSD plugin (outside OSD).

Let's call this plugin `MyConfigurationClientPlugin`.

First, this plugin will need to implement a class `MyConfigurationClient` based on interface `ConfigurationClient` defined in the `types.ts` under directory `src/plugins/application_config/server/types.ts`. Below are the functions inside the interface.

```
getConfig(): Promise<Map<string, string>>;
getEntityConfig(entity: string): Promise<string>;
updateEntityConfig(entity: string, newValue: string): Promise<string>;
deleteEntityConfig(entity: string): Promise<string>;
```

Second, this plugin needs to declare `applicationConfig` as its dependency by adding it to `requiredPlugins` in its own `opensearch_dashboards.json`.

Third, the plugin will define a new type called `AppPluginSetupDependencies` as follows in its own `types.ts`.

```
export interface AppPluginSetupDependencies {
applicationConfig: ApplicationConfigPluginSetup;
}
```

Then the plugin will import the new type `AppPluginSetupDependencies` and add to its own setup input. Below is the skeleton of the class `MyConfigurationClientPlugin`.

```
// MyConfigurationClientPlugin
public setup(core: CoreSetup, { applicationConfig }: AppPluginSetupDependencies) {
...
// The function createClient provides an instance of ConfigurationClient which
// could have a underlying DynamoDB or Postgres implementation.
const myConfigurationClient: ConfigurationClient = this.createClient();
applicationConfig.registerConfigurationClient(myConfigurationClient);
...
return {};
}
```

## Onboarding Configurations

Since the APIs and interfaces can take an entity, a new use case to this plugin could just pass their entity into the parameters. There is no need to implement new APIs or interfaces. To programmatically call the functions in `ConfigurationClient` from a plugin (the caller plugin), below is the code example.

Similar to [section](#external-configuration-clients), a new type `AppPluginSetupDependencies` which encapsulates `ApplicationConfigPluginSetup` is needed. Then it can be imported into the `setup` function of the caller plugin. Then the caller plugin will have access to the `getConfigurationClient` and `registerConfigurationClient` exposed by `ApplicationConfigPluginSetup`.

## Development

See the [OpenSearch Dashboards contributing
guide](https://github.com/opensearch-project/OpenSearch-Dashboards/blob/main/CONTRIBUTING.md) for instructions
setting up your development environment.
7 changes: 7 additions & 0 deletions src/plugins/application_config/common/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

export const PLUGIN_ID = 'applicationConfig';
export const PLUGIN_NAME = 'application_config';
12 changes: 12 additions & 0 deletions src/plugins/application_config/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

import { schema, TypeOf } from '@osd/config-schema';

export const configSchema = schema.object({
enabled: schema.boolean({ defaultValue: false }),
});

export type ApplicationConfigSchema = TypeOf<typeof configSchema>;
9 changes: 9 additions & 0 deletions src/plugins/application_config/opensearch_dashboards.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{
"id": "applicationConfig",
"version": "opensearchDashboards",
"opensearchDashboardsVersion": "opensearchDashboards",
"server": true,
"ui": false,
"requiredPlugins": [],
"optionalPlugins": []
}
23 changes: 23 additions & 0 deletions src/plugins/application_config/server/index.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
/*
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*/

import { PluginConfigDescriptor, PluginInitializerContext } from '../../../core/server';
import { ApplicationConfigSchema, configSchema } from '../config';
import { ApplicationConfigPlugin } from './plugin';

/*
This exports static code and TypeScript types,
as well as, OpenSearch Dashboards Platform `plugin()` initializer.
*/

export const config: PluginConfigDescriptor<ApplicationConfigSchema> = {
schema: configSchema,
};

export function plugin(initializerContext: PluginInitializerContext) {
return new ApplicationConfigPlugin(initializerContext);
}

export { ApplicationConfigPluginSetup, ApplicationConfigPluginStart } from './types';
Loading

0 comments on commit c7ddbda

Please sign in to comment.