Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement model, model group and connector apis #170

Merged
merged 12 commits into from
Dec 18, 2023
5 changes: 3 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,9 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
- Added class docs generator ([#96](https://github.com/opensearch-project/opensearch-php/pull/96))
- Added support for Amazon OpenSearch Serverless SigV4 signing ([#119](https://github.com/opensearch-project/opensearch-php/pull/119))
- Added `includePortInHostHeader` option to `ClientBuilder::fromConfig` ([#118](https://github.com/opensearch-project/opensearch-php/pull/118))
- Added the `RefreshSearchAnalyzers` endpoint ([[#152](https://github.com/opensearch-project/opensearch-php/issues/152))
- Added the `RefreshSearchAnalyzers` endpoint ([[#152](https://github.com/opensearch-project/opensearch-php/issues/152))
- Added support for `format` parameter to specify the sql response format ([#161](https://github.com/opensearch-project/opensearch-php/pull/161))
- Added ml commons model, model group and connector APIs ([#170](https://github.com/opensearch-project/opensearch-php/pull/170))

### Changed

Expand All @@ -30,4 +31,4 @@ Inspired from [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
### Security


[Unreleased]: https://github.com/opensearch-project/opensearch-php/compare/2.0...HEAD
[Unreleased]: https://github.com/opensearch-project/opensearch-php/compare/2.0...HEAD
30 changes: 17 additions & 13 deletions USER_GUIDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ class MyOpenSearchClass

public function __construct()
{
//simple Setup
// Simple Setup
$this->client = OpenSearch\ClientBuilder::fromConfig([
'hosts' => [
'https://localhost:9200'
Expand Down Expand Up @@ -66,7 +66,7 @@ class MyOpenSearchClass
var_dump($this->client->info());
}

// Create a document
// Create a document
public function create()
{
$time = time();
Expand Down Expand Up @@ -259,15 +259,15 @@ class MyOpenSearchClass
]);
var_dump($docs['hits']['total']['value'] > 0);
}

public function searchByPointInTime()
{
$result = $this->client->createPointInTime([
'index' => INDEX_NAME,
'keep_alive' => '10m'
]);
$pitId = $result['pit_id'];

// Get first page of results in Point-in-Time
$result = $this->client->search([
'body' => [
Expand All @@ -283,7 +283,7 @@ class MyOpenSearchClass
]
]);
var_dump($result['hits']['total']['value'] > 0);

$last = end($result['hits']['hits']);
$lastSort = $last['sort'] ?? null;

Expand All @@ -303,7 +303,7 @@ class MyOpenSearchClass
]
]);
var_dump($result['hits']['total']['value'] > 0);

// Close Point-in-Time
$result = $this->client->deletePointInTime([
'body' => [
Expand Down Expand Up @@ -405,15 +405,15 @@ $client = (new \OpenSearch\ClientBuilder())
->setSigV4Region('us-east-2')

->setSigV4Service('es')

// Default credential provider.
->setSigV4CredentialProvider(true)

->setSigV4CredentialProvider([
'key' => 'awskeyid',
'secret' => 'awssecretkey',
])

->build();
```

Expand All @@ -426,15 +426,15 @@ $client = (new \OpenSearch\ClientBuilder())
->setSigV4Region('us-east-2')

->setSigV4Service('aoss')

// Default credential provider.
->setSigV4CredentialProvider(true)

->setSigV4CredentialProvider([
'key' => 'awskeyid',
'secret' => 'awssecretkey',
])

->build();
```

Expand Down Expand Up @@ -481,7 +481,7 @@ $client = (new \OpenSearch\ClientBuilder())
## Disabling Port Modification

To prevent port modifications, include the `includePortInHostHeader` option into `ClientBuilder::fromConfig`.
This will ensure that the port from the supplied URL is unchanged.
This will ensure that the port from the supplied URL is unchanged.

The following example will force port `9100` usage.

Expand All @@ -501,3 +501,7 @@ $client = \OpenSearch\ClientBuilder::fromConfig($config);

...
```

## Advanced Features

* [ML Commons](guides/ml-commons.md)
71 changes: 71 additions & 0 deletions guides/ml-commons.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Machine Learning Example Usage

Walks through the process of setting up a model to generate
vector embeddings from OpenAI on AWS managed Opensearch.

### Prerequisites

* This example assumes you are using the AWS managed OpenSearch
service. See [The ml commons documentation](https://github.com/opensearch-project/ml-commons/blob/main/docs/remote_inference_blueprints/openai_connector_embedding_blueprint.md) for more examples and further information.
* You will need an API key from OpenAI. [Sign up](https://platform.openai.com/signup)
* The API key must be stored in [AWS Secrets Manager](https://aws.amazon.com/secrets-manager/)

```php
<?php

# Register a model group.
$modelGroupResponse = $client->ml()->registerModelGroup([
'body' => [
'name' => 'openai_model_group',
'description' => 'Group containing models for OpenAI',
],
]);

# Create the connector.
$connectorResponse = $client->ml()->createConnector([
'body' => [
'name' => "Open AI Embedding Connector",
'description' => "Creates a connector to Open AI's embedding endpoint",
'version' => 1,
'protocol' => 'http',
'parameters' => ['model' => 'text-embedding-ada-002'],
'credential' => [
"secretArn" => '<Your Secret ARN from AWS Secrets Manager>',
"roleArn" => '<Your IAM role ARN>',
]
'actions' => [
[
'action_type' => 'predict',
'method' => 'POST',
'url' => 'https://api.openai.com/v1/embeddings',
'headers' => [
'Authorization': 'Bearer ${credential.secretArn.<Your Open AI Secret in Secrets Manager>}'
],
'request_body' => "{ \"input\": \${parameters.input}, \"model\": \"\${parameters.model}\" }",
'pre_process_function' => "connector.pre_process.openai.embedding",
'post_process_function' => "connector.post_process.openai.embedding",
],
],
],
]);

# Register the model.
$registerModelResponse = $client->ml()->registerModel([
'body' => [
'name' => 'OpenAI embedding model',
'function_name' => 'remote',
'model_group_id' => $modelGroupResponse['model_group_id'],
'description' => 'Model for retrieving vector embeddings from OpenAI',
'connector_id' => $connectorResponse['connector_id'],
],
]);

# Monitor the state of the register model task.
$taskResponse = $client->ml()->getTask(['id' => $registerModelResponse['task_id']]);

assert($taskResponse['state'] === 'COMPLETED');

# Finally deploy the model. You will now be able to generate vector
# embeddings from OpenSearch (via OpenAI).
$client->ml()->deployModel(['id' => $taskResponse['model_id']]);
```
12 changes: 12 additions & 0 deletions src/OpenSearch/Client.php
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@
use OpenSearch\Common\Exceptions\TransportException;
use OpenSearch\Endpoints\AbstractEndpoint;
use OpenSearch\Namespaces\AbstractNamespace;
use OpenSearch\Namespaces\MachineLearningNamespace;
use OpenSearch\Namespaces\NamespaceBuilderInterface;
use OpenSearch\Namespaces\BooleanRequestWrapper;
use OpenSearch\Namespaces\CatNamespace;
Expand Down Expand Up @@ -153,6 +154,11 @@ class Client
*/
protected $sql;

/**
* @var MachineLearningNamespace
*/
protected $ml;

/**
* Client constructor
*
Expand All @@ -179,6 +185,7 @@ public function __construct(Transport $transport, callable $endpoint, array $reg
$this->security = new SecurityNamespace($transport, $endpoint);
$this->ssl = new SslNamespace($transport, $endpoint);
$this->sql = new SqlNamespace($transport, $endpoint);
$this->ml = new MachineLearningNamespace($transport, $endpoint);

$this->registeredNamespaces = $registeredNamespaces;
}
Expand Down Expand Up @@ -1348,6 +1355,11 @@ public function sql(): SqlNamespace
return $this->sql;
}

public function ml(): MachineLearningNamespace
{
return $this->ml;
}

/**
* Catchall for registered namespaces
*
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
<?php

declare(strict_types=1);

/**
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

namespace OpenSearch\Endpoints\MachineLearning\Connectors;

use OpenSearch\Endpoints\AbstractEndpoint;

class CreateConnector extends AbstractEndpoint
{
/**
* @return string[]
*/
public function getParamWhitelist(): array
{
return [];
}

/**
* @return string
*/
public function getURI(): string
{
return "/_plugins/_ml/connectors/_create";
}

/**
* @return string
*/
public function getMethod(): string
{
return 'POST';
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<?php

declare(strict_types=1);

/**
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

namespace OpenSearch\Endpoints\MachineLearning\Connectors;

use OpenSearch\Common\Exceptions\RuntimeException;
use OpenSearch\Endpoints\AbstractEndpoint;

class DeleteConnector extends AbstractEndpoint
{
/**
* @return string[]
*/
public function getParamWhitelist(): array
{
return [];
}

/**
* @return string
*/
public function getURI(): string
{
if ($this->id) {
return "/_plugins/_ml/connectors/$this->id";
}

throw new RuntimeException(
'id is required for delete'
);

}

/**
* @return string
*/
public function getMethod(): string
{
return 'DELETE';
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
<?php

declare(strict_types=1);

/**
* Copyright OpenSearch Contributors
* SPDX-License-Identifier: Apache-2.0
*
* The OpenSearch Contributors require contributions made to
* this file be licensed under the Apache-2.0 license or a
* compatible open source license.
*/

namespace OpenSearch\Endpoints\MachineLearning\Connectors;

use OpenSearch\Common\Exceptions\RuntimeException;
use OpenSearch\Endpoints\AbstractEndpoint;

class GetConnector extends AbstractEndpoint
{
/**
* @return string[]
*/
public function getParamWhitelist(): array
{
return [];
}

/**
* @return string
*/
public function getURI(): string
{
if ($this->id) {
return "/_plugins/_ml/connectors/$this->id";
}

throw new RuntimeException(
'id is required for get'
);

}

/**
* @return string
*/
public function getMethod(): string
{
return 'GET';
}
}
Loading
Loading