Skip to content

Commit

Permalink
adding blueprint examples for remote inference (#1155)
Browse files Browse the repository at this point in the history 
* adding blueprint examples for remote inference

Signed-off-by: Dhrubo Saha <[email protected]>

* updating blue prints with sample response

Signed-off-by: Dhrubo Saha <[email protected]>

* addressing comments

Signed-off-by: Dhrubo Saha <[email protected]>

* addressed comments for sagemaker

Signed-off-by: Dhrubo Saha <[email protected]>

---------

Signed-off-by: Dhrubo Saha <[email protected]>
  • Loading branch information
@dhrubo-os
dhrubo-os authored Jul 25, 2023
1 parent f6530e0 commit 8b241a9
Show file tree
Hide file tree
Showing 5 changed files with 402 additions and 0 deletions.
81 changes: 81 additions & 0 deletions docs/remote_inference_blueprints/cohere_connector_embedding_blueprint.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
### Cohere connector blueprint example for embedding:

#### this blueprint is created from Cohere doc: https://docs.cohere.com/reference/embed

```json
POST /_plugins/_ml/connectors/_create
{
"name": "<YOUR MODEL NAME>",
"description": "<YOUR MODEL DESCRIPTION>",
"version": "<YOUR MODEL VERSION>",
"protocol": "http",
"credential": {
"cohere_key": "<PLEASE ADD YOUR Cohere API KEY HERE>"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://api.cohere.ai/v1/embed",
"headers": {
"Authorization": "Bearer ${credential.cohere_key}"
},
"request_body": "{ \"texts\": ${parameters.prompt}, \"truncate\": \"END\" }"
}
]
}
```
#### Sample response
```json
{
"connector_id": "XU5UiokBpXT9icfOM0vt"
}
```


### Corresponding Predict request example:

```json
POST /_plugins/_ml/models/<ENTER MODEL ID HERE>/_predict
{
"parameters": {
"prompt": ["Say this is a test"]
}
}
```

#### Sample response
```json
{
"inference_results": [
{
"output": [
{
"name": "response",
"dataAsMap": {
"id": "39097276-d926-4ca1-92e6-d54a3c969d42",
"texts": [
"Say this is a test"
],
"embeddings": [
[
-0.76953125,
-0.12731934,
-0.52246094,
-1.2714844,
........
........
]
],
"meta": {
"api_version": {
"version": "1"
}
}
}
}
]
}
]
}
```
103 changes: 103 additions & 0 deletions docs/remote_inference_blueprints/cohere_connector_rerank_blueprint.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
### Cohere connector blueprint example for rerank:

#### this blueprint is created from Cohere doc: https://docs.cohere.com/reference/rerank-1

```json
POST /_plugins/_ml/connectors/_create
{
"name": "<YOUR MODEL NAME>",
"description": "<YOUR MODEL DESCRIPTION>",
"version": "<YOUR MODEL VERSION>",
"protocol": "http",
"parameters": {
"endpoint": "api.cohere.ai",
"model": "rerank-english-v2.0",
"top_n": 3,
"return_documents": true
},
"credential": {
"api_key": "<PLEASE ADD YOUR Cohere API KEY HERE>"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://${parameters.endpoint}/v1/rerank",
"headers": {
"Authorization": "Bearer ${credential.api_key}"
},
"request_body": "{ \"model\": \"${parameters.model}\", \"query\" : \"${parameters.query}\", \"documents\" : ${parameters.documents}, \"top_n\" : ${parameters.top_n}, \"return_documents\" : ${parameters.return_documents} }"
}
]
}
```
#### Sample response
```json
{
"connector_id": "XU5UiokBpXT9icfOM0vt"
}
```


### Corresponding Predict request example:

```json
POST /_plugins/_ml/models/<ENTER MODEL ID HERE>/_predict
{
"parameters": {
"query": "What is the capital of the United States?",
"documents": [
"Carson City is the capital city of the American state of Nevada.",
"The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean. Its capital is Saipan.",
"Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district.",
"Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."
]
}
}
```

#### Sample response
```json
{
"inference_results": [
{
"output": [
{
"name": "response",
"dataAsMap": {
"id": "c536e12e-c7fe-414c-9a00-d1c5c6757b34",
"results": [
{
"document": {
"text": "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district."
},
"index": 2,
"relevance_score": 0.98005307
},
{
"document": {
"text": "Capital punishment (the death penalty) has existed in the United States since beforethe United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states."
},
"index": 3,
"relevance_score": 0.27904198
},
{
"document": {
"text": "Carson City is the capital city of the American state of Nevada."
},
"index": 0,
"relevance_score": 0.10194652
}
],
"meta": {
"api_version": {
"version": "1"
}
}
}
}
]
}
]
}
```
93 changes: 93 additions & 0 deletions docs/remote_inference_blueprints/open_ai_connector_chat_blueprint.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
### OpenAI connector blueprint example for chat:

#### this blueprint is created from OpenAI doc: https://platform.openai.com/docs/api-reference/chat
```json
POST /_plugins/_ml/connectors/_create
{
"name": "<YOUR MODEL NAME>",
"description": "<YOUR MODEL DESCRIPTION>",
"version": "<YOUR MODEL VERSION>",
"protocol": "http",
"parameters": {
"endpoint": "api.openai.com",
"model": "gpt-3.5-turbo"
},
"credential": {
"openAI_key": "<PLEASE ADD YOUR OPENAI API KEY HERE>"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://${parameters.endpoint}/v1/chat/completions",
"headers": {
"Authorization": "Bearer ${credential.openAI_key}"
},
"request_body": "{ \"model\": \"${parameters.model}\", \"messages\": ${parameters.messages} }"
}
]
}
```

#### Sample response
```json
{
"connector_id": "XU5UiokBpXT9icfOM0vt"
}
```

### Corresponding Predict request example:

```json
POST /_plugins/_ml/models/<ENTER MODEL ID HERE>/_predict
{
"parameters": {
"messages": [
{
"role": "system",
"content": "You are a helpful assistant."
},
{
"role": "user",
"content": "Hello!"
}
]
}
}
```

#### Sample response
```json
{
"inference_results": [
{
"output": [
{
"name": "response",
"dataAsMap": {
"id": "chatcmpl-7g0QJH6nuFW94l8tDkJzxm0ntaPNd",
"object": "chat.completion",
"created": 1690245759,
"model": "gpt-3.5-turbo-0613",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "Hello! How can I assist you today?"
},
"finish_reason": "stop"
}
],
"usage": {
"prompt_tokens": 19,
"completion_tokens": 9,
"total_tokens": 28
}
}
}
]
}
]
}
```
86 changes: 86 additions & 0 deletions docs/remote_inference_blueprints/open_ai_connector_completion_blueprint.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,86 @@
### OpenAI connector blueprint example for completion:

#### this blueprint is created from OpenAI doc: https://platform.openai.com/docs/api-reference/completions

```json
POST /_plugins/_ml/connectors/_create
{
"name": "<YOUR MODEL NAME>",
"description": "<YOUR MODEL DESCRIPTION>",
"version": "<YOUR MODEL VERSION>",
"protocol": "http",
"parameters": {
"endpoint": "api.openai.com",
"max_tokens": 7,
"temperature": 0,
"model": "text-davinci-003"
},
"credential": {
"openAI_key": "<PLEASE ADD YOUR OPENAI API KEY HERE>"
},
"actions": [
{
"action_type": "predict",
"method": "POST",
"url": "https://${parameters.endpoint}/v1/completions",
"headers": {
"Authorization": "Bearer ${credential.openAI_key}"
},
"request_body": "{ \"model\": \"${parameters.model}\", \"prompt\": \"${parameters.prompt}\", \"max_tokens\": ${parameters.max_tokens}, \"temperature\": ${parameters.temperature} }"
}
]
}
```

#### Sample response
```json
{
"connector_id": "XU5UiokBpXT9icfOM0vt"
}
```

### Corresponding Predict request example:

```json
POST /_plugins/_ml/models/<ENTER MODEL ID HERE>/_predict
{
"parameters": {
"prompt": ["Say this is a test"]
}
}
```

#### Sample response
```json
{
"inference_results": [
{
"output": [
{
"name": "response",
"dataAsMap": {
"id": "cmpl-7g0NPOJd8IvXTdhecdlR0VGfrLMWE",
"object": "text_completion",
"created": 1690245579,
"model": "text-davinci-003",
"choices": [
{
"text": """

This is indeed a test""",
"index": 0,
"finish_reason": "length"
}
],
"usage": {
"prompt_tokens": 5,
"completion_tokens": 7,
"total_tokens": 12
}
}
}
]
}
]
}
```
Loading

0 comments on commit 8b241a9

Please sign in to comment.