Skip to content

Commit

Permalink
Updates instructions for self-managed connectors
Browse files Browse the repository at this point in the history
  • Loading branch information
kosabogi committed Oct 17, 2024
1 parent 84f3199 commit 14e257d
Showing 1 changed file with 59 additions and 41 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -24,35 +24,35 @@ Connector clients are available for the following third-party data sources:
<DocAccordion buttonContent="Click to expand">

{/* TODO: Update links if these references move*/}
- [Azure Blob Storage](https://www.elastic.co/guide/en/enterprise-search/master/connectors-azure-blob.html)
- [Box](https://www.elastic.co/guide/en/enterprise-search/master/connectors-box.html)
- [Confluence](https://www.elastic.co/guide/en/enterprise-search/master/connectors-confluence.html)
- [Dropbox](https://www.elastic.co/guide/en/enterprise-search/master/connectors-dropbox.html)
- [GitHub](https://www.elastic.co/guide/en/enterprise-search/master/connectors-github.html)
- [Gmail](https://www.elastic.co/guide/en/enterprise-search/master/connectors-gmail.html)
- [Google Cloud Storage](https://www.elastic.co/guide/en/enterprise-search/master/connectors-google-cloud.html)
- [Google Drive](https://www.elastic.co/guide/en/enterprise-search/master/connectors-google-drive.html)
- [GraphQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-graphql.html)
- [Jira](https://www.elastic.co/guide/en/enterprise-search/master/connectors-jira.html)
- [MicrosoftSQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-ms-sql.html)
- [MongoDB](https://www.elastic.co/guide/en/enterprise-search/master/connectors-mongodb.html)
- [MySQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-mysql.html)
- [Network drive](https://www.elastic.co/guide/en/enterprise-search/master/connectors-network-drive.html)
- [Notion](https://www.elastic.co/guide/en/enterprise-search/master/connectors-notion.html)
- [OneDrive](https://www.elastic.co/guide/en/enterprise-search/master/connectors-onedrive.html)
- [OpenText Documentum](https://www.elastic.co/guide/en/enterprise-search/master/connectors-opentext.html)
- [Oracle](https://www.elastic.co/guide/en/enterprise-search/master/connectors-oracle.html)
- [Outlook](https://www.elastic.co/guide/en/enterprise-search/master/connectors-outlook.html)
- [PostgreSQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-postgresql.html)
- [Redis](https://www.elastic.co/guide/en/enterprise-search/master/connectors-redis.html)
- [S3](https://www.elastic.co/guide/en/enterprise-search/master/connectors-s3.html)
- [Salesforce](https://www.elastic.co/guide/en/enterprise-search/master/connectors-salesforce.html)
- [ServiceNow](https://www.elastic.co/guide/en/enterprise-search/master/connectors-servicenow.html)
- [SharePoint Online](https://www.elastic.co/guide/en/enterprise-search/master/connectors-sharepoint-online.html)
- [SharePoint Server](https://www.elastic.co/guide/en/enterprise-search/master/connectors-sharepoint.html)
- [Slack](https://www.elastic.co/guide/en/enterprise-search/master/connectors-slack.html)
- [Teams](https://www.elastic.co/guide/en/enterprise-search/master/connectors-teams.html)
- [Zoom](https://www.elastic.co/guide/en/enterprise-search/master/connectors-zoom.html)
- [Azure Blob Storage](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-azure-blob.html)
- [Box](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-box.html)
- [Confluence](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-confluence.html)
- [Dropbox](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-dropbox.html)
- [GitHub](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-github.html)
- [Gmail](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-gmail.html)
- [Google Cloud Storage](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-google-cloud.html)
- [Google Drive](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-google-drive.html)
- [GraphQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-graphql.html)
- [Jira](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-jira.html)
- [MicrosoftSQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-ms-sql.html)
- [MongoDB](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-mongodb.html)
- [MySQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-mysql.html)
- [Network drive](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-network-drive.html)
- [Notion](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-notion.html)
- [OneDrive](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-onedrive.html)
- [OpenText Documentum](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-opentext.html)
- [Oracle](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-oracle.html)
- [Outlook](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-outlook.html)
- [PostgreSQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-postgresql.html)
- [Redis](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-redis.html)
- [S3](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-s3.html)
- [Salesforce](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-salesforce.html)
- [ServiceNow](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-servicenow.html)
- [SharePoint Online](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-sharepoint-online.html)
- [SharePoint Server](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-sharepoint.html)
- [Slack](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-slack.html)
- [Teams](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-teams.html)
- [Zoom](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-zoom.html)
</DocAccordion>

## Overview
Expand Down Expand Up @@ -82,20 +82,19 @@ You'll need to check the [individual connector documentation](#available-connect
## Initial setup in UI

In your project's UI, go to **((es)) → Connectors**.
Read the high-level instructions under **Create a connector** to make sure you understand the process.
Follow these steps:

1. Select **Create a connector**.
2. Choose a third-party service from the list of connector types.
3. Add a name and optional description to identify the connector.
4. Copy the `connector_id`, `service_type`, and `elasticsearch.host` values printed to the screen.
You'll need to update these values in your [`config.yml`](https://github.com/elastic/connectors/blob/main/config.yml) file.
5. Make a note of your **((es)) endpoint** and **API key** values. You can create a new API key under **Security → API keys**.
6. Run the connector code using one of the [options](#deploy-self-managed-connector) documented on this page.
5. Navigate to **Elasticsearch → Home**, and make a note of your **((es)) endpoint** and **API key** values. You can create a new API key by clicking on **New** in the **API key** section.
6. Run the connector code either from source or with Docker, following the instructions below.

## Deploy your self-managed connector

To use connector clients, you must deploy the connector service so your connector can talk to your ((es)) instance.
To use connector clients, you must deploy the connector service so your connector can talk to your ((es)) instance.
The source code is hosted in the `elastic/connectors` repository.

You have two deployment options:
Expand All @@ -104,8 +103,8 @@ You have two deployment options:

<DocCallOut title="">
You'll need the following values handy to update your `config.yml` file:
- `elasticsearch.host`: Your ((es)) endpoint. Find this in the **Get started** step in the UI.
- `elasticsearch.api_key`: Your ((es)) API key. You can create API keys under **Security → API keys**. Once you connector is running, you'll be able to create a new API key that is limited to only access the connector's index.
- `elasticsearch.host`: Your ((es)) endpoint. Printed to the screen when you create a new connector.
- `elasticsearch.api_key`: Your ((es)) API key. You can create API keys by navigating to **Home**, and clicking **New** in the **API key** section. Once your connector is running, you'll be able to create a new API key that is limited to only access the connector's index.
- `connector_id`: Unique id for your connector. Printed to the screen when you create a new connector.
- `service_type`: Original data source type. Printed to the screen when you create a new connector.
</DocCallOut>
Expand All @@ -121,8 +120,26 @@ Follow these steps:
```shell
git clone https://github.com/elastic/connectors
```
- Create a `config.yml` configuration file by copying this [example](https://github.com/elastic/connectors/blob/main/config.yml.example).
- Replace the values for `host` (your ((es)) endpoint), `api_key`, `connector_id`, and `service_type`.
- Open the `config.yml.example` file in the `connectors` repository and rename it to `config.yml`.
- Update the following settings to match your environment:

* `elasticsearch.host`
* `elasticsearch.api_key`
* `connector id`
* `service_type`

Your configuration file should look like this:

```yaml
elasticsearch.host: <ELASTICSEARCH_ENDPOINT>
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>

connectors:
-
connector_id: <CONNECTOR_ID_FROM_UI>
service_type: <SERVICE-NAME> # sharepoint_online (example)
api_key: <CONNECTOR_API_KEY> # Optional. If not provided, the connector will use the elasticsearch.api_key instead
```
<DocCallOut title="Tip">
Learn more about the `config.yml` file in the [repo docs](https://github.com/elastic/connectors/blob/main/docs/CONFIG.md).
Expand All @@ -142,7 +159,7 @@ In your terminal or IDE:
make run
```

The connector service should now be running in your terminal. Your project's UI will let you know that the connector has successfully connected to your ((es)) instance.
The connector service should now be running in your terminal. If the connection to your ((es)) instance was successful, the **Configure your connector** step will be activated in the project's UI.

Here we're working locally. In a production setup, you'll deploy the connector service to your own infrastructure.

Expand All @@ -163,11 +180,12 @@ Change the `--output` argument value to the path where you want to save the conf

**Step 2: Update the configuration file for your self-managed connector**

Update the following settings to match your environment:
- Update the following settings to match your environment:

* `elasticsearch.host`
* `elasticsearch.api_key`
* `connectors`
* `connector id`
* `service_type`

Your configuration file should look like this:

Expand Down Expand Up @@ -241,7 +259,7 @@ Learn [how syncing works](https://github.com/elastic/connectors/blob/main/docs/D

## Learn more

- Read the main [Elastic connectors documentation](https://www.elastic.co/guide/en/enterprise-search/current/connectors.html)
- Read the main [Elastic connectors documentation](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors.html)
- The [Elastic connector framework](https://github.com/elastic/connectors/tree/main#connector-framework) enables you to:
- Customize existing connector clients.
- Build your own connector clients.

0 comments on commit 14e257d

Please sign in to comment.