diff --git a/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.mdx b/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.mdx index e6bf6545e8..b70440ba02 100644 --- a/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.mdx +++ b/serverless/pages/ingest-your-data-ingest-data-through-integrations-connector-client.mdx @@ -24,35 +24,35 @@ Connector clients are available for the following third-party data sources: {/* TODO: Update links if these references move*/} -- [Azure Blob Storage](https://www.elastic.co/guide/en/enterprise-search/master/connectors-azure-blob.html) -- [Box](https://www.elastic.co/guide/en/enterprise-search/master/connectors-box.html) -- [Confluence](https://www.elastic.co/guide/en/enterprise-search/master/connectors-confluence.html) -- [Dropbox](https://www.elastic.co/guide/en/enterprise-search/master/connectors-dropbox.html) -- [GitHub](https://www.elastic.co/guide/en/enterprise-search/master/connectors-github.html) -- [Gmail](https://www.elastic.co/guide/en/enterprise-search/master/connectors-gmail.html) -- [Google Cloud Storage](https://www.elastic.co/guide/en/enterprise-search/master/connectors-google-cloud.html) -- [Google Drive](https://www.elastic.co/guide/en/enterprise-search/master/connectors-google-drive.html) -- [GraphQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-graphql.html) -- [Jira](https://www.elastic.co/guide/en/enterprise-search/master/connectors-jira.html) -- [MicrosoftSQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-ms-sql.html) -- [MongoDB](https://www.elastic.co/guide/en/enterprise-search/master/connectors-mongodb.html) -- [MySQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-mysql.html) -- [Network drive](https://www.elastic.co/guide/en/enterprise-search/master/connectors-network-drive.html) -- [Notion](https://www.elastic.co/guide/en/enterprise-search/master/connectors-notion.html) -- [OneDrive](https://www.elastic.co/guide/en/enterprise-search/master/connectors-onedrive.html) -- [OpenText Documentum](https://www.elastic.co/guide/en/enterprise-search/master/connectors-opentext.html) -- [Oracle](https://www.elastic.co/guide/en/enterprise-search/master/connectors-oracle.html) -- [Outlook](https://www.elastic.co/guide/en/enterprise-search/master/connectors-outlook.html) -- [PostgreSQL](https://www.elastic.co/guide/en/enterprise-search/master/connectors-postgresql.html) -- [Redis](https://www.elastic.co/guide/en/enterprise-search/master/connectors-redis.html) -- [S3](https://www.elastic.co/guide/en/enterprise-search/master/connectors-s3.html) -- [Salesforce](https://www.elastic.co/guide/en/enterprise-search/master/connectors-salesforce.html) -- [ServiceNow](https://www.elastic.co/guide/en/enterprise-search/master/connectors-servicenow.html) -- [SharePoint Online](https://www.elastic.co/guide/en/enterprise-search/master/connectors-sharepoint-online.html) -- [SharePoint Server](https://www.elastic.co/guide/en/enterprise-search/master/connectors-sharepoint.html) -- [Slack](https://www.elastic.co/guide/en/enterprise-search/master/connectors-slack.html) -- [Teams](https://www.elastic.co/guide/en/enterprise-search/master/connectors-teams.html) -- [Zoom](https://www.elastic.co/guide/en/enterprise-search/master/connectors-zoom.html) +- [Azure Blob Storage](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-azure-blob.html) +- [Box](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-box.html) +- [Confluence](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-confluence.html) +- [Dropbox](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-dropbox.html) +- [GitHub](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-github.html) +- [Gmail](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-gmail.html) +- [Google Cloud Storage](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-google-cloud.html) +- [Google Drive](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-google-drive.html) +- [GraphQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-graphql.html) +- [Jira](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-jira.html) +- [MicrosoftSQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-ms-sql.html) +- [MongoDB](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-mongodb.html) +- [MySQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-mysql.html) +- [Network drive](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-network-drive.html) +- [Notion](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-notion.html) +- [OneDrive](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-onedrive.html) +- [OpenText Documentum](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-opentext.html) +- [Oracle](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-oracle.html) +- [Outlook](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-outlook.html) +- [PostgreSQL](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-postgresql.html) +- [Redis](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-redis.html) +- [S3](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-s3.html) +- [Salesforce](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-salesforce.html) +- [ServiceNow](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-servicenow.html) +- [SharePoint Online](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-sharepoint-online.html) +- [SharePoint Server](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-sharepoint.html) +- [Slack](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-slack.html) +- [Teams](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-teams.html) +- [Zoom](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors-zoom.html) ## Overview @@ -79,10 +79,9 @@ Note that each data source will have specific prerequisites you'll need to meet For example, certain data sources may require you to create an OAuth application, or create a service account. You'll need to check the [individual connector documentation](#available-connectors) for these details. -## Initial setup in UI +## Step 1: Initial setup in UI In your project's UI, go to **((es)) → Connectors**. -Read the high-level instructions under **Create a connector** to make sure you understand the process. Follow these steps: 1. Select **Create a connector**. @@ -90,65 +89,29 @@ Follow these steps: 3. Add a name and optional description to identify the connector. 4. Copy the `connector_id`, `service_type`, and `elasticsearch.host` values printed to the screen. You'll need to update these values in your [`config.yml`](https://github.com/elastic/connectors/blob/main/config.yml) file. -5. Make a note of your **((es)) endpoint** and **API key** values. You can create a new API key under **Security → API keys**. -6. Run the connector code using one of the [options](#deploy-self-managed-connector) documented on this page. +5. Navigate to **Elasticsearch → Home**, and make a note of your **((es)) endpoint** and **API key** values. You can create a new API key by clicking on **New** in the **API key** section. +6. Run the connector code either from source or with Docker, following the instructions below. -## Deploy your self-managed connector +## Step 2: Deploy your self-managed connector -To use connector clients, you must deploy the connector service so your connector can talk to your ((es)) instance. +To use connector clients, you must deploy the connector service so your connector can talk to your ((es)) instance. The source code is hosted in the `elastic/connectors` repository. You have two deployment options: +- Run with [Docker](#run-with-docker) (recommended) - Run from [source](#run-from-source) -- Run with [Docker](#run-with-docker) You'll need the following values handy to update your `config.yml` file: - - `elasticsearch.host`: Your ((es)) endpoint. Find this in the **Get started** step in the UI. - - `elasticsearch.api_key`: Your ((es)) API key. You can create API keys under **Security → API keys**. Once you connector is running, you'll be able to create a new API key that is limited to only access the connector's index. + - `elasticsearch.host`: Your ((es)) endpoint. Printed to the screen when you create a new connector. + - `elasticsearch.api_key`: Your ((es)) API key. You can create API keys by navigating to **Home**, and clicking **New** in the **API key** section. Once your connector is running, you'll be able to create a new API key that is limited to only access the connector's index. - `connector_id`: Unique id for your connector. Printed to the screen when you create a new connector. - `service_type`: Original data source type. Printed to the screen when you create a new connector. -### Run from source - -This is a good option if you're comfortable working with Python and want to iterate quickly locally. -First, you need to clone the `elastic/connectors` repository. - -Follow these steps: - -- Clone or fork the repository locally with the following command: - ```shell - git clone https://github.com/elastic/connectors - ``` -- Create a `config.yml` configuration file by copying this [example](https://github.com/elastic/connectors/blob/main/config.yml.example). -- Replace the values for `host` (your ((es)) endpoint), `api_key`, `connector_id`, and `service_type`. - - - Learn more about the `config.yml` file in the [repo docs](https://github.com/elastic/connectors/blob/main/docs/CONFIG.md). - - -**Run the connector service** - -Once you've configured the connector code, you can run the connector service. - -In your terminal or IDE: - -- `cd` into the root of your `elastic/connectors` clone/fork. -- Run the following commands to compile and run the connector service: - - ```shell - make install - make run - ``` - -The connector service should now be running in your terminal. Your project's UI will let you know that the connector has successfully connected to your ((es)) instance. - -Here we're working locally. In a production setup, you'll deploy the connector service to your own infrastructure. - ### Run with Docker -You can also deploy connector clients using Docker. +You can deploy connector clients using Docker. Follow these instructions. **Step 1: Download sample configuration file** @@ -163,11 +126,12 @@ Change the `--output` argument value to the path where you want to save the conf **Step 2: Update the configuration file for your self-managed connector** -Update the following settings to match your environment: +- Update the following settings to match your environment: * `elasticsearch.host` * `elasticsearch.api_key` -* `connectors` +* `connector id` +* `service_type` Your configuration file should look like this: @@ -202,7 +166,61 @@ Find all available Docker images in the [official Elastic Docker registry](https Each individual connector client reference contain instructions for deploying specific connectors using Docker. -## Enter data source details in UI +### Run from source + +Running from source requires cloning the repository and running the code locally. +Use this approach if you're actively customizing connectors. + +Follow these steps: + +- Clone or fork the repository locally with the following command: + ```shell + git clone https://github.com/elastic/connectors + ``` +- Open the `config.yml.example` file in the `connectors` repository and rename it to `config.yml`. +- Update the following settings to match your environment: + +* `elasticsearch.host` +* `elasticsearch.api_key` +* `connector id` +* `service_type` + +Your configuration file should look like this: + +```yaml +elasticsearch.host: +elasticsearch.api_key: + +connectors: + - + connector_id: + service_type: # sharepoint_online (example) + api_key: # Optional. If not provided, the connector will use the elasticsearch.api_key instead +``` + + + Learn more about the `config.yml` file in the [repo docs](https://github.com/elastic/connectors/blob/main/docs/CONFIG.md). + + +**Run the connector service** + +Once you've configured the connector code, you can run the connector service. + +In your terminal or IDE: + +- `cd` into the root of your `elastic/connectors` clone/fork. +- Run the following commands to compile and run the connector service: + + ```shell + make install + make run + ``` + +The connector service should now be running in your terminal. If the connection to your ((es)) instance was successful, the **Configure your connector** step will be activated in the project's UI. + +Here we're working locally. In a production setup, you'll deploy the connector service to your own infrastructure. + +## Step 3: Enter data source details in UI Once the connector service is running, it's time to head back to the UI to finalize the connector configuration. You should now see the **Configure your connector** step in your project's UI. @@ -217,7 +235,7 @@ For example, the Sharepoint Online connector requires the following details abou - **Secret value** - **Comma-separated list of tables** -## Connect to an index +## Step 4: Connect to an index Once you've entered the data source details, you need to connect to an index. This is the final step in your project's UI, before you can run a sync. @@ -241,7 +259,7 @@ Learn [how syncing works](https://github.com/elastic/connectors/blob/main/docs/D ## Learn more -- Read the main [Elastic connectors documentation](https://www.elastic.co/guide/en/enterprise-search/current/connectors.html) +- Read the main [Elastic connectors documentation](https://www.elastic.co/guide/en/elasticsearch/reference/master/es-connectors.html) - The [Elastic connector framework](https://github.com/elastic/connectors/tree/main#connector-framework) enables you to: - Customize existing connector clients. - Build your own connector clients.