Skip to content

Commit d8284a7

Browse files
authored
Merge branch 'main' into saasbyocmessagingmetadata
2 parents 95a53dd + 581a323 commit d8284a7

File tree

19 files changed

+669
-1071
lines changed

19 files changed

+669
-1071
lines changed
Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
title: Exasol Hybrid Runner | Collate High-Performance Database
3+
description: Connect Exasol database to Collate using the Hybrid Runner. Deploy the ingestion agent in your environment for secure, private network metadata extraction.
4+
slug: /connectors/database/exasol/hybrid-runner
5+
sidebarTitle: Hybrid Runner
6+
---
7+
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
8+
import ExternalIngestionDeployment from '/snippets/connectors/external-ingestion-deployment.mdx'
9+
import TestConnection from '/snippets/connectors/test-connection.mdx'
10+
import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
11+
import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
12+
import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
13+
import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
14+
import Related from '/snippets/connectors/database/related.mdx'
15+
16+
<ConnectorDetailsHeader
17+
icon='/public/images/connectors/exasol.webp'
18+
name="Exasol"
19+
stage="PROD"
20+
availableFeatures={["Metadata"]}
21+
unavailableFeatures={["Query Usage", "Lineage", "Column-level Lineage", "Data Profiler", "Data Quality", "Owners", "dbt", "Tags", "Stored Procedures", "Sample Data", "Auto-Classification"]} />
22+
In this section, we provide guides and references for using the Exasol connector.
23+
Configure and schedule Exasol metadata from the OpenMetadata UI:
24+
- [Requirements](#requirements)
25+
- [Metadata Ingestion](#metadata-ingestion)
26+
- [Troubleshooting](/connectors/database/exasol/troubleshooting)
27+
<ExternalIngestionDeployment />
28+
<Card title="OpenMetadata 1.6.1 or later" href="/deployment" >
29+
To deploy OpenMetadata, check the Deployment guides.
30+
</Card>
31+
## Requirements
32+
The connector requires **Exasol version 7.1 or higher** to function correctly. Ensure your Exasol instance meets this minimum version requirement before proceeding.
33+
## Metadata Ingestion
34+
<MetadataIngestionUi connector={"Exasol"} selectServicePath={"/public/images/connectors/exasol/select-service.png"} addNewServicePath={"/public/images/connectors/exasol/add-new-service.png"} serviceConnectionPath={"/public/images/connectors/exasol/service-connection.png"} />
35+
# Connection Options
36+
<Steps>
37+
<Step title="Connection Options">
38+
**Connection Scheme**: Specifies the SQLAlchemy driver scheme options required to connect to Exasol.
39+
**Username**: The username used to connect to the Exasol database. Ensure that this user has sufficient privileges to read all the metadata from Exasol.
40+
**Password**: The password associated with the user connecting to Exasol.
41+
**Host and Port**: Defines the host and port of the Exasol instance. Provide this as a string in the format `hostname:port`. For example: `localhost:8563`
42+
- If running the OpenMetadata ingestion in a Docker container and your services are hosted on `localhost`, use `host.docker.internal:8563`.
43+
**SSL/TLS Settings**: Specifies the mode or settings for SSL/TLS validation during the connection. Available options:
44+
**validate-certificate (Default)**: Enables Transport Layer Security (TLS) and validates the server certificate using system certificate stores.
45+
**ignore-certificate**: Enables Transport Layer Security (TLS) but disables validation of the server certificate.
46+
<Tip>
47+
- This mode should not be used in production. It is useful for testing with self-signed certificates.
48+
</Tip>
49+
**disable-tls**: Disables Transport Layer Security (TLS). Data is sent in plain text (no encryption).
50+
[!WARNING]
51+
This mode is not recommended for production and should only be used in debugging scenarios.
52+
<Tip>
53+
Ensure that the appropriate TLS setting is chosen based on your security and deployment requirements.
54+
</Tip>
55+
</Step>
56+
<AdvancedConfiguration />
57+
<TestConnection />
58+
<ConfigureIngestion />
59+
<IngestionScheduleAndDeploy />
60+
</Steps>
61+
62+
<Tip>
63+
When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
64+
```
65+
password: secret:/my/database/password
66+
```
67+
This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
68+
For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
69+
</Tip>
70+
71+
## Troubleshooting
72+
73+
<Columns cols={2}>
74+
<Card
75+
title="Exasol Troubleshooting"
76+
href="/connectors/database/exasol/troubleshooting">
77+
Learn more about how to troubleshoot common Exasol connector issues and resolve configuration or ingestion errors.
78+
</Card>
79+
</Columns>
80+
81+
<Related />

connectors/database/exasol/index.mdx

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-
1111
import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
1212
import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
1313
import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
14+
import Related from '/snippets/connectors/database/related.mdx'
1415

1516
<ConnectorDetailsHeader
1617
icon='/public/images/connectors/exasol.webp'
@@ -57,3 +58,15 @@ Ensure that the appropriate TLS setting is chosen based on your security and dep
5758
<ConfigureIngestion />
5859
<IngestionScheduleAndDeploy />
5960
</Steps>
61+
62+
## Troubleshooting
63+
64+
<Columns cols={2}>
65+
<Card
66+
title="Exasol Troubleshooting"
67+
href="/connectors/database/exasol/troubleshooting">
68+
Learn more about how to troubleshoot common Exasol connector issues and resolve configuration or ingestion errors.
69+
</Card>
70+
</Columns>
71+
72+
<Related />

connectors/database/exasol/yaml.mdx

Lines changed: 0 additions & 126 deletions
This file was deleted.
Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
---
2+
title: GCS Datalake Hybrid Runner | Collate Cloud Storage Integration
3+
description: Connect your Google Cloud Storage data lake to Collate using the Hybrid Runner. Deploy the ingestion agent in your environment for secure, private network metadata extraction.
4+
slug: /connectors/database/gcs-datalake/hybrid-runner
5+
sidebarTitle: Hybrid Runner
6+
---
7+
import { ConnectorDetailsHeader } from '/snippets/components/ConnectorDetailsHeader/ConnectorDetailsHeader.jsx'
8+
import TestConnection from '/snippets/connectors/test-connection.mdx'
9+
import IngestionScheduleAndDeploy from '/snippets/connectors/ingestion-schedule-and-deploy.mdx'
10+
import ConfigureIngestion from '/snippets/connectors/database/configure-ingestion.mdx'
11+
import AdvancedConfiguration from '/snippets/connectors/database/advanced-configuration.mdx'
12+
import { MetadataIngestionUi } from '/snippets/components/MetadataIngestionUi.jsx'
13+
import Related from '/snippets/connectors/database/related.mdx'
14+
15+
<ConnectorDetailsHeader
16+
icon='/public/images/connectors/gcs.webp'
17+
name="GCS Datalake"
18+
stage="PROD"
19+
availableFeatures={["Metadata", "Data Profiler", "Data Quality", "Sample Data", "Auto-Classification"]}
20+
unavailableFeatures={["Query Usage", "Lineage", "Column-level Lineage", "Owners", "dbt", "Tags", "Stored Procedures"]} />
21+
In this section, we provide guides and references to use the GCS Datalake connector.
22+
Configure and schedule GCS Datalake metadata and profiler workflows from the OpenMetadata UI:
23+
- [Requirements](#requirements)
24+
- [Metadata Ingestion](#metadata-ingestion)
25+
- [Data Profiler](/how-to-guides/data-quality-observability/profiler/profiler-workflow)
26+
- [Data Quality](/how-to-guides/data-quality-observability/quality)
27+
- [Troubleshooting](/connectors/database/gcs-datalake/troubleshooting)
28+
## Requirements
29+
<Tip>
30+
The GCS Datalake connector supports extracting metadata from file types `JSON`, `CSV`, `TSV` & `Parquet`.
31+
</Tip>
32+
## Metadata Ingestion
33+
<MetadataIngestionUi connector={"GCS Datalake"} selectServicePath={"/public/images/connectors/datalake/select-service.png"} addNewServicePath={"/public/images/connectors/datalake/add-new-service.png"} serviceConnectionPath={"/public/images/connectors/datalake/service-connection.png"} />
34+
## Connection Details
35+
<Steps>
36+
<Step title="Connection Details for GCS">
37+
- **Bucket Name**: A bucket name in DataLake is a unique identifier used to organize and store data objects.
38+
It's similar to a folder name, but it's used for object storage rather than file storage.
39+
- **Prefix**: The prefix of a data source in datalake refers to the first part of the data path that identifies the source or origin of the data. It's used to organize and categorize data within the datalake, and can help users easily locate and access the data they need.
40+
**GCS Credentials**
41+
We support two ways of authenticating to GCS:
42+
1. Passing the raw credential values provided by BigQuery. This requires us to provide the following information, all provided by BigQuery:
43+
1. Credentials type, e.g. `service_account`.
44+
2. Project ID
45+
3. Private Key ID
46+
4. Private Key
47+
5. Client Email
48+
6. Client ID
49+
7. Auth URI, [https://accounts.google.com/o/oauth2/auth](https://accounts.google.com/o/oauth2/auth) by default
50+
8. Token URI, **https://oauth2.googleapis.com/token** by default
51+
9. Authentication Provider X509 Certificate URL, [https://www.googleapis.com/oauth2/v1/certs](https://www.googleapis.com/oauth2/v1/certs) by default
52+
10. Client X509 Certificate URL
53+
</Step>
54+
<AdvancedConfiguration />
55+
<TestConnection />
56+
<ConfigureIngestion />
57+
<IngestionScheduleAndDeploy />
58+
</Steps>
59+
60+
<Tip>
61+
When using a **Hybrid Ingestion Runner**, any sensitive credential fields—such as passwords, API keys, or private keys—must reference secrets using the following format:
62+
```
63+
password: secret:/my/database/password
64+
```
65+
This applies **only to fields marked as secrets** in the connection form (these typically mask input and show a visibility toggle icon).
66+
For a complete guide on managing secrets in hybrid setups, see the [Hybrid Ingestion Runner Secret Management Guide](https://docs.getcollate.io/getting-started/day-1/hybrid-saas/hybrid-ingestion-runner#3.-manage-secrets-securely).
67+
</Tip>
68+
69+
## Troubleshooting
70+
71+
<Columns cols={2}>
72+
<Card
73+
title="GCS Datalake Troubleshooting"
74+
href="/connectors/database/gcs-datalake/troubleshooting">
75+
Learn more about how to troubleshoot common GCS Datalake connector issues and resolve configuration or ingestion errors.
76+
</Card>
77+
</Columns>
78+
79+
<Related />

connectors/database/gcs-datalake/index.mdx

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -56,4 +56,15 @@ We support two ways of authenticating to GCS:
5656
<ConfigureIngestion />
5757
<IngestionScheduleAndDeploy />
5858
</Steps>
59+
60+
## Troubleshooting
61+
62+
<Columns cols={2}>
63+
<Card
64+
title="GCS Datalake Troubleshooting"
65+
href="/connectors/database/gcs-datalake/troubleshooting">
66+
Learn more about how to troubleshoot common GCS Datalake connector issues and resolve configuration or ingestion errors.
67+
</Card>
68+
</Columns>
69+
5970
<Related />

0 commit comments

Comments
 (0)