Skip to content

Commit cb8e3ba

Browse files
ApmatsApmats
committed
Adding documentation for the gitlab content connector
1 parent e46f211 commit cb8e3ba

File tree

1 file changed

+326
-0
lines changed

1 file changed

+326
-0
lines changed

‎docs/reference/search-connectors/es-connectors-gitlab.md‎

Lines changed: 326 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,326 @@
1+
---
2+
navigation_title: "GitLab"
3+
mapped_pages:
4+
- https://www.elastic.co/guide/en/elasticsearch/reference/current/es-connectors-gitlab.html
5+
---
6+
7+
# Elastic GitLab connector reference [es-connectors-gitlab]
8+
9+
10+
The *Elastic GitLab connector* is a [connector](/reference/search-connectors/index.md) for [GitLab](https://www.gitlab.com). This connector is written in Python using the [Elastic connector framework](https://github.com/elastic/connectors/tree/main).
11+
12+
View the [**source code** for this connector](https://github.com/elastic/connectors/tree/main/app/connectors_service/connectors/sources/gitlab) (branch *main*, compatible with Elastic *9.3*).
13+
14+
::::{important}
15+
As of Elastic 9.0, managed connectors on Elastic Cloud Hosted are no longer available. All connectors must be [self-managed](/reference/search-connectors/self-managed-connectors.md).
16+
::::
17+
18+
19+
## **Self-managed connector** [es-connectors-gitlab-connector-client-reference]
20+
21+
### Availability and prerequisites [es-connectors-gitlab-client-availability-prerequisites]
22+
23+
This connector is available as a self-managed connector.
24+
25+
This self-managed connector is compatible with Elastic versions **9.3.0+**.
26+
27+
To use this connector, satisfy all [self-managed connector requirements](/reference/search-connectors/self-managed-connectors.md).
28+
29+
30+
### Create a GitLab connector [es-connectors-gitlab-create-connector-client]
31+
32+
33+
#### Use the UI [es-connectors-gitlab-client-create-use-the-ui]
34+
35+
To create a new GitLab connector:
36+
37+
1. In the Kibana UI, navigate to the **Search → Content → Connectors** page from the main menu, or use the [global search field](docs-content://explore-analyze/query-filter/filtering.md#_finding_your_apps_and_objects).
38+
2. Follow the instructions to create a new **GitLab** self-managed connector.
39+
40+
41+
#### Use the API [es-connectors-gitlab-client-create-use-the-api]
42+
43+
You can use the {{es}} [Create connector API](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-connector) to create a new self-managed GitLab self-managed connector.
44+
45+
For example:
46+
47+
```console
48+
PUT _connector/my-gitlab-connector
49+
{
50+
"index_name": "my-elasticsearch-index",
51+
"name": "Content synced from GitLab",
52+
"service_type": "gitlab"
53+
}
54+
```
55+
% TEST[skip:can't test in isolation]
56+
57+
:::::{dropdown} You'll also need to create an API key for the connector to use.
58+
::::{note}
59+
The user needs the cluster privileges `manage_api_key`, `manage_connector` and `write_connector_secrets` to generate API keys programmatically.
60+
61+
::::
62+
63+
64+
To create an API key for the connector:
65+
66+
1. Run the following command, replacing values where indicated. Note the `encoded` return values from the response:
67+
68+
```console
69+
POST /_security/api_key
70+
{
71+
"name": "connector_name-connector-api-key",
72+
"role_descriptors": {
73+
"connector_name-connector-role": {
74+
"cluster": [
75+
"monitor",
76+
"manage_connector"
77+
],
78+
"indices": [
79+
{
80+
"names": [
81+
"index_name",
82+
".search-acl-filter-index_name",
83+
".elastic-connectors*"
84+
],
85+
"privileges": [
86+
"all"
87+
],
88+
"allow_restricted_indices": false
89+
}
90+
]
91+
}
92+
}
93+
}
94+
```
95+
96+
2. Update your `config.yml` file with the API key `encoded` value.
97+
98+
:::::
99+
100+
101+
Refer to the [{{es}} API documentation](https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-connector) for details of all available Connector APIs.
102+
103+
104+
### Usage [es-connectors-gitlab-client-usage]
105+
106+
To use this connector as a **self-managed connector**, see [*Self-managed connectors*](/reference/search-connectors/self-managed-connectors.md) For additional usage operations, see [*Connectors UI in {{kib}}*](/reference/search-connectors/connectors-ui-in-kibana.md).
107+
108+
109+
#### GitLab personal access token [es-connectors-gitlab-client-personal-access-token]
110+
111+
Configure a GitLab personal access token to fetch data from GitLab.
112+
113+
Follow these steps to generate a GitLab personal access token:
114+
115+
* Go to **User Settings → Access Tokens** (or for project tokens: **Settings → Access Tokens → Project Access Tokens**).
116+
* Click `Add new token`.
117+
* Enter a token name and optional expiration date.
118+
* Select the following scopes:
119+
* `api` - Required for GraphQL API access (note: the connector only performs read-only operations)
120+
* `read_api` - Grants read access to REST API endpoints
121+
* `read_repository` - Grants read access to repository files
122+
123+
* Click `Create personal access token` and copy the token.
124+
125+
126+
### Compatibility [es-connectors-gitlab-client-compatibility]
127+
128+
Both GitLab Cloud (gitlab.com) and GitLab Self-Managed are supported.
129+
130+
131+
### Configuration [es-connectors-gitlab-client-configuration]
132+
133+
The following configuration fields are required:
134+
135+
`token`
136+
: GitLab personal access token to authenticate with the GitLab instance. The token must have `api`, `read_api`, and `read_repository` scopes.
137+
138+
`projects`
139+
: List of project paths to sync (e.g., `group/project`, `username/project`). Use `*` or leave empty (`[]`) to sync all projects where the token's user is a member.
140+
141+
Default value is `[]` (empty list, which syncs all projects).
142+
143+
Examples:
144+
145+
* `elastic/elasticsearch`, `elastic/kibana`
146+
* `*`
147+
* `[]` (syncs all projects)
148+
149+
150+
::::{tip}
151+
**Project path format**
152+
153+
Projects should be specified using their full path including the namespace (group or username).
154+
155+
In the examples provided here:
156+
157+
* `elastic/elasticsearch` syncs the Elasticsearch project from the elastic group
158+
* `elastic/kibana` syncs the Kibana project from the elastic group
159+
160+
When using `*`, all projects where the authenticated user is a member will be synced.
161+
162+
::::
163+
164+
165+
`use_text_extraction_service`
166+
: Requires a separate deployment of the [Elastic Text Extraction Service](/reference/search-connectors/es-connectors-content-extraction.md#es-connectors-content-extraction-local). Requires that pipeline settings disable text extraction. Default value is `False`.
167+
168+
169+
170+
171+
### Deployment using Docker [es-connectors-gitlab-client-docker]
172+
173+
You can deploy the GitLab connector as a self-managed connector using Docker. Follow these instructions.
174+
175+
::::{dropdown} Step 1: Download sample configuration file
176+
Download the sample configuration file. You can either download it manually or run the following command:
177+
178+
```sh
179+
curl https://raw.githubusercontent.com/elastic/connectors/main/app/connectors_service/config.yml.example --output ~/connectors-config/config.yml
180+
```
181+
% NOTCONSOLE
182+
183+
Remember to update the `--output` argument value if your directory name is different, or you want to use a different config file name.
184+
185+
::::
186+
187+
188+
::::{dropdown} Step 2: Update the configuration file for your self-managed connector
189+
Update the configuration file with the following settings to match your environment:
190+
191+
* `elasticsearch.host`
192+
* `elasticsearch.api_key`
193+
* `connectors`
194+
195+
If you're running the connector service against a Dockerized version of Elasticsearch and Kibana, your config file will look like this:
196+
197+
```yaml
198+
# When connecting to your cloud deployment you should edit the host value
199+
elasticsearch.host: http://host.docker.internal:9200
200+
elasticsearch.api_key: <ELASTICSEARCH_API_KEY>
201+
202+
connectors:
203+
-
204+
connector_id: <CONNECTOR_ID_FROM_KIBANA>
205+
service_type: gitlab
206+
api_key: <CONNECTOR_API_KEY_FROM_KIBANA> # Optional. If not provided, the connector will use the elasticsearch.api_key instead
207+
```
208+
209+
Using the `elasticsearch.api_key` is the recommended authentication method. However, you can also use `elasticsearch.username` and `elasticsearch.password` to authenticate with your Elasticsearch instance.
210+
211+
Note: You can change other default configurations by simply uncommenting specific settings in the configuration file and modifying their values.
212+
213+
::::
214+
215+
216+
::::{dropdown} Step 3: Run the Docker image
217+
Run the Docker image with the Connector Service using the following command:
218+
219+
```sh subs=true
220+
docker run \
221+
-v ~/connectors-config:/config \
222+
--network "elastic" \
223+
--tty \
224+
--rm \
225+
docker.elastic.co/integrations/elastic-connectors:{{version.stack}} \
226+
/app/bin/elastic-ingest \
227+
-c /config/config.yml
228+
```
229+
230+
::::
231+
232+
233+
Refer to [`DOCKER.md`](https://github.com/elastic/connectors/tree/main/docs/DOCKER.md) in the `elastic/connectors` repo for more details.
234+
235+
Find all available Docker images in the [official registry](https://www.docker.elastic.co/r/integrations/elastic-connectors).
236+
237+
::::{tip}
238+
We also have a quickstart self-managed option using Docker Compose, so you can spin up all required services at once: Elasticsearch, Kibana, and the connectors service. Refer to this [README](https://github.com/elastic/connectors/tree/main/scripts/stack#readme) in the `elastic/connectors` repo for more information.
239+
240+
::::
241+
242+
243+
244+
### Documents and syncs [es-connectors-gitlab-client-documents-syncs]
245+
246+
The connector syncs the following objects and entities:
247+
248+
* **Projects**
249+
* **Issues** (using Work Items API)
250+
* **Merge Requests**
251+
* **Epics** (using Work Items API, group-level, requires Premium/Ultimate tier)
252+
* **Releases** (project-level version releases with changelogs)
253+
* **README Files** (.md, .rst, .txt)
254+
255+
Only the following file extensions are ingested for README files:
256+
257+
* `.md`
258+
* `.rst`
259+
* `.txt`
260+
261+
::::{note}
262+
* Content of files bigger than 10 MB won't be extracted.
263+
* The connector uses GitLab's Work Items API for Issues and Epics, which provides a unified interface for different work item types.
264+
* Epics are only available for Premium/Ultimate GitLab tiers and are synced at the group level.
265+
* **Epic syncing behavior**: Epics are fetched only for groups that contain synced projects.
266+
* Permissions are not synced. **All documents** indexed to an Elastic deployment will be visible to **all users with access** to that Elasticsearch Index.
267+
268+
::::
269+
270+
271+
272+
#### Sync types [es-connectors-gitlab-client-sync-types]
273+
274+
[Full syncs](/reference/search-connectors/content-syncs.md#es-connectors-sync-types-full) are supported by default for all connectors.
275+
276+
This connector does not currently support [incremental syncs](/reference/search-connectors/content-syncs.md#es-connectors-sync-types-incremental).
277+
278+
279+
### Sync rules [es-connectors-gitlab-client-sync-rules]
280+
281+
*Basic* sync rules are identical for all connectors and are available by default. For more information read [Types of sync rule](/reference/search-connectors/es-sync-rules.md#es-sync-rules-types).
282+
283+
284+
#### Advanced sync rules [es-connectors-gitlab-client-sync-rules-advanced]
285+
286+
Advanced sync rules are not currently supported for this connector. This feature may be added in future releases.
287+
288+
289+
### Content Extraction [es-connectors-gitlab-client-content-extraction]
290+
291+
See [Content extraction](/reference/search-connectors/es-connectors-content-extraction.md).
292+
293+
294+
### Self-managed connector operations [es-connectors-gitlab-client-connector-client-operations]
295+
296+
297+
### End-to-end testing [es-connectors-gitlab-client-testing]
298+
299+
The connector framework enables operators to run functional tests against a real data source. Refer to [Connector testing](/reference/search-connectors/self-managed-connectors.md#es-build-connector-testing) for more details.
300+
301+
To perform E2E testing for the GitLab connector, run the following command:
302+
303+
```shell
304+
$ make ftest NAME=gitlab
305+
```
306+
307+
For faster tests, add the `DATA_SIZE=small` flag:
308+
309+
```shell
310+
make ftest NAME=gitlab DATA_SIZE=small
311+
```
312+
313+
314+
### Known issues [es-connectors-gitlab-client-known-issues]
315+
316+
There are currently no known issues for this connector. Refer to [Known issues](/release-notes/known-issues.md) for a list of known issues for all connectors.
317+
318+
319+
### Troubleshooting [es-connectors-gitlab-client-troubleshooting]
320+
321+
See [Troubleshooting](/reference/search-connectors/es-connectors-troubleshooting.md).
322+
323+
324+
### Security [es-connectors-gitlab-client-security]
325+
326+
See [Security](/reference/search-connectors/es-connectors-security.md).

0 commit comments

Comments
 (0)