Skip to content

Commit 4539e88

Browse files
author
Kostiantyn Krykun
committed
docs: Add Tekton Results to docs
1 parent d7154e8 commit 4539e88

File tree

2 files changed

+62
-114
lines changed

2 files changed

+62
-114
lines changed

‎cspell.config.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,7 @@ words:
189189
- petclinic
190190
- pgcrypto
191191
- pgdb
192+
- pgbackrest
192193
- PGPASSWORD
193194
- pipelinerun
194195
- pipelineruns

‎docs/operator-guide/ci/tekton-long-term-storage.md‎

Lines changed: 61 additions & 114 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
---
22
title: "Tekton Long-Term Log Storage"
3-
description: "Learn how to configure and access long-term logs for Tekton pipelines in KubeRocketCI using OpenSearch, enhancing log management and observability."
3+
description: "Tekton Results helps you logically group CI/CD workload history and separate long-term result storage from the Pipeline controller: add custom Result metadata (e.g. post-run actions), group related TaskRuns and PipelineRuns, keep result history independent of the Pipeline CRD controller to free etcd resources, and store logs so completed Runs can be cleaned."
44
sidebar_label: "Tekton Long-Term Log Storage"
55
---
66
<!-- markdownlint-disable MD025 -->
@@ -14,151 +14,98 @@ import TabItem from '@theme/TabItem';
1414
<link rel="canonical" href="https://docs.kuberocketci.io/docs/operator-guide/ci/tekton-long-term-storage" />
1515
</head>
1616

17-
KubeRocketCI Portal provides the capability to view long-term logs for pipelines. The long-term logs are stored in the OpenSearch cluster and can be viewed in the KubeRocketCI Portal after pipeline cleanup or deletion. This guide describes how to configure and view long-term logs in the KubeRocketCI Portal.
17+
Tekton Results aims to help users logically group CI/CD workload history and separate out long term result storage away from the Pipeline controller. This allows you to:
18+
19+
- Provide custom Results metadata about your CI/CD workflows not available in the Tekton TaskRun/PipelineRun CRDs (for example: post-run actions).
20+
- Group related workloads together (e.g. bundle related TaskRuns and PipelineRuns into a single unit).
21+
- Make long-term result history independent of the Pipeline CRD controller, letting you free up etcd resources for Run execution.
22+
- Store logs produced by the TaskRuns/PipelineRuns so that completed Runs can be cleaned to save resources.
1823

1924
## Long-Term Log Access Workflow
2025

2126
The following diagram illustrates the workflow for accessing long-term logs for pipelines in the KubeRocketCI Portal:
2227

28+
2329
```mermaid
2430
sequenceDiagram
2531
actor U as User
2632
participant KR as KubeRocketCI Portal
2733
participant PC as Pipeline Controller
28-
participant RW as Fluent Bit Agent
29-
participant KD as KrakenD API Gateway
30-
participant RA as OpenSearch
34+
participant RW as Result Watcher
35+
participant RA as Result API
3136
32-
U->>KR: Trigger or create a pipeline
37+
U->>KR: Trigger or create a PipelineRun/TaskRun
3338
KR->>PC: Start pipeline execution
34-
RW-->>PC: Collect pipeline logs
35-
RW->>RA: Send logs to OpenSearch
36-
Note over KR,RA: Pipeline is deleted or cleaned up
39+
RW-->>PC: Watch PipelineRun/TaskRun
40+
Note over PC,RW: Wait for PipelineRun/TaskRun Completion
41+
RW->>RA: Update results database
3742
U->>KR: View pipeline logs
38-
KR-->>KD: Request long-term logs
39-
KD-->>RA: Fetch long-term logs
43+
KR-->>RA: Get long-term logs
4044
KR->>U: Return long-term logs
45+
4146
```
4247

43-
## Configuration
44-
45-
To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:
46-
47-
1. Install and configure OpenSearch cluster:
48-
49-
:::note
50-
To install OpenSearch cluster, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/opensearch) repository, which contains already prepared configurations for OpenSearch installation.
51-
:::
52-
53-
To install OpenSearch cluster using the add-ons repository, follow the steps below:
54-
55-
1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
56-
57-
2. Navigate to the `clusters/core/addons/opensearch` directory and configure the `values.yaml` file with the necessary values for OpenSearch cluster installation.
58-
59-
:::note
60-
Ensure that [Fluent Bit](https://github.com/epam/edp-cluster-add-ons/blob/44ca88c079d464c826fcae38f3f03fe983d1f984/clusters/core/addons/opensearch/values.yaml#L391) is configured to send container logs to the `logstash-edp` index in the OpenSearch cluster, as the KubeRocketCI Portal uses this index to retrieve long-term logs.
61-
:::
62-
63-
3. Install the OpenSearch cluster using Helm or [Argo CD](../add-ons-overview.md) methods.
64-
65-
2. Install and configure KrakenD API Gateway:
66-
67-
:::note
68-
To install KrakenD API Gateway with OpenSearch connection, we recommend to use the [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/tree/main/clusters/core/addons/krakend) repository, which contains already prepared configurations for KrakenD installation.
69-
For more details, refer to the [KrakenD Integration](../extensions/krakend.md) guide.
70-
:::
71-
72-
To install KrakenD API Gateway with OpenSearch connection, follow the steps below:
73-
74-
1. Clone the forked [edp-cluster-add-ons](https://github.com/epam/edp-cluster-add-ons/) repository.
75-
76-
2. Navigate to the `clusters/core/addons/krakend` directory and configure the `values.yaml` file with the necessary values for KrakenD installation.
48+
## Install Tekton Results
7749

78-
:::note
79-
Ensure that the KrakenD endpoint object for OpenSearch has the `/search/logs` endpoint name, as the KubeRocketCI Portal uses this endpoint to fetch long-term logs.
80-
:::
50+
Tekton Results is deployed as part of the Tekton Pipelines installation. For installing Tekton Pipelines, we recommend using the [add-ons approach](https://github.com/epam/edp-cluster-add-ons/blob/4456631feb6510ff875b5b839534968c3846da7a/clusters/core/apps/values.yaml#L286C1-L289C30). Here's an example of how to configure values.yaml before installation:
8151

82-
3. Ensure that the KrakenD [configuration secret](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/krakend/templates/external-secrets/krakend.yaml) contains the OpenSearch connection variables.
52+
```yaml
53+
tekton:
54+
createNamespace: true
55+
enable: true
56+
namespace: tekton-pipelines
57+
```
8358

84-
<Tabs
85-
defaultValue="externalsecret"
86-
values={[
87-
{label: 'Manifests', value: 'manifests'},
88-
{label: 'External Secrets Operator', value: 'externalsecret'},
89-
]}>
90-
91-
<TabItem value="manifests">
92-
93-
```yaml
94-
apiVersion: v1
95-
kind: Secret
96-
metadata:
97-
name: krakend
98-
namespace: krakend
99-
type: Opaque
100-
stringData:
101-
OPENSEARCH_URL: https://opensearch-cluster-master.logging:9200
102-
OPENSEARCH_CREDS: <base64-encoded-credentials>
103-
```
104-
105-
</TabItem>
106-
107-
<TabItem value="externalsecret">
108-
109-
```json
110-
{
111-
"OPENSEARCH_URL": "https://opensearch-cluster-master.logging:9200",
112-
"OPENSEARCH_CREDS": "<base64-encoded-credentials>"
113-
}
114-
```
115-
</TabItem>
116-
117-
</Tabs>
118-
119-
4. Install the KrakenD API Gateway using Helm or [Argo CD](../add-ons-overview.md) methods.
120-
121-
## Viewing Long-Term Logs
122-
123-
After configuring long-term log storage for Pipelines in the KubeRocketCI Portal, follow the steps below to view long-term logs:
124-
125-
1. Navigate to the KubeRocketCI Portal and sign in with appropriate credentials.
126-
127-
![KubeRocketCI Portal](../../assets/operator-guide/ci/kuberocketci-portal.png "KubeRocketCI Portal")
128-
129-
2. In the **Components** section, navigate to the appropriate component (if component is not created yet, refer to the [Add Application](../../user-guide/add-application.md) guide to create a new one).
130-
131-
![KubeRocketCI Portal Components](../../assets/operator-guide/ci/portal-components.png "KubeRocketCI Portal Components")
132-
133-
3. In the component window, in the right upper corner, click the **GIT** button to navigate to the component repository (e.g., GitHub repository).
134-
135-
![Component Git Repository](../../assets/operator-guide/ci/component-git-repository.png "Component Git Repository")
59+
## Configuration
13660

137-
4. In the component repository, create a Pull Request to trigger the review pipeline execution.
61+
To configure long-term log storage for pipelines in the KubeRocketCI Portal, follow the steps below:
13862

139-
![Create Pull Request](../../assets/operator-guide/ci/create-pull-request.png "Create Pull Request")
63+
1. Storage sizes in the [results-pg.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results-pg.yaml) file define the allocated volumes for the Tekton Results PostgreSQL cluster:
14064

141-
5. In the Pull Request checks section, click the **Show all checks** button and navigate to the review pipeline in the KubeRocketCI Portal by clicking the **Details** button.
65+
- **Database instance storage**
66+
- Path: `spec.instances.dataVolumeClaimSpec.resources.requests.storage`
67+
- Default: `2Gi`
68+
- Description: Specifies the size of the main PostgreSQL data volume.
14269

143-
![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
70+
- **Backup repository storage**
71+
- Path: `spec.backups.pgbackrest.repos.volume.volumeClaimSpec.resources.requests.storage`
72+
- Default: `2Gi`
73+
- Description: Specifies the size of the pgBackRest backup repository volume.
14474

145-
6. In the pipeline window, check the pipeline status. Ensure that the pipeline is successfully executed and the logs are available.
75+
2. Storage and retention settings in the [results.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results.yaml) file define how Tekton Results result records are retained in the database and how related storage is configured:
14676

147-
![Pipeline Logs](../../assets/operator-guide/ci/pipeline-logs.png "Pipeline Logs")
77+
- **Log storage size**
78+
- Path: `PersistentVolumeClaim.spec.resources.requests.storage`
79+
- Default: `5Gi`
80+
- Description: Specifies the allocated disk size for storing pipeline logs.
14881

149-
7. Navigate to the **Pipelines** section and find the executed review pipeline. Delete the pipeline by clicking the **Delete** button.
82+
- **Retention period**
83+
- Path: `ConfigMap.data.defaultRetention` in `tekton-results-config-results-retention-policy`
84+
- Default: `"30"` days
85+
- Description: Defines how long log files are retained before being automatically removed.
15086

151-
![Delete Pipeline](../../assets/operator-guide/ci/delete-pipeline.png "Delete Pipeline")
87+
- **Cleanup schedule**
88+
- Path: `ConfigMap.data.runAt` in `tekton-results-config-results-retention-policy`
89+
- Default: `"0 18 * * *0*"`
90+
- Description: Specifies the cron expression that determines when automated log cleanup runs (daily at 6:00 PM UTC).
15291

153-
8. Return to the Pull Request and click the **Show all checks** button. Navigate to the review pipeline again by clicking the **Details** button.
92+
:::warning
93+
The retention policy agent removes only database records from PostgreSQL. It does not delete associated log files stored on persistent volumes, S3, or GCS backends. As a result, log data may remain on disk and consume storage even after the corresponding records are expired.
15494

155-
![PR Checks](../../assets/operator-guide/ci/pr-checks.png "PR Checks")
95+
To clean log files from PVCs, you can use the `tekton-results-clean-old-logs` CronJob.
96+
:::
15697

157-
9. After the pipeline deletion, the long-term logs are displayed in the pipeline window.
98+
3. To remove physical log files from PVCs (not just database records), use the settings in the [results-clean-old-logs-cronjob.yaml](https://github.com/epam/edp-cluster-add-ons/blob/main/clusters/core/addons/tekton/results-clean-old-logs-cronjob.yaml) file:
15899

159-
![Long Term Logs](../../assets/operator-guide/ci/reserved-logs.png "Long Term Logs")
100+
- **Cleanup schedule**
101+
- Path: `spec.schedule`
102+
- Default: `"0 18 * * *"`
103+
- Description: Specifies the cron expression that determines when the CronJob runs (daily at 6:00 PM UTC).
160104

161-
This example demonstrates how to view long-term logs for review pipeline, but the same approach can be applied to other pipeline types.
105+
- **Physical log retention (days)**
106+
- Path: `spec.jobTemplate.spec.template.spec.containers.command` (the `-mtime +N` value in the `find` command)
107+
- Default: `30` days
108+
- Description: Physical log files in `/tekton-results/logs/` older than this many days are deleted from the PVC. The value is the number in `-mtime +N` (e.g. `+30` keeps files for 30 days).
162109

163110
## Related Articles
164111

0 commit comments

Comments
 (0)