Skip to content

Commit b18f881

Browse files
jrcastro2kpsherva
authored andcommitted
vnext: add job logs index template update
1 parent 48f25d8 commit b18f881

File tree

1 file changed

+60
-0
lines changed

1 file changed

+60
-0
lines changed

docs/releases/vNext/upgrade-vNext.md

Lines changed: 60 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,66 @@ oaipmh_service = current_rdm_records.oaipmh_server_service
101101
oaipmh_service.rebuild_index(identity=system_identity)
102102
```
103103

104+
#### Job Logs Index Template Update
105+
106+
The job logs datastream index template has been updated to explicitly map two new context fields used for task tracking:
107+
108+
- `task_id` — unique identifier for the Celery task that produced the log
109+
- `parent_task_id` — identifier of the parent task (null for root tasks, set for subtasks)
110+
111+
Because the datastream index has `"dynamic": true` on the `context` object, existing indexed logs are unaffected. However, to ensure the new fields are properly mapped in future write indices, you need to:
112+
113+
1. **Update the index template** — add the two new fields under `mappings.properties.context.properties` of the `<PREFIX>-job-logs-v1.0.0` template.
114+
115+
Via the OpenSearch Dashboards UI: navigate to **Index Management → Templates**, find the template and edit it.
116+
117+
Or via curl:
118+
119+
```bash
120+
curl -X POST "http://<OPENSEARCH_HOST>/_index_template/<PREFIX>-job-logs-v1.0.0" \
121+
-H "Content-Type: application/json" \
122+
-d '{
123+
"index_patterns": ["<PREFIX>-job-logs*"],
124+
"data_stream": {},
125+
"template": {
126+
"mappings": {
127+
"properties": {
128+
"@timestamp": { "type": "date" },
129+
"level": { "type": "keyword" },
130+
"message": { "type": "text" },
131+
"module": { "type": "keyword" },
132+
"function": { "type": "keyword" },
133+
"line": { "type": "integer" },
134+
"context": {
135+
"type": "object",
136+
"properties": {
137+
"job_id": { "type": "keyword" },
138+
"run_id": { "type": "keyword" },
139+
"identity_id": { "type": "keyword" },
140+
"task_id": { "type": "keyword" },
141+
"parent_task_id": { "type": "keyword" }
142+
},
143+
"dynamic": true
144+
}
145+
}
146+
}
147+
}
148+
}'
149+
```
150+
151+
2. **Trigger a rollover of the datastream** — this creates a new backing index using the updated template, cleanly separating old and new log documents.
152+
153+
Via the OpenSearch Dashboards UI: navigate to **Index Management → Data Streams**, find the `<PREFIX>-job-logs` datastream, click **Actions** (top right) and select **Roll over**.
154+
155+
Or via curl:
156+
157+
```bash
158+
curl -X POST "http://<OPENSEARCH_HOST>/<PREFIX>-job-logs/_rollover"
159+
```
160+
161+
!!! note
162+
The template update is done in the **Index Templates** section and the rollover in the **Data Streams** section of the OpenSearch Dashboards UI.
163+
104164
#### OAuth client changes
105165

106166
The `extra_data` column of the `oauthclient_remoteaccount` table, storing remote-specific user information as required by various integrations, has been migrated from the `JSON` type to the `JSONB` type (only on PostgreSQL databases).

0 commit comments

Comments
 (0)