Skip to content

Commit 15184ff

Browse files
BulkBeingBulkBeing
authored
Merge branch 'main' into pulsar-e2e-tests
2 parents cd54c7b + ade70d8 commit 15184ff

File tree

133 files changed

+9224
-3030
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

133 files changed

+9224
-3030
lines changed

‎.github/workflows/nightly-build.yml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ jobs:
3535
- name: Make checksums
3636
run: make checksums
3737
- name: store artifacts
38-
uses: actions/upload-artifact@v3
38+
uses: actions/upload-artifact@v4
3939
with:
4040
name: binaries
4141
path: dist
@@ -65,7 +65,7 @@ jobs:
6565
- name: Rename binary
6666
run: cp -pv target/x86_64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-amd64
6767
- name: Upload numaflow binary
68-
uses: actions/upload-artifact@v3
68+
uses: actions/upload-artifact@v4
6969
with:
7070
name: numaflow-rs-linux-amd64
7171
path: rust/numaflow-rs-linux-amd64
@@ -98,7 +98,7 @@ jobs:
9898
- name: Rename binary
9999
run: cp -pv target/aarch64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-arm64
100100
- name: Upload numaflow binary
101-
uses: actions/upload-artifact@v3
101+
uses: actions/upload-artifact@v4
102102
with:
103103
name: numaflow-rs-linux-arm64
104104
path: rust/numaflow-rs-linux-arm64
@@ -127,19 +127,19 @@ jobs:
127127
uses: docker/setup-buildx-action@v2
128128

129129
- name: Download Go binaries
130-
uses: actions/download-artifact@v3
130+
uses: actions/download-artifact@v4
131131
with:
132132
name: binaries
133133
path: dist/
134134

135135
- name: Download Rust amd64 binaries
136-
uses: actions/download-artifact@v3
136+
uses: actions/download-artifact@v4
137137
with:
138138
name: numaflow-rs-linux-amd64
139139
path: dist/
140140

141141
- name: Download Rust arm64 binaries
142-
uses: actions/download-artifact@v3
142+
uses: actions/download-artifact@v4
143143
with:
144144
name: numaflow-rs-linux-arm64
145145
path: dist/

‎.github/workflows/release.yml‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,7 @@ jobs:
3333
- name: Make checksums
3434
run: make checksums
3535
- name: store artifacts
36-
uses: actions/upload-artifact@v3
36+
uses: actions/upload-artifact@v4
3737
with:
3838
name: binaries
3939
path: dist
@@ -56,7 +56,7 @@ jobs:
5656
- name: Rename binary
5757
run: cp -pv target/x86_64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-amd64
5858
- name: Upload numaflow binary
59-
uses: actions/upload-artifact@v3
59+
uses: actions/upload-artifact@v4
6060
with:
6161
name: numaflow-rs-linux-amd64
6262
path: rust/numaflow-rs-linux-amd64
@@ -82,7 +82,7 @@ jobs:
8282
- name: Rename binary
8383
run: cp -pv target/aarch64-unknown-linux-gnu/release/numaflow numaflow-rs-linux-arm64
8484
- name: Upload numaflow binary
85-
uses: actions/upload-artifact@v3
85+
uses: actions/upload-artifact@v4
8686
with:
8787
name: numaflow-rs-linux-arm64
8888
path: rust/numaflow-rs-linux-arm64
@@ -111,19 +111,19 @@ jobs:
111111
uses: docker/setup-buildx-action@v2
112112

113113
- name: Download Go binaries
114-
uses: actions/download-artifact@v3
114+
uses: actions/download-artifact@v4
115115
with:
116116
name: binaries
117117
path: dist/
118118

119119
- name: Download Rust amd64 binaries
120-
uses: actions/download-artifact@v3
120+
uses: actions/download-artifact@v4
121121
with:
122122
name: numaflow-rs-linux-amd64
123123
path: dist/
124124

125125
- name: Download Rust arm64 binaries
126-
uses: actions/download-artifact@v3
126+
uses: actions/download-artifact@v4
127127
with:
128128
name: numaflow-rs-linux-arm64
129129
path: dist/
@@ -171,7 +171,7 @@ jobs:
171171
- run: bom generate --image quay.io/numaproj/numaflow:$VERSION -o /tmp/numaflow.spdx
172172
# pack the boms into one file to make it easy to download
173173
- run: cd /tmp && tar -zcf sbom.tar.gz *.spdx
174-
- uses: actions/upload-artifact@v3
174+
- uses: actions/upload-artifact@v4
175175
with:
176176
name: sbom.tar.gz
177177
path: /tmp/sbom.tar.gz
@@ -191,11 +191,11 @@ jobs:
191191
echo "VERSION=${GITHUB_REF##*/}" >> $GITHUB_ENV
192192
fi
193193
- name: Download binaries
194-
uses: actions/download-artifact@v3
194+
uses: actions/download-artifact@v4
195195
with:
196196
name: binaries
197197
path: dist/
198-
- uses: actions/download-artifact@v3
198+
- uses: actions/download-artifact@v4
199199
with:
200200
name: sbom.tar.gz
201201
path: /tmp

‎CHANGELOG.md‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,25 @@
11
# Changelog
22

3+
## v1.4.2 (2024-12-11)
4+
5+
* [c9dc38f4](https://github.com/numaproj/numaflow/commit/c9dc38f4cce2b5db598536a7539f2a35febcf1ca) Update manifests to v1.4.2
6+
* [fea792b3](https://github.com/numaproj/numaflow/commit/fea792b36bd342adcdcdd96768b6fdd68921bfd2) fix: set max decode size of proto message (#2275)
7+
8+
### Contributors
9+
10+
* Sidhant Kohli
11+
12+
## v1.4.1 (2024-12-05)
13+
14+
* [346f2a73](https://github.com/numaproj/numaflow/commit/346f2a7321d158fa9ce9392cfdcc76d671d6f577) Update manifests to v1.4.1
15+
* [1343e4d4](https://github.com/numaproj/numaflow/commit/1343e4d47934afcea324d4426df810dd9e99d9ab) feat: add sdk infomation metrics (#2208)
16+
* [1abb5ede](https://github.com/numaproj/numaflow/commit/1abb5ede3577016b7c2a923755e1445146efdb05) fix: Fix Sink Config to respect Fallback (#2261)
17+
18+
### Contributors
19+
20+
* Derek Wang
21+
* Yashash H L
22+
323
## v1.4.0 (2024-11-08)
424

525
* [6892c115](https://github.com/numaproj/numaflow/commit/6892c11590ea482c186724e55837dbcfb2100ce3) Update manifests to v1.4.0
@@ -283,6 +303,15 @@
283303
* samhith-kakarla
284304
* xdevxy
285305

306+
## v1.2.2 (2024-11-15)
307+
308+
* [61adf4e9](https://github.com/numaproj/numaflow/commit/61adf4e9805c2772d937a7513afcb3c14048127c) Update manifests to v1.2.2
309+
* [623cc4e2](https://github.com/numaproj/numaflow/commit/623cc4e2aaa2d67d196cb972bd525a60544d2148) fix: update key len (#2223)
310+
311+
### Contributors
312+
313+
* Sidhant Kohli
314+
286315
## v1.2.1 (2024-05-07)
287316

288317
* [89ea33f1](https://github.com/numaproj/numaflow/commit/89ea33f1d69785f6f5f17f1d5854ac189003918a) Update manifests to v1.2.1

‎Dockerfile‎

Lines changed: 1 addition & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -69,8 +69,6 @@ COPY --from=base /bin/numaflow /bin/numaflow
6969
COPY --from=base /bin/numaflow-rs /bin/numaflow-rs
7070
COPY ui/build /ui/build
7171

72-
COPY ./rust/serving/config config
73-
7472
ENTRYPOINT [ "/bin/numaflow" ]
7573

7674
####################################################################################################
@@ -89,4 +87,4 @@ RUN chmod +x /bin/e2eapi
8987
####################################################################################################
9088
FROM scratch AS e2eapi
9189
COPY --from=testbase /bin/e2eapi .
92-
ENTRYPOINT ["/e2eapi"]
90+
ENTRYPOINT ["/e2eapi"]

‎api/json-schema/schema.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21764,7 +21764,7 @@
2176421764
"description": "BackOff specifies the parameters for the backoff strategy, controlling how delays between retries should increase."
2176521765
},
2176621766
"onFailure": {
21767-
"description": "OnFailure specifies the action to take when a retry fails. The default action is to retry.",
21767+
"description": "OnFailure specifies the action to take when the specified retry strategy fails. The possible values are: 1. \"retry\": start another round of retrying the operation, 2. \"fallback\": re-route the operation to a fallback sink and 3. \"drop\": drop the operation and perform no further action. The default action is to retry.",
2176821768
"type": "string"
2176921769
}
2177021770
},

‎api/openapi-spec/swagger.json‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21751,7 +21751,7 @@
2175121751
"$ref": "#/definitions/io.numaproj.numaflow.v1alpha1.Backoff"
2175221752
},
2175321753
"onFailure": {
21754-
"description": "OnFailure specifies the action to take when a retry fails. The default action is to retry.",
21754+
"description": "OnFailure specifies the action to take when the specified retry strategy fails. The possible values are: 1. \"retry\": start another round of retrying the operation, 2. \"fallback\": re-route the operation to a fallback sink and 3. \"drop\": drop the operation and perform no further action. The default action is to retry.",
2175521755
"type": "string"
2175621756
}
2175721757
}

‎config/advanced-install/namespaced-numaflow-server.yaml‎

Lines changed: 120 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -143,6 +143,77 @@ data:
143143
# example for local prometheus service
144144
# url: http://prometheus-operated.monitoring.svc.cluster.local:9090
145145
patterns:
146+
- name: vertex_gauge
147+
object: vertex
148+
title: Vertex Pending Messages
149+
description: This query is the total number of pending messages for the vertex
150+
expr: |
151+
sum($metric_name{$filters}) by ($dimension, period)
152+
params:
153+
- name: start_time
154+
required: false
155+
- name: end_time
156+
required: false
157+
metrics:
158+
- metric_name: vertex_pending_messages
159+
display_name: Vertex Pending Messages
160+
# set "Units" or unset for default behaviour
161+
# unit: Units
162+
required_filters:
163+
- namespace
164+
- pipeline
165+
- vertex
166+
dimensions:
167+
- name: pod
168+
# expr: optional expression for prometheus query
169+
# overrides the default expression
170+
filters:
171+
- name: pod
172+
required: false
173+
- name: period
174+
required: false
175+
- name: vertex
176+
# expr: optional expression for prometheus query
177+
# overrides the default expression
178+
filters:
179+
- name: period
180+
required: false
181+
182+
- name: mono_vertex_gauge
183+
object: mono-vertex
184+
title: Pending Messages Lag
185+
description: This query is the total number of pending messages for the mono vertex
186+
expr: |
187+
sum($metric_name{$filters}) by ($dimension, period)
188+
params:
189+
- name: start_time
190+
required: false
191+
- name: end_time
192+
required: false
193+
metrics:
194+
- metric_name: monovtx_pending
195+
display_name: Mono Vertex Pending Messages
196+
# set "Units" or unset for default behaviour
197+
# unit: Units
198+
required_filters:
199+
- namespace
200+
- mvtx_name
201+
dimensions:
202+
- name: pod
203+
# expr: optional expression for prometheus query
204+
# overrides the default expression
205+
filters:
206+
- name: pod
207+
required: false
208+
- name: period
209+
required: false
210+
- name: mono-vertex
211+
# expr: optional expression for prometheus query
212+
# overrides the default expression
213+
filters:
214+
- name: period
215+
required: false
216+
146217
- name: mono_vertex_histogram
147218
object: mono-vertex
148219
title: Processing Time Latency
@@ -160,32 +231,34 @@ data:
160231
required: false
161232
metrics:
162233
- metric_name: monovtx_processing_time_bucket
234+
display_name: Mono Vertex Processing Time Latency
235+
# set "Units" or unset for default behaviour otherwise set "s" or "ms" for latency metrics
236+
# Note: latency values are in μs
237+
# unit: s
163238
required_filters:
164239
- namespace
165240
- mvtx_name
166241
dimensions:
242+
- name: mono-vertex
167243
- name: pod
168-
# expr: optional expression for prometheus query
169-
# overrides the default expression
170244
filters:
171245
- name: pod
172246
required: false
247+
- metric_name: monovtx_sink_time_bucket
248+
display_name: Mono Vertex Sink Write Time Latency
249+
# set "Units" or unset for default behaviour otherwise set "s" or "ms" for latency metrics
250+
# Note: latency values are in μs
251+
# unit: ms
252+
required_filters:
253+
- namespace
254+
- mvtx_name
255+
dimensions:
173256
- name: mono-vertex
174-
# expr: optional expression for prometheus query
175-
# overrides the default expression
176-
# Add histogram metrics similar to the pattern above
177-
#- metric_name: monovtx_sink_time_bucket
178-
# required_filters:
179-
# - namespace
180-
# - mvtx_name
181-
# dimensions:
182-
# - name: pod
183-
# #expr: optional
184-
# filters:
185-
# - name: pod
186-
# required: false
187-
# - name: mono-vertex
188-
# #expr: optional
257+
- name: pod
258+
filters:
259+
- name: pod
260+
required: false
261+
189262
- name: vertex_throughput
190263
object: vertex
191264
title: Vertex Throughput and Message Rates
@@ -200,14 +273,42 @@ data:
200273
required: false
201274
metrics:
202275
- metric_name: forwarder_data_read_total
276+
display_name: Vertex Read Processing Rate
277+
# set "Units" or unset for default behaviour
278+
# unit: Units
203279
required_filters:
204280
- namespace
205281
- pipeline
206282
- vertex
207283
dimensions:
208284
- name: vertex
209-
# expr: optional expression for prometheus query
210-
# overrides the default expression
285+
- name: pod
286+
filters:
287+
- name: pod
288+
required: false
289+
290+
- name: mono_vertex_throughput
291+
object: mono-vertex
292+
title: Mono-Vertex Throughput and Message Rates
293+
description: This pattern measures the throughput of a mono-vertex in messages per second across different dimensions
294+
expr: sum(rate($metric_name{$filters}[$duration])) by ($dimension)
295+
params:
296+
- name: duration
297+
required: true
298+
- name: start_time
299+
required: false
300+
- name: end_time
301+
required: false
302+
metrics:
303+
- metric_name: monovtx_read_total
304+
display_name: Mono Vertex Read Processing Rate
305+
# set "Units" or unset for default behaviour
306+
# unit: Units
307+
required_filters:
308+
- namespace
309+
- mvtx_name
310+
dimensions:
311+
- name: mono-vertex
211312
- name: pod
212313
filters:
213314
- name: pod

0 commit comments

Comments
 (0)