Skip to content

Commit 9809da0

Browse files
authored
DOC-1246 Add aggregate option to mongodb output and processor (#224)
1 parent f679813 commit 9809da0

File tree

2 files changed

+84
-88
lines changed

2 files changed

+84
-88
lines changed

modules/components/pages/outputs/mongodb.adoc

Lines changed: 43 additions & 48 deletions
Original file line numberDiff line numberDiff line change
@@ -21,30 +21,30 @@ Common::
2121
--
2222
2323
```yml
24-
# Common config fields, showing default values
24+
# Common configuration fields, showing default values
2525
output:
2626
label: ""
2727
mongodb:
2828
url: mongodb://localhost:27017 # No default (required)
2929
database: "" # No default (required)
30-
username: ""
31-
password: ""
30+
username: "" # No default (optional)
31+
password: "" # No default (optional)
3232
collection: "" # No default (required)
3333
operation: update-one
3434
write_concern:
35-
w: ""
35+
w: "" # No default (optional)
3636
j: false
37-
w_timeout: ""
38-
document_map: ""
39-
filter_map: ""
40-
hint_map: ""
37+
w_timeout: "" # No default (optional)
38+
document_map: "" # No default (optional)
39+
filter_map: "" # No default (optional)
40+
hint_map: "" # No default (optional)
4141
upsert: false
4242
max_in_flight: 64
4343
batching:
4444
count: 0
4545
byte_size: 0
46-
period: ""
47-
check: ""
46+
period: "" # No default (optional)
47+
check: "" # No default (optional)
4848
```
4949
5050
--
@@ -53,31 +53,31 @@ Advanced::
5353
--
5454
5555
```yml
56-
# All config fields, showing default values
56+
# All configuration fields, showing default values
5757
output:
5858
label: ""
5959
mongodb:
6060
url: mongodb://localhost:27017 # No default (required)
6161
database: "" # No default (required)
62-
username: ""
63-
password: ""
62+
username: "" # No default (optional)
63+
password: "" # No default (optional)
6464
app_name: benthos
6565
collection: "" # No default (required)
6666
operation: update-one
6767
write_concern:
68-
w: ""
68+
w: "" # No default (optional)
6969
j: false
70-
w_timeout: ""
71-
document_map: ""
72-
filter_map: ""
73-
hint_map: ""
70+
w_timeout: "" # No default (optional)
71+
document_map: "" # No default (optional)
72+
filter_map: "" # No default (optional)
73+
hint_map: "" # No default (optional)
7474
upsert: false
7575
max_in_flight: 64
7676
batching:
7777
count: 0
7878
byte_size: 0
79-
period: ""
80-
check: ""
79+
period: "" # No default (optional)
80+
check: "" # No default (optional)
8181
processors: [] # No default (optional)
8282
```
8383
@@ -87,9 +87,9 @@ output:
8787

8888
== Performance
8989

90-
This output benefits from sending multiple messages in flight in parallel for improved performance. You can tune the max number of in flight messages (or message batches) with the field `max_in_flight`.
90+
This output benefits from sending multiple messages in flight, in parallel, for improved performance. You can tune the maximum number of in flight messages (or message batches) using the `max_in_flight` field.
9191

92-
This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. You can find out more xref:configuration:batching.adoc[in this doc].
92+
This output benefits from sending messages as a batch for improved performance. Batches can be formed at both the input and output level. For more information, see xref:configuration:batching.adoc[Message Batching].
9393

9494
== Fields
9595

@@ -117,7 +117,7 @@ The name of the target MongoDB database.
117117

118118
=== `username`
119119

120-
The username to connect to the database.
120+
The username required to connect to the database.
121121

122122

123123
*Type*: `string`
@@ -126,12 +126,11 @@ The username to connect to the database.
126126

127127
=== `password`
128128

129-
The password to connect to the database.
129+
The password required to connect to the database.
130130

131131
include::components:partial$secret_warning.adoc[]
132132

133133

134-
135134
*Type*: `string`
136135

137136
*Default*: `""`
@@ -140,10 +139,10 @@ include::components:partial$secret_warning.adoc[]
140139

141140
The client application name.
142141

143-
144142
*Type*: `string`
145143

146-
*Default*: `"benthos"`
144+
*Default*: `benthos`
145+
147146

148147
=== `collection`
149148

@@ -155,32 +154,29 @@ The name of the target collection. This field supports xref:configuration:interp
155154

156155
=== `operation`
157156

158-
The mongodb operation to perform.
159-
157+
The MongoDB database operation to perform.
160158

161159
*Type*: `string`
162160

163-
*Default*: `"update-one"`
161+
*Default*: `update-one`
164162

165163
Options:
166164
`insert-one`
167165
, `delete-one`
168166
, `delete-many`
169167
, `replace-one`
170168
, `update-one`
171-
.
172169

173170
=== `write_concern`
174171

175-
The write concern settings for the mongo connection.
176-
172+
The https://www.mongodb.com/docs/manual/reference/write-concern/[write concern settings^] for the MongoDB connection.
177173

178174
*Type*: `object`
179175

180176

181177
=== `write_concern.w`
182178

183-
W requests acknowledgement that write operations propagate to the specified number of mongodb instances.
179+
The `w` requests acknowledgement, which write operations propagate to the specified number of MongoDB instances.
184180

185181

186182
*Type*: `string`
@@ -189,8 +185,7 @@ W requests acknowledgement that write operations propagate to the specified numb
189185

190186
=== `write_concern.j`
191187

192-
J requests acknowledgement from MongoDB that write operations are written to the journal.
193-
188+
The `j` requests acknowledgement from MongoDB, which is created when write operations are written to the journal.
194189

195190
*Type*: `bool`
196191

@@ -207,7 +202,7 @@ The write concern timeout.
207202

208203
=== `document_map`
209204

210-
A bloblang map representing a document to store within MongoDB, expressed as https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/[extended JSON in canonical form^]. The document map is required for the operations insert-one, replace-one and update-one.
205+
A Bloblang map that represents a document to store in MongoDB, expressed as https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/[extended JSON in canonical form^]. The `document_map` parameter is required for the following database operations: `insert-one`, `replace-one`, and `update-one`.
211206

212207

213208
*Type*: `string`
@@ -224,8 +219,9 @@ document_map: |-
224219

225220
=== `filter_map`
226221

227-
A bloblang map representing a filter for a MongoDB command, expressed as https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/[extended JSON in canonical form^]. The filter map is required for all operations except insert-one. It is used to find the document(s) for the operation. For example in a delete-one case, the filter map should have the fields required to locate the document to delete.
222+
A Bloblang map that represents a filter for a MongoDB command, expressed as https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/[extended JSON in canonical form^]. The `filter_map` parameter is required for all database operations except `insert-one`.
228223

224+
This output uses `filter_map` to find documents for the specified operation. For example, for a `delete-one` operation, the filter map should include the fields required to locate the document for deletion.
229225

230226
*Type*: `string`
231227

@@ -241,8 +237,9 @@ filter_map: |-
241237

242238
=== `hint_map`
243239

244-
A bloblang map representing the hint for the MongoDB command, expressed as https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/[extended JSON in canonical form^]. This map is optional and is used with all operations except insert-one. It is used to improve performance of finding the documents in the mongodb.
240+
A Bloblang map that represents a hint or index for a MongoDB command to use, expressed as https://www.mongodb.com/docs/manual/reference/mongodb-extended-json/[extended JSON in canonical form^]. This map is optional, and is used with all operations except `insert-one`.
245241

242+
Define a `hint_map` to improve performance when finding documents in the MongoDB database.
246243

247244
*Type*: `string`
248245

@@ -258,17 +255,17 @@ hint_map: |-
258255

259256
=== `upsert`
260257

261-
The upsert setting is optional and only applies for update-one and replace-one operations. If the filter specified in filter_map matches, the document is updated or replaced accordingly, otherwise it is created.
262-
258+
The `upsert` parameter is optional, and only applies for `update-one` and `replace-one` operations. If the filter specified in `filter_map` matches an existing document, this operation updates or replaces the document, otherwise a new document is created.
263259

264260
*Type*: `bool`
265261

266262
*Default*: `false`
263+
267264
Requires version 3.60.0 or newer
268265

269266
=== `max_in_flight`
270267

271-
The maximum number of messages to have in flight at a given time. Increase this to improve throughput.
268+
The maximum number of messages to have in flight at a given time. Increase this number to improve throughput.
272269

273270

274271
*Type*: `int`
@@ -303,25 +300,23 @@ batching:
303300

304301
=== `batching.count`
305302

306-
A number of messages at which the batch should be flushed. If `0` disables count based batching.
307-
303+
The number of messages after which the batch is flushed. Set to `0` to disable count-based batching.
308304

309305
*Type*: `int`
310306

311307
*Default*: `0`
312308

313309
=== `batching.byte_size`
314310

315-
An amount of bytes at which the batch should be flushed. If `0` disables size based batching.
316-
311+
The number of bytes at which the batch is flushed. Set to `0` to disable size-based batching.
317312

318313
*Type*: `int`
319314

320315
*Default*: `0`
321316

322317
=== `batching.period`
323318

324-
A period in which an incomplete batch should be flushed regardless of its size.
319+
The period after which an incomplete batch is flushed regardless of its size.
325320

326321

327322
*Type*: `string`
@@ -355,7 +350,7 @@ check: this.type == "end_of_transaction"
355350

356351
=== `batching.processors`
357352

358-
A list of xref:components:processors/about.adoc[processors] to apply to a batch as it is flushed. This allows you to aggregate and archive the batch however you see fit. Please note that all resulting messages are flushed as a single batch, therefore splitting the batch into smaller batches using these processors is a no-op.
353+
For aggregating and archiving message batches, you can add a list of xref:components:processors/about.adoc[processors] to apply to a batch as it is flushed. All resulting messages are flushed as a single batch even when you configure processors to split the batch into smaller batches.
359354

360355

361356
*Type*: `array`

0 commit comments

Comments
 (0)