Skip to content

Commit 2a08639

Browse files
committed
Phase 2 - stream group commands
1 parent dfbe01a commit 2a08639

21 files changed

Lines changed: 692 additions & 183 deletions

content/commands/xack.md

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,6 +71,26 @@ so that such message does not get processed again, and as a side effect,
7171
the PEL entry about this message is also purged, releasing memory from the
7272
Redis server.
7373

74+
## Required arguments
75+
76+
<details open><summary><code>key</code></summary>
77+
78+
The stream key.
79+
80+
</details>
81+
82+
<details open><summary><code>group</code></summary>
83+
84+
The consumer group name.
85+
86+
</details>
87+
88+
<details open><summary><code>id [id ...]</code></summary>
89+
90+
One or more message IDs to acknowledge.
91+
92+
</details>
93+
7494
## Examples
7595

7696
```

content/commands/xautoclaim.md

Lines changed: 57 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -79,22 +79,22 @@ title: XAUTOCLAIM
7979
This command transfers ownership of pending stream entries that match the specified criteria. Conceptually, `XAUTOCLAIM` is equivalent to calling [`XPENDING`]({{< relref "/commands/xpending" >}}) and then [`XCLAIM`]({{< relref "/commands/xclaim" >}}),
8080
but provides a more straightforward way to deal with message delivery failures via [`SCAN`]({{< relref "/commands/scan" >}})-like semantics.
8181

82-
Like [`XCLAIM`]({{< relref "/commands/xclaim" >}}), the command operates on the stream entries at `<key>` and in the context of the provided `<group>`.
83-
It transfers ownership to `<consumer>` of messages pending for more than `<min-idle-time>` milliseconds and having an equal or greater ID than `<start>`.
82+
Like [`XCLAIM`]({{< relref "/commands/xclaim" >}}), the command operates on the stream entries at `key` and in the context of the provided `group`.
83+
It transfers ownership to `consumer` of messages pending for more than `min-idle-time` milliseconds and having an equal or greater ID than `start`.
8484

85-
The optional `<count>` argument, which defaults to 100, is the upper limit of the number of entries that the command attempts to claim.
86-
Internally, the command begins scanning the consumer group's Pending Entries List (PEL) from `<start>` and filters out entries having an idle time less than or equal to `<min-idle-time>`.
87-
The maximum number of pending entries that the command scans is the product of multiplying `<count>`'s value by 10 (hard-coded).
85+
The optional `count` argument, which defaults to 100, is the upper limit of the number of entries that the command attempts to claim.
86+
Internally, the command begins scanning the consumer group's Pending Entries List (PEL) from `start` and filters out entries having an idle time less than or equal to `min-idle-time`.
87+
The maximum number of pending entries that the command scans is the product of multiplying `count`'s value by 10 (hard-coded).
8888
It is possible, therefore, that the number of entries claimed will be less than the specified value.
8989

9090
The optional `JUSTID` argument changes the reply to return just an array of IDs of messages successfully claimed, without returning the actual message.
9191
Using this option means the retry counter is not incremented.
9292

93-
The command returns the claimed entries as an array. It also returns a stream ID intended for cursor-like use as the `<start>` argument for its subsequent call.
93+
The command returns the claimed entries as an array. It also returns a stream ID intended for cursor-like use as the `start` argument for its subsequent call.
9494
When there are no remaining PEL entries, the command returns the special `0-0` ID to signal completion.
95-
However, note that you may want to continue calling `XAUTOCLAIM` even after the scan is complete with the `0-0` as `<start>` ID, because enough time passed, so older pending entries may now be eligible for claiming.
95+
However, note that you may want to continue calling `XAUTOCLAIM` even after the scan is complete with the `0-0` as `start` ID, because enough time passed, so older pending entries may now be eligible for claiming.
9696

97-
Note that only messages that are idle longer than `<min-idle-time>` are claimed, and claiming a message resets its idle time.
97+
Note that only messages that are idle longer than `min-idle-time` are claimed, and claiming a message resets its idle time.
9898
This ensures that only a single consumer can successfully claim a given pending message at a specific instant of time and trivially reduces the probability of processing the same message multiple times.
9999

100100
Messages that have been released back to the group using [`XNACK`]({{< relref "/commands/xnack" >}}) are immediately claimable since their delivery time is set to 0, satisfying any minimum idle time requirement.
@@ -105,6 +105,52 @@ These message IDs are returned to the caller as a part of `XAUTOCLAIM`s reply.
105105
Lastly, claiming a message with `XAUTOCLAIM` also increments the attempted deliveries count for that message, unless the `JUSTID` option has been specified (which only delivers the message ID, not the message itself).
106106
Messages that cannot be processed for some reason - for example, because consumers systematically crash when processing them - will exhibit high attempted delivery counts that can be detected by monitoring.
107107

108+
## Required arguments
109+
110+
<details open><summary><code>key</code></summary>
111+
112+
The stream key.
113+
114+
</details>
115+
116+
<details open><summary><code>group</code></summary>
117+
118+
The consumer group name.
119+
120+
</details>
121+
122+
<details open><summary><code>consumer</code></summary>
123+
124+
The name of the consumer that will own the claimed messages.
125+
126+
</details>
127+
128+
<details open><summary><code>min-idle-time</code></summary>
129+
130+
Claim only messages that have been idle for at least this long, in milliseconds.
131+
132+
</details>
133+
134+
<details open><summary><code>start</code></summary>
135+
136+
The message ID to start scanning from. Use `0` to start at the beginning of the PEL.
137+
138+
</details>
139+
140+
## Optional arguments
141+
142+
<details open><summary><code>COUNT count</code></summary>
143+
144+
The number of messages to attempt to claim per call. Defaults to 100.
145+
146+
</details>
147+
148+
<details open><summary><code>JUSTID</code></summary>
149+
150+
Return only the IDs of the claimed messages, without their fields, and do not increment their retry counters.
151+
152+
</details>
153+
108154
## Examples
109155

110156
```
@@ -116,10 +162,9 @@ Messages that cannot be processed for some reason - for example, because consume
116162
3) (empty array)
117163
```
118164

119-
In the above example, we attempt to claim up to 25 entries that are pending and idle (not having been acknowledged or claimed) for at least an hour, starting at the stream's beginning.
120-
The consumer "Alice" from the "mygroup" group acquires ownership of these messages.
121-
Note that the stream ID returned in the example is `0-0`, indicating that the entire stream was scanned.
122-
We can also see that `XAUTOCLAIM` did not stumble upon any deleted messages (the third reply element is an empty array).
165+
In the previous example, you claim up to 25 pending entries that have been idle (having not been acknowledged or claimed) for at least one hour, starting from the beginning of the stream. The consumer Alice in the mygroup group takes ownership of these messages.
166+
167+
The returned stream ID is 0-0, which indicates that XAUTOCLAIM scanned the entire stream. The third reply element is an empty array, which means XAUTOCLAIM did not find any deleted messages.
123168

124169
## Redis Software and Redis Cloud compatibility
125170

content/commands/xcfgset.md

Lines changed: 13 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -81,13 +81,25 @@ When the capacity is reached, the oldest idempotent IDs for that producer are ev
8181

8282
</details>
8383

84-
## Behavior
84+
## Details
85+
86+
### Behavior
8587

8688
- Calling `XCFGSET` clears all existing producer IDMP maps for the stream.
8789
- At least one of `IDMP-DURATION` or `IDMP-MAXSIZE` must be specified.
8890
- The stream must exist before calling this command.
8991
- Configuration changes apply immediately to all future IDMP operations.
9092

93+
### Error conditions
94+
95+
The command returns an error in the following cases:
96+
97+
- `WRONGTYPE`: The key exists but is not a stream.
98+
- `ERR no such key`: The stream does not exist.
99+
- `ERR syntax error`: Invalid command syntax or missing required arguments.
100+
- `ERR invalid duration`: Duration value is outside the valid range (1-86,400).
101+
- `ERR invalid maxsize`: Maxsize value is outside the valid range (1-10,000).
102+
91103
## Examples
92104

93105
```redis-cli
@@ -116,13 +128,3 @@ XCFGSET mystream IDMP-DURATION 600 IDMP-MAXSIZE 500
116128
[Simple string reply](../../develop/reference/protocol-spec#simple-strings): `OK` if the configuration was set successfully.
117129

118130
{{< /multitabs >}}
119-
120-
## Error conditions
121-
122-
The command returns an error in the following cases:
123-
124-
- **WRONGTYPE**: The key exists but is not a stream
125-
- **ERR no such key**: The stream does not exist
126-
- **ERR syntax error**: Invalid command syntax or missing required arguments
127-
- **ERR invalid duration**: Duration value is outside the valid range (1-86,400)
128-
- **ERR invalid maxsize**: Maxsize value is outside the valid range (1-10,000)

content/commands/xclaim.md

Lines changed: 70 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -121,18 +121,77 @@ Moreover, as a side effect, `XCLAIM` will increment the count of attempted deliv
121121
In both cases the reply will not contain a corresponding entry to that message (i.e. the length of the reply array may be smaller than the number of IDs provided to `XCLAIM`).
122122
In the latter case, the message will also be deleted from the PEL in which it was found. This feature was introduced in Redis 7.0.
123123

124-
## Command options
124+
## Required arguments
125125

126-
The command has multiple options, however most are mainly for internal use in
127-
order to transfer the effects of `XCLAIM` or other commands to the AOF file
128-
and to propagate the same effects to the replicas, and are unlikely to be
129-
useful to normal users:
126+
<details open><summary><code>key</code></summary>
130127

131-
1. `IDLE <ms>`: Set the idle time (last time it was delivered) of the message. If IDLE is not specified, an IDLE of 0 is assumed, that is, the time count is reset because the message has now a new owner trying to process it.
132-
2. `TIME <ms-unix-time>`: This is the same as IDLE but instead of a relative amount of milliseconds, it sets the idle time to a specific Unix time (in milliseconds). This is useful in order to rewrite the AOF file generating `XCLAIM` commands.
133-
3. `RETRYCOUNT <count>`: Set the retry counter to the specified value. If not set, `XCLAIM` will increment the retry counter every time a message is delivered again.
134-
4. `FORCE`: Creates the pending message entry in the PEL even if certain specified IDs are not already in the PEL assigned to a different client. However the message must be exist in the stream, otherwise the IDs of non existing messages are ignored.
135-
5. `JUSTID`: Return just an array of IDs of messages successfully claimed, without returning the actual message. Using this option means the retry counter is not incremented.
128+
The stream key.
129+
130+
</details>
131+
132+
<details open><summary><code>group</code></summary>
133+
134+
The consumer group name.
135+
136+
</details>
137+
138+
<details open><summary><code>consumer</code></summary>
139+
140+
The name of the consumer that will own the claimed messages.
141+
142+
</details>
143+
144+
<details open><summary><code>min-idle-time</code></summary>
145+
146+
Claim only messages that have been idle for at least this long, in milliseconds.
147+
148+
</details>
149+
150+
<details open><summary><code>id [id ...]</code></summary>
151+
152+
One or more message IDs to claim.
153+
154+
</details>
155+
156+
## Optional arguments
157+
158+
Most of the following options are mainly for internal use, to transfer the effects of `XCLAIM` to the AOF file and replicas, and are unlikely to be useful to normal users.
159+
160+
<details open><summary><code>IDLE ms</code></summary>
161+
162+
Set the idle time (last time it was delivered) of the message. If `IDLE` is not specified, an idle time of `0` is assumed, that is, the time count is reset because the message now has a new owner trying to process it.
163+
164+
</details>
165+
166+
<details open><summary><code>TIME unix-time-milliseconds</code></summary>
167+
168+
The same as `IDLE` but sets the idle time to a specific Unix time (in milliseconds) instead of a relative amount of milliseconds. This is useful in order to rewrite the AOF file generating `XCLAIM` commands.
169+
170+
</details>
171+
172+
<details open><summary><code>RETRYCOUNT count</code></summary>
173+
174+
Set the retry counter to the specified value. If not set, `XCLAIM` increments the retry counter every time a message is delivered again.
175+
176+
</details>
177+
178+
<details open><summary><code>FORCE</code></summary>
179+
180+
Create the pending message entry in the PEL even if the specified IDs are not already in the PEL assigned to a different client. However, the message must exist in the stream, otherwise the IDs of non-existing messages are ignored.
181+
182+
</details>
183+
184+
<details open><summary><code>JUSTID</code></summary>
185+
186+
Return just an array of IDs of messages successfully claimed, without returning the actual messages. Using this option means the retry counter is not incremented.
187+
188+
</details>
189+
190+
<details open><summary><code>LASTID lastid</code></summary>
191+
192+
Update the consumer group's last-delivered ID to the given ID.
193+
194+
</details>
136195

137196
## Examples
138197

@@ -143,7 +202,7 @@ useful to normal users:
143202
2) "orange"
144203
```
145204

146-
In the above example we claim the message with ID `1526569498055-0`, only if the message is idle for at least one hour without the original consumer or some other consumer making progresses (acknowledging or claiming it), and assigns the ownership to the consumer `Alice`.
205+
In the previous example, you claim the message with ID 1526569498055-0 and assign it to the consumer Alice, but only if neither the original consumer nor another consumer has acknowledged or claimed it for at least one hour.
147206

148207
## Redis Software and Redis Cloud compatibility
149208

content/commands/xdel.md

Lines changed: 31 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -60,24 +60,19 @@ however Redis streams are represented in memory, so we are also able to
6060
delete entries. This may be useful, for instance, in order to comply with
6161
certain privacy policies.
6262

63-
## Understanding the low level details of entries deletion
63+
## Required arguments
6464

65-
Redis streams are represented in a way that makes them memory efficient:
66-
a radix tree is used in order to index macro-nodes that pack linearly tens
67-
of stream entries. Normally what happens when you delete an entry from a stream
68-
is that the entry is not *really* evicted, it just gets marked as deleted.
65+
<details open><summary><code>key</code></summary>
6966

70-
Eventually if all the entries in a macro-node are marked as deleted, the whole
71-
node is destroyed and the memory reclaimed. This means that if you delete
72-
a large amount of entries from a stream, for instance more than 50% of the
73-
entries appended to the stream, the memory usage per entry may increment, since
74-
what happens is that the stream will become fragmented. However the stream
75-
performance will remain the same.
67+
The stream key.
7668

77-
In future versions of Redis it is possible that we'll trigger a node garbage
78-
collection in case a given macro-node reaches a given amount of deleted
79-
entries. Currently with the usage we anticipate for this data structure, it is
80-
not a good idea to add such complexity.
69+
</details>
70+
71+
<details open><summary><code>id [id ...]</code></summary>
72+
73+
One or more message IDs to delete.
74+
75+
</details>
8176

8277
## Examples
8378

@@ -99,6 +94,27 @@ not a good idea to add such complexity.
9994
2) "3"
10095
```
10196

97+
## Details
98+
99+
### Understanding the low level details of entries deletion
100+
101+
Redis streams are represented in a way that makes them memory efficient:
102+
a radix tree is used in order to index macro-nodes that pack linearly tens
103+
of stream entries. Normally what happens when you delete an entry from a stream
104+
is that the entry is not *really* evicted, it just gets marked as deleted.
105+
106+
Eventually if all the entries in a macro-node are marked as deleted, the whole
107+
node is destroyed and the memory reclaimed. This means that if you delete
108+
a large amount of entries from a stream, for instance more than 50% of the
109+
entries appended to the stream, the memory usage per entry may increment, since
110+
what happens is that the stream will become fragmented. However the stream
111+
performance will remain the same.
112+
113+
In future versions of Redis it is possible that we'll trigger a node garbage
114+
collection in case a given macro-node reaches a given amount of deleted
115+
entries. Currently with the usage we anticipate for this data structure, it is
116+
not a good idea to add such complexity.
117+
102118
## Redis Software and Redis Cloud compatibility
103119

104120
| Redis<br />Software | Redis<br />Cloud | <span style="min-width: 9em; display: table-cell">Notes</span> |

content/commands/xgroup-create.md

Lines changed: 37 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -72,20 +72,20 @@ summary: Creates a consumer group.
7272
syntax_fmt: "XGROUP CREATE key group <id | $> [MKSTREAM]\n [ENTRIESREAD\_entries-read]"
7373
title: XGROUP CREATE
7474
---
75-
Create a new consumer group uniquely identified by `<groupname>` for the stream stored at `<key>`
75+
Create a new consumer group uniquely identified by `groupname` for the stream stored at `key`
7676

7777
Every group has a unique name in a given stream.
7878
When a consumer group with the same name already exists, the command returns a `-BUSYGROUP` error.
7979

80-
The command's `<id>` argument specifies the last delivered entry in the stream from the new group's perspective.
80+
The command's `id` argument specifies the last delivered entry in the stream from the new group's perspective.
8181
The special ID `$` is the ID of the last entry in the stream, but you can substitute it with any valid ID.
8282

8383
For example, if you want the group's consumers to fetch the entire stream from the beginning, use zero as the starting ID for the consumer group:
8484

8585
XGROUP CREATE mystream mygroup 0
8686

8787
By default, the `XGROUP CREATE` command expects that the target stream exists, and returns an error when it doesn't.
88-
If a stream does not exist, you can create it automatically with length of 0 by using the optional `MKSTREAM` subcommand as the last argument after the `<id>`:
88+
If a stream does not exist, you can create it automatically with length of 0 by using the optional `MKSTREAM` subcommand as the last argument after the `id`:
8989

9090
XGROUP CREATE mystream mygroup $ MKSTREAM
9191

@@ -94,6 +94,40 @@ An arbitrary ID is any ID that isn't the ID of the stream's first entry, last en
9494
Use it to find out how many entries are between the arbitrary ID (excluding it) and the stream's last entry.
9595
Set the `entries_read` the stream's `entries_added` subtracted by the number of entries.
9696

97+
## Required arguments
98+
99+
<details open><summary><code>key</code></summary>
100+
101+
The stream key.
102+
103+
</details>
104+
105+
<details open><summary><code>group</code></summary>
106+
107+
The name of the consumer group to create.
108+
109+
</details>
110+
111+
<details open><summary><code>id | $</code></summary>
112+
113+
The ID of the last delivered message the group starts reading after. Use `$` for the last message currently in the stream, or `0` to read from the start.
114+
115+
</details>
116+
117+
## Optional arguments
118+
119+
<details open><summary><code>MKSTREAM</code></summary>
120+
121+
Create the stream as an empty stream if it does not already exist.
122+
123+
</details>
124+
125+
<details open><summary><code>ENTRIESREAD entries-read</code></summary>
126+
127+
Set the group's initial entries-read counter, used to compute its lag.
128+
129+
</details>
130+
97131
## Redis Software and Redis Cloud compatibility
98132

99133
| Redis<br />Software | Redis<br />Cloud | <span style="min-width: 9em; display: table-cell">Notes</span> |

0 commit comments

Comments
 (0)