Skip to content

BITFIELD and BITFIELD_RO feature #2107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 15 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
157 changes: 157 additions & 0 deletions src/StackExchange.Redis/APITypes/BitfieldCommandBuilder.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
using System.Collections.Generic;

namespace StackExchange.Redis;

/// <summary>
/// Builder for bitfield commands that take multiple sub-commands.
/// </summary>
public class BitfieldCommandBuilder
{
private readonly LinkedList<RedisValue> _args = new LinkedList<RedisValue>();
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linked list is usually suboptimal; honestly I think we should default to List-T here

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Been a little while since I put this together so my recollection might be a bit off - IIRC my thinking was:

  1. We don't know the size of the array ahead of time - hence we can't really initialize a List without knowing it won't need to be resized.
  2. LinkedLists allow O(1) tail insertion
  3. We only need to enumerate when sending it, hence the O(N) scan is what you'd expect anyway.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Though now that I think of it, I suppose you are performing allocation each time you perform the addlast 🤔

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

frankly, linked-list just barely gets used - I'd sooner use a List<T>, but again I wonder whether this is actually a List<SomeUnionStructThatIsGetSetAndIncrby>, so each element in the list is not an argument but a logical operation - this might also make it much easier to do very efficient single-shot operations, which I expect to be relatively common. Let me have a think here - I like the ideas in this PR, but I think we can iterate the API a little.

private bool _eligibleForReadOnly;

/// <summary>
/// Builds a subcommand for a Bitfield GET, which returns the number stored in the specified offset of a bitfield at the given encoding.
/// </summary>
/// <param name="encoding">The encoding for the subcommand.</param>
/// <param name="offset">The offset into the bitfield for the subcommand.</param>
public BitfieldCommandBuilder Get(BitfieldEncoding encoding, BitfieldOffset offset)
{
_eligibleForReadOnly = true;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

surely we can only ever set this to false, with it starting true? if we Get(...).Set(...).Get(...) we're not eligible for readonly

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yep, definitely.

_args.AddLast(RedisLiterals.GET);
_args.AddLast(encoding.RedisValue);
_args.AddLast(offset.RedisValue);
return this;
}

/// <summary>
/// Builds a Bitfield subcommand which SETs the specified range of bits to the specified value.
/// </summary>
/// <param name="encoding">The encoding of the subcommand.</param>
/// <param name="offset">The offset of the subcommand.</param>
/// <param name="value">The value to set.</param>
public BitfieldCommandBuilder Set(BitfieldEncoding encoding, BitfieldOffset offset, long value)
{
_eligibleForReadOnly = false;
_args.AddLast(RedisLiterals.SET);
_args.AddLast(encoding.RedisValue);
_args.AddLast(offset.RedisValue);
_args.AddLast(value);
return this;
}

/// <summary>
/// Builds a subcommand for Bitfield INCRBY, which increments the number at the specified range of bits by the provided value
/// </summary>
/// <param name="encoding">The number's encoding.</param>
/// <param name="offset">The offset into the bitfield to increment.</param>
/// <param name="increment">The value to increment by.</param>
/// <param name="overflowHandling">How overflows will be handled when incrementing.</param>
public BitfieldCommandBuilder Incrby(BitfieldEncoding encoding, BitfieldOffset offset, long increment, BitfieldOverflowHandling overflowHandling = BitfieldOverflowHandling.Wrap)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be Increment for consistency with StringIncrement

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense.

{
_eligibleForReadOnly = false;
if (overflowHandling != BitfieldOverflowHandling.Wrap)
{
_args.AddLast(RedisLiterals.OVERFLOW);
_args.AddLast(overflowHandling.AsRedisValue());
}

_args.AddLast(RedisLiterals.INCRBY);
_args.AddLast(encoding.RedisValue);
_args.AddLast(offset.RedisValue);
_args.AddLast(increment);
return this;
}

internal BitfieldCommandMessage Build(int db, RedisKey key, CommandFlags flags, RedisBase redisBase, out ServerEndPoint? server)
{
var features = redisBase.GetFeatures(key, flags, out server);
var command = _eligibleForReadOnly && features.ReadOnlyBitfield ? RedisCommand.BITFIELD_RO : RedisCommand.BITFIELD;
return new BitfieldCommandMessage(db, flags, key, command, _args);
}
}

internal class BitfieldCommandMessage : Message
{
private readonly LinkedList<RedisValue> _args;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto List, although I think if possible we should skip an alloc here and go straight to an array; for now I'd settle for List-T, though

private readonly RedisKey _key;
public BitfieldCommandMessage(int db, CommandFlags flags, RedisKey key, RedisCommand command, LinkedList<RedisValue> args) : base(db, flags, command)
{
_key = key;
_args = args;
}

public override int ArgCount => 1 + _args.Count;

protected override void WriteImpl(PhysicalConnection physical)
{
physical.WriteHeader(Command, ArgCount);
physical.Write(_key);
foreach (var arg in _args)
{
physical.WriteBulkString(arg);
}
}
}

/// <summary>
/// The encoding that a sub-command should use. This is either a signed or unsigned integer of a specified length.
/// </summary>
public readonly struct BitfieldEncoding
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should have full struct equality impl - IEquatable, override GetHashCode, override Equals (via => obj is BitFieldEncoding other && Equals(other);

{
internal RedisValue RedisValue => $"{(IsSigned ? 'i' : 'u')}{Size}";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems uber-allocatey; I wonder if a static cache would work here, i.e.

=> s_Cache[Size + (IsSigned ? 0 : 64)] ??= $"...."


/// <summary>
/// Whether the integer is signed or not.
/// </summary>
public bool IsSigned { get; }

/// <summary>
/// The size of the integer.
/// </summary>
public byte Size { get; }

/// <summary>
/// Initializes the BitfieldEncoding.
/// </summary>
/// <param name="isSigned">Whether the encoding is signed.</param>
/// <param name="size">The size of the integer.</param>
public BitfieldEncoding(bool isSigned, byte size)
{
IsSigned = isSigned;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

validation on permitted sizes; 1-64I and 1-63U ?

Size = size;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should there be some pre-rolled values here? public static BitfieldEncoding Int32 {get}=new(true, 32); etc?

}
}

/// <summary>
/// An offset into a bitfield. This is either a literal offset (number of bits from the beginning of the bitfield) or an
/// encoding based offset, based off the encoding of the sub-command.
/// </summary>
public readonly struct BitfieldOffset
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto struct equality

{
/// <summary>
/// Returns the BitfieldOffset as a RedisValue.
/// </summary>
internal RedisValue RedisValue => $"{(ByEncoding ? "#" : string.Empty)}{Offset}";
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmmm; can't cache this one - I wonder if RedisValue is hampering us here, and we should be using a custom internal struct with a custom writer.... meh, leave it like this for now, we can change that later - let's just get it correct for now

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

note: this is unnecessarily allocatey in the raw offset case; should be IMO: => ByEncoding ? new($"#{Offset}") : new(Offset);


/// <summary>
/// Whether or not the BitfieldOffset will work off of the sub-commands integer encoding.
/// </summary>
public bool ByEncoding { get; }

/// <summary>
/// The number of either bits or encoded integers to offset into the bitfield.
/// </summary>
public long Offset { get; }

/// <summary>
/// Initializes a bitfield offset
/// </summary>
/// <param name="byEncoding">Whether or not the BitfieldOffset will work off of the sub-commands integer encoding.</param>
/// <param name="offset">The number of either bits or encoded integers to offset into the bitfield.</param>
public BitfieldOffset(bool byEncoding, long offset)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be (long offset, long byEncoding = true), perhaps even with an implicit conversion operator that does the same

{
ByEncoding = byEncoding;
Offset = offset;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this have to be non-negative?

}
}
33 changes: 33 additions & 0 deletions src/StackExchange.Redis/Enums/BitfieldOverflowHandling.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
using System;

namespace StackExchange.Redis;

/// <summary>
/// Defines the overflow behavior of a BITFIELD command.
/// </summary>
public enum BitfieldOverflowHandling
{
/// <summary>
/// Wraps around to the most negative value of signed integers, or zero for unsigned integers
/// </summary>
Wrap,
/// <summary>
/// Uses saturation arithmetic, stopping at the highest possible value for overflows, and the lowest possible value for underflows.
/// </summary>
Saturate,
/// <summary>
/// If an overflow is encountered, associated subcommand fails, and the result will be NULL.
/// </summary>
Fail,
}

internal static class BitfieldOverflowHandlingExtensions
{
internal static RedisValue AsRedisValue(this BitfieldOverflowHandling handling) => handling switch
{
BitfieldOverflowHandling.Fail => RedisLiterals.FAIL,
BitfieldOverflowHandling.Saturate => RedisLiterals.SAT,
BitfieldOverflowHandling.Wrap => RedisLiterals.WRAP,
_ => throw new ArgumentOutOfRangeException(nameof(handling))
};
}
4 changes: 4 additions & 0 deletions src/StackExchange.Redis/Enums/RedisCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ internal enum RedisCommand
BGREWRITEAOF,
BGSAVE,
BITCOUNT,
BITFIELD,
BITFIELD_RO,
BITOP,
BITPOS,
BLPOP,
Expand Down Expand Up @@ -259,6 +261,7 @@ internal static bool IsPrimaryOnly(this RedisCommand command)
// for example spreading load via a .DemandReplica flag in the caller.
// Basically: would it fail on a read-only replica in 100% of cases? Then it goes in the list.
case RedisCommand.APPEND:
case RedisCommand.BITFIELD:
case RedisCommand.BITOP:
case RedisCommand.BLPOP:
case RedisCommand.BRPOP:
Expand Down Expand Up @@ -358,6 +361,7 @@ internal static bool IsPrimaryOnly(this RedisCommand command)
case RedisCommand.BGREWRITEAOF:
case RedisCommand.BGSAVE:
case RedisCommand.BITCOUNT:
case RedisCommand.BITFIELD_RO:
case RedisCommand.BITPOS:
case RedisCommand.CLIENT:
case RedisCommand.CLUSTER:
Expand Down
54 changes: 54 additions & 0 deletions src/StackExchange.Redis/Interfaces/IDatabase.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2547,6 +2547,60 @@ IEnumerable<SortedSetEntry> SortedSetScan(RedisKey key,
/// <remarks><seealso href="https://redis.io/commands/bitcount"/></remarks>
long StringBitCount(RedisKey key, long start = 0, long end = -1, StringIndexType indexType = StringIndexType.Byte, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Executes a set of Bitfield subcommands as constructed by the <paramref name="builder"/> against the bitfield at the provided <paramref name="key"/>.
/// Will run as a <c>BITFIELD_RO</c> if all operations are read-only and the command is available.
/// </summary>
/// <param name="key">The key of the string.</param>
/// <param name="builder">The subcommands to execute against the bitfield.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>An array of numbers corresponding to the result of each sub-command. For increment subcommands, these can be null.</returns>
/// <remarks>
/// <seealso href="https://redis.io/commands/bitfield"/>,
/// <seealso href="https://redis.io/commands/bitfield_ro"/>
/// </remarks>
long?[] StringBitfield(RedisKey key, BitfieldCommandBuilder builder, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Pulls a single number out of a bitfield of the provided <paramref name="encoding"/> at the given offset.
/// Will execute a <c>BITFIELD_RO</c> if possible.
/// </summary>
/// <param name="key">The key for the string.</param>
/// <param name="encoding">The encoding of the number.</param>
/// <param name="offset">The offset into the bitfield to pull the number from.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>The number of the given <paramref name="encoding"/> at the provided <paramref name="offset"/>.</returns>
/// <remarks>
/// <seealso href="https://redis.io/commands/bitfield"/>,
/// <seealso href="https://redis.io/commands/bitfield_ro"/>
/// </remarks>
long StringBitfieldGet(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Sets a single number in a bitfield at the provided <paramref name="offset"/> to the <paramref name="value"/> provided, in the given <paramref name="encoding"/>.
/// </summary>
/// <param name="key">The key for the string.</param>
/// <param name="encoding">The encoding of the number.</param>
/// <param name="offset">The offset into the bitfield to pull the number from.</param>
/// <param name="value">the value to set the bitfield to.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>The previous value as an <see cref="long"/> at the provided <paramref name="offset"/>.</returns>
/// <remarks><seealso href="https://redis.io/commands/bitfield"/></remarks>
long StringBitfieldSet(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, long value, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Increments a single number in a bitfield at the provided <paramref name="offset"/> in the provided <paramref name="encoding"/> by the given <paramref name="increment"/>.
/// </summary>
/// <param name="key">The key for the string.</param>
/// <param name="encoding">The encoding of the number.</param>
/// <param name="offset">The offset into the bitfield to pull the number from.</param>
/// <param name="increment">the value to increment the bitfield by.</param>
/// <param name="overflowHandling">The way integer overflows are handled.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>The new value of the given at the provided <paramref name="offset"/> after the <c>INCRBY</c> is applied, represented as an <see cref="long"/>. Returns <see langword="null"/> if the operation fails.</returns>
/// <remarks><seealso href="https://redis.io/commands/bitfield"/></remarks>
long? StringBitfieldIncrement(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, long increment, BitfieldOverflowHandling overflowHandling = BitfieldOverflowHandling.Wrap, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Perform a bitwise operation between multiple keys (containing string values) and store the result in the destination key.
/// The BITOP command supports four bitwise operations; note that NOT is a unary operator: the second key should be omitted in this case
Expand Down
54 changes: 54 additions & 0 deletions src/StackExchange.Redis/Interfaces/IDatabaseAsync.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2499,6 +2499,60 @@ IAsyncEnumerable<SortedSetEntry> SortedSetScanAsync(RedisKey key,
/// <remarks><seealso href="https://redis.io/commands/bitcount"/></remarks>
Task<long> StringBitCountAsync(RedisKey key, long start = 0, long end = -1, StringIndexType indexType = StringIndexType.Byte, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Executes a set of Bitfield subcommands as constructed by the <paramref name="builder"/> against the bitfield at the provided <paramref name="key"/>.
/// Will run as a <c>BITFIELD_RO</c> if all operations are read-only and the command is available.
/// </summary>
/// <param name="key">The key of the string.</param>
/// <param name="builder">The subcommands to execute against the bitfield.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>An array of numbers corresponding to the result of each sub-command. For increment subcommands, these can be null.</returns>
/// <remarks>
/// <seealso href="https://redis.io/commands/bitfield"/>,
/// <seealso href="https://redis.io/commands/bitfield_ro"/>
/// </remarks>
Task<long?[]> StringBitfieldAsync(RedisKey key, BitfieldCommandBuilder builder, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Pulls a single number out of a bitfield of the provided <paramref name="encoding"/> at the given offset.
/// Will execute a <c>BITFIELD_RO</c> if possible.
/// </summary>
/// <param name="key">The key for the string.</param>
/// <param name="encoding">The encoding of the number.</param>
/// <param name="offset">The offset into the bitfield to pull the number from.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>The number of the given <paramref name="encoding"/> at the provided <paramref name="offset"/>.</returns>
/// <remarks>
/// <seealso href="https://redis.io/commands/bitfield"/>,
/// <seealso href="https://redis.io/commands/bitfield_ro"/>
/// </remarks>
Task<long> StringBitfieldGetAsync(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Sets a single number in a bitfield at the provided <paramref name="offset"/> to the <paramref name="value"/> provided, in the given <paramref name="encoding"/>.
/// </summary>
/// <param name="key">The key for the string.</param>
/// <param name="encoding">The encoding of the number.</param>
/// <param name="offset">The offset into the bitfield to pull the number from.</param>
/// <param name="value">the value to set the bitfield to.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>The previous value as an <see cref="long"/> at the provided <paramref name="offset"/>.</returns>
/// <remarks><seealso href="https://redis.io/commands/bitfield"/></remarks>
Task<long> StringBitfieldSetAsync(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, long value, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Increments a single number in a bitfield at the provided <paramref name="offset"/> in the provided <paramref name="encoding"/> by the given <paramref name="increment"/>.
/// </summary>
/// <param name="key">The key for the string.</param>
/// <param name="encoding">The encoding of the number.</param>
/// <param name="offset">The offset into the bitfield to pull the number from.</param>
/// <param name="increment">the value to increment the bitfield by.</param>
/// <param name="overflowHandling">The way integer overflows are handled.</param>
/// <param name="flags">The flags to use for this operation.</param>
/// <returns>The new value of the given at the provided <paramref name="offset"/> after the <c>INCRBY</c> is applied, represented as an <see cref="long"/>. Returns <see langword="null"/> if the operation fails.</returns>
/// <remarks><seealso href="https://redis.io/commands/bitfield"/></remarks>
Task<long?> StringBitfieldIncrementAsync(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, long increment, BitfieldOverflowHandling overflowHandling = BitfieldOverflowHandling.Wrap, CommandFlags flags = CommandFlags.None);

/// <summary>
/// Perform a bitwise operation between multiple keys (containing string values) and store the result in the destination key.
/// The BITOP command supports four bitwise operations; note that NOT is a unary operator: the second key should be omitted in this case
Expand Down
14 changes: 13 additions & 1 deletion src/StackExchange.Redis/KeyspaceIsolation/DatabaseWrapper.cs
Original file line number Diff line number Diff line change
Expand Up @@ -594,10 +594,22 @@ public long StringAppend(RedisKey key, RedisValue value, CommandFlags flags = Co

public long StringBitCount(RedisKey key, long start, long end, CommandFlags flags) =>
Inner.StringBitCount(ToInner(key), start, end, flags);

public long StringBitCount(RedisKey key, long start = 0, long end = -1, StringIndexType indexType = StringIndexType.Byte, CommandFlags flags = CommandFlags.None) =>
Inner.StringBitCount(ToInner(key), start, end, indexType, flags);

public long StringBitfieldGet(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, CommandFlags flags = CommandFlags.None) =>
Inner.StringBitfieldGet(ToInner(key), encoding, offset, flags);

public long StringBitfieldSet(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, long value, CommandFlags flags = CommandFlags.None) =>
Inner.StringBitfieldSet(ToInner(key), encoding, offset, value, flags);

public long? StringBitfieldIncrement(RedisKey key, BitfieldEncoding encoding, BitfieldOffset offset, long increment, BitfieldOverflowHandling overflowHandling = BitfieldOverflowHandling.Wrap, CommandFlags flags = CommandFlags.None) =>
Inner.StringBitfieldIncrement(ToInner(key), encoding, offset, increment, overflowHandling, flags);

public long?[] StringBitfield(RedisKey key, BitfieldCommandBuilder builder, CommandFlags flags = CommandFlags.None) =>
Inner.StringBitfield(key, builder, flags);

public long StringBitOperation(Bitwise operation, RedisKey destination, RedisKey[] keys, CommandFlags flags = CommandFlags.None) =>
Inner.StringBitOperation(operation, ToInner(destination), ToInner(keys), flags);

Expand Down
Loading