-
Notifications
You must be signed in to change notification settings - Fork 15
feat: add StreamedListObjects support #156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 10 commits
Commits
Show all changes
30 commits
Select commit
Hold shift + click to select a range
84992d8
feat: add StreamedListObjects support
daniel-jonathan bead1e7
feat: add StreamedListObjects support
daniel-jonathan 5ce96b4
refactor: use using statement for cancellation registration
daniel-jonathan 35fb401
fix: improve streaming cancellation and test reliability
daniel-jonathan 71df2e8
Merge branch 'main' into feat/streamed_list_objects
daniel-jonathan eb03120
test: use FgaConstants for test values instead of hardcoded strings
daniel-jonathan 1064700
docs: simplify StreamedListObjects example and enhance README
daniel-jonathan de2ffb9
docs: add StreamedListObjects section to main README
daniel-jonathan a66c34b
test: add custom headers and rate limit tests for StreamedListObjects
daniel-jonathan 4a2ec7a
refactor: use using statement for CancellationTokenSource
daniel-jonathan 0b61657
fix: address CodeQL and resource disposal issues
daniel-jonathan d5aa28b
fix: address CodeQL and resource ContainsKey issue Addresses CodeQL, …
daniel-jonathan b28428a
refactor: add using statements for all IDisposable resources in tests
daniel-jonathan 2e4e35e
refactor: remove useless assignments in tests
daniel-jonathan bd0aa89
refactor: Updated commenting and Assertion messaging based on CodeRab…
daniel-jonathan 520e78e
refactor: Fixed dispose issue based on CodeQL feedback
daniel-jonathan 2f4d426
refactor: improve example to demonstrate computed relations
daniel-jonathan 1becd72
fix: batch tuple writes to respect 100-tuple limit
daniel-jonathan aa40ebd
fix: prevent logging sensitive data in error handler
daniel-jonathan 826d8e5
refactor: Updated the example README based on review feedback
daniel-jonathan db885db
test: add comprehensive tests for partial NDJSON streaming handling
daniel-jonathan bb2bea8
Merge branch 'main' into feat/streamed_list_objects
daniel-jonathan 99cca29
feat: add StreamedListObjects API support
daniel-jonathan 9ca91cd
perf: optimize dictionary lookups in model Equals methods
daniel-jonathan 4c98c13
perf: optimize model Equals methods for CodeQL compliance
daniel-jonathan 67d019e
Merge branch 'main' into feat/streamed_list_objects
SoulPancake 1c8b560
Merge branch 'main' into feat/streamed_list_objects
SoulPancake a50dce3
Merge branch 'main' into feat/streamed_list_objects
SoulPancake c01c35f
Merge branch 'main' into feat/streamed_list_objects
SoulPancake 677c4d5
fix: remove dupl package reference
SoulPancake File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,94 @@ | ||
| # Streamed List Objects Example | ||
|
|
||
| Demonstrates using `StreamedListObjects` to retrieve objects via the streaming API in the .NET SDK. | ||
|
|
||
| ## What is StreamedListObjects? | ||
|
|
||
| The Streamed ListObjects API is very similar to the ListObjects API, with two key differences: | ||
|
|
||
| 1. **Streaming Results**: Instead of collecting all objects before returning a response, it streams them to the client as they are collected. | ||
| 2. **No Pagination Limit**: The number of results returned is only limited by the execution timeout specified in the flag `OPENFGA_LIST_OBJECTS_DEADLINE`. | ||
|
|
||
| This makes it ideal for scenarios where you need to retrieve large numbers of objects without being constrained by pagination limits. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - .NET 6.0 or higher | ||
daniel-jonathan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| - OpenFGA server running on `http://localhost:8080` (or set `FGA_API_URL`) | ||
daniel-jonathan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
|
||
| ## Running | ||
|
|
||
| ```bash | ||
| # From the SDK root directory, build the SDK first | ||
| dotnet build src/OpenFga.Sdk/OpenFga.Sdk.csproj | ||
|
|
||
| # Then run the example | ||
| cd example/StreamedListObjectsExample | ||
| dotnet run | ||
| ``` | ||
|
|
||
| ## What it does | ||
|
|
||
| - Creates a temporary store | ||
| - Writes a simple authorization model | ||
| - Adds 2000 tuples | ||
| - Streams results via `StreamedListObjects` | ||
| - Shows progress (first 3 objects and every 500th) | ||
| - Cleans up the store | ||
|
|
||
| ## Key Features Demonstrated | ||
|
|
||
| ### IAsyncEnumerable Pattern | ||
|
|
||
| The `StreamedListObjects` method returns `IAsyncEnumerable<StreamedListObjectsResponse>`, which is the idiomatic .NET way to handle streaming data: | ||
|
|
||
| ```csharp | ||
| await foreach (var response in fgaClient.StreamedListObjects(request)) { | ||
| Console.WriteLine($"Received: {response.Object}"); | ||
| } | ||
| ``` | ||
|
|
||
| ### Early Break and Cleanup | ||
|
|
||
| The streaming implementation properly handles early termination: | ||
|
|
||
| ```csharp | ||
| await foreach (var response in fgaClient.StreamedListObjects(request)) { | ||
| Console.WriteLine(response.Object); | ||
| if (someCondition) { | ||
| break; // Stream is automatically cleaned up | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### Cancellation Support | ||
|
|
||
| Full support for `CancellationToken`: | ||
|
|
||
| ```csharp | ||
| using var cts = new CancellationTokenSource(); | ||
| cts.CancelAfter(TimeSpan.FromSeconds(5)); | ||
|
|
||
| try { | ||
| await foreach (var response in fgaClient.StreamedListObjects(request, options, cts.Token)) { | ||
| Console.WriteLine(response.Object); | ||
| } | ||
| } | ||
| catch (OperationCanceledException) { | ||
| Console.WriteLine("Operation cancelled"); | ||
| } | ||
| ``` | ||
|
|
||
| ## Benefits Over ListObjects | ||
|
|
||
| - **No Pagination**: Retrieve all objects in a single streaming request | ||
| - **Lower Memory**: Objects are processed as they arrive, not held in memory | ||
| - **Early Termination**: Can stop streaming at any point without wasting resources | ||
| - **Better for Large Results**: Ideal when expecting hundreds or thousands of objects | ||
|
|
||
| ## Performance Considerations | ||
|
|
||
| - Streaming starts immediately - no need to wait for all results | ||
| - HTTP connection remains open during streaming | ||
| - Properly handles cleanup if consumer stops early | ||
| - Supports all the same options as `ListObjects` (consistency, contextual tuples, etc.) | ||
100 changes: 100 additions & 0 deletions
100
example/StreamedListObjectsExample/StreamedListObjectsExample.cs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| using OpenFga.Sdk.Client; | ||
daniel-jonathan marked this conversation as resolved.
Show resolved
Hide resolved
|
||
| using OpenFga.Sdk.Client.Model; | ||
| using OpenFga.Sdk.Configuration; | ||
| using OpenFga.Sdk.Model; | ||
|
|
||
| namespace StreamedListObjectsExample; | ||
|
|
||
| public class StreamedListObjectsExample { | ||
| public static async Task Main() { | ||
| try { | ||
| var apiUrl = Environment.GetEnvironmentVariable("FGA_API_URL") ?? "http://localhost:8080"; | ||
|
|
||
| var client = new OpenFgaClient(new ClientConfiguration { ApiUrl = apiUrl }); | ||
|
|
||
| Console.WriteLine("Creating temporary store"); | ||
| var store = await client.CreateStore(new ClientCreateStoreRequest { Name = "streamed-list-objects" }); | ||
|
|
||
| var clientWithStore = new OpenFgaClient(new ClientConfiguration { | ||
| ApiUrl = apiUrl, | ||
| StoreId = store.Id | ||
| }); | ||
|
|
||
| Console.WriteLine("Writing authorization model"); | ||
| var authModel = await clientWithStore.WriteAuthorizationModel(new ClientWriteAuthorizationModelRequest { | ||
| SchemaVersion = "1.1", | ||
| TypeDefinitions = new List<TypeDefinition> { | ||
| new() { | ||
| Type = "user", | ||
| Relations = new Dictionary<string, Userset>() | ||
| }, | ||
| new() { | ||
| Type = "document", | ||
| Relations = new Dictionary<string, Userset> { | ||
| { | ||
| "can_read", new Userset { | ||
| This = new object() | ||
| } | ||
| } | ||
| }, | ||
| Metadata = new Metadata { | ||
| Relations = new Dictionary<string, RelationMetadata> { | ||
| { | ||
| "can_read", new RelationMetadata { | ||
| DirectlyRelatedUserTypes = new List<RelationReference> { | ||
| new() { Type = "user" } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| }); | ||
|
|
||
| var fga = new OpenFgaClient(new ClientConfiguration { | ||
| ApiUrl = apiUrl, | ||
| StoreId = store.Id, | ||
| AuthorizationModelId = authModel.AuthorizationModelId | ||
| }); | ||
|
|
||
| Console.WriteLine("Writing tuples"); | ||
| var tuples = new List<ClientTupleKey>(); | ||
| for (int i = 1; i <= 2000; i++) { | ||
| tuples.Add(new ClientTupleKey { | ||
| User = "user:anne", | ||
| Relation = "can_read", | ||
| Object = $"document:{i}" | ||
daniel-jonathan marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| }); | ||
| } | ||
| await fga.WriteTuples(tuples); | ||
| Console.WriteLine($"Wrote {tuples.Count} tuples"); | ||
|
|
||
| Console.WriteLine("Streaming objects..."); | ||
| var count = 0; | ||
| await foreach (var response in fga.StreamedListObjects( | ||
| new ClientListObjectsRequest { | ||
| User = "user:anne", | ||
| Relation = "can_read", | ||
| Type = "document" | ||
| }, | ||
| new ClientListObjectsOptions { | ||
| Consistency = ConsistencyPreference.HIGHERCONSISTENCY | ||
| })) { | ||
| count++; | ||
| if (count <= 3 || count % 500 == 0) { | ||
| Console.WriteLine($"- {response.Object}"); | ||
| } | ||
| } | ||
| Console.WriteLine($"✓ Streamed {count} objects"); | ||
|
|
||
| Console.WriteLine("Cleaning up..."); | ||
| await fga.DeleteStore(); | ||
| Console.WriteLine("Done"); | ||
| } | ||
| catch (Exception ex) { | ||
| Console.Error.WriteLine($"Error: {ex.Message}"); | ||
| Environment.Exit(1); | ||
| } | ||
| } | ||
| } | ||
22 changes: 22 additions & 0 deletions
22
example/StreamedListObjectsExample/StreamedListObjectsExample.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| <Project Sdk="Microsoft.NET.Sdk"> | ||
|
|
||
| <PropertyGroup> | ||
| <OutputType>Exe</OutputType> | ||
| <TargetFramework>net9.0</TargetFramework> | ||
| <ImplicitUsings>enable</ImplicitUsings> | ||
| <Nullable>enable</Nullable> | ||
| <DockerDefaultTargetOS>Linux</DockerDefaultTargetOS> | ||
| </PropertyGroup> | ||
|
|
||
| <!-- To target the released version, uncomment this section --> | ||
| <!-- <ItemGroup>--> | ||
| <!-- <PackageReference Include="OpenFga.Sdk" Version="0.5.1"><PrivateAssets>all</PrivateAssets></PackageReference>--> | ||
| <!-- </ItemGroup>--> | ||
|
|
||
| <!-- To target the local build, use project reference --> | ||
| <ItemGroup> | ||
| <ProjectReference Include="..\..\src\OpenFga.Sdk\OpenFga.Sdk.csproj" /> | ||
| </ItemGroup> | ||
|
|
||
| </Project> | ||
|
|
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.