Skip to content

test(e2e): Update e2e tests to support newer spec submodule #604

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 17 commits into
base: main
Choose a base branch
from
Open

test(e2e): Update e2e tests to support newer spec submodule #604

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
b92e939
Initial update
kylejuliandev Oct 15, 2025
3727780
Fix ProviderNotReady tests
kylejuliandev Oct 15, 2025
77aa26d
Fix test on provider status
kylejuliandev Oct 15, 2025
0869166
Add remaining E2E tests where possible
kylejuliandev Oct 19, 2025
c8c39ea
Merge branch 'main' into update-e2e-tests
kylejuliandev Oct 30, 2025
c0ee595
Reduce duplicate code for fetching Enum from descriptions
kylejuliandev Oct 30, 2025
f5c2c76
Reduce code duplication
kylejuliandev Oct 30, 2025
c8e0648
Add FlagDictionaryJsonConverter instead of mapping manually
kylejuliandev Oct 30, 2025
034f15a
Remove README and test-flags.json
kylejuliandev Oct 30, 2025
072f812
Add tests to handle Object flag types
kylejuliandev Oct 30, 2025
b8469be
Fix build by ensuring test-flags.json is linked from the submodule
kylejuliandev Oct 30, 2025
fa6072f
Fix issue when running e2e tests in non e2e ci
kylejuliandev Oct 30, 2025
56bf219
Reduce code duplication by adopting StepArgumentTransformation
kylejuliandev Nov 1, 2025
d0e3580
Remove step definitions for deprecated gherkin file
kylejuliandev Nov 1, 2025
f5470eb
Align step definition files like it is in other languages and sdks
kylejuliandev Nov 1, 2025
3924c28
Rename provider step definition file
kylejuliandev Nov 1, 2025
4c43260
Merge branch 'main' into update-e2e-tests
kylejuliandev Nov 2, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -351,5 +351,7 @@ ASALocalRun/
# integration tests
test/OpenFeature.E2ETests/Features/*.feature
test/OpenFeature.E2ETests/Features/*.feature.cs
test/OpenFeature.E2ETests/Features/README.md
test/OpenFeature.E2ETests/Features/test-flags.json
cs-report.json
specification.json
2 changes: 1 addition & 1 deletion spec
Open in desktop
Submodule spec updated 6 files
+148 −0 specification/assets/gherkin/README.md
+3 −2 specification/assets/gherkin/evaluation.feature
+596 −0 specification/assets/gherkin/evaluation_v2.feature
+3 −3 specification/assets/gherkin/hooks.feature
+2 −2 specification/assets/gherkin/metadata.feature
+221 −0 specification/assets/gherkin/test-flags.json
13 changes: 11 additions & 2 deletions test/OpenFeature.E2ETests/OpenFeature.E2ETests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
<Project Sdk="Microsoft.NET.Sdk">
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<TargetFrameworks>net8.0;net9.0</TargetFrameworks>
Expand All @@ -18,6 +18,7 @@
</PackageReference>
<PackageReference Include="Reqnroll.xUnit" />
<PackageReference Include="Microsoft.NET.Test.Sdk" />
<PackageReference Include="NSubstitute" />
<PackageReference Include="xunit" />
<PackageReference Include="xunit.runner.visualstudio">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
Expand All @@ -33,7 +34,15 @@
<ItemGroup>
<GherkinFiles Include="..\..\spec\specification\assets\gherkin\*" />
</ItemGroup>
<Copy SourceFiles="@(GherkinFiles)" DestinationFolder="Features/"/>
<Copy SourceFiles="@(GherkinFiles)" DestinationFolder="Features/" />
</Target>

<!-- Only link test-flags.json if the submodule is cloned -->
<!-- If no submodule is cloned, Reqnroll wont generate the e2e tests -->
<ItemGroup Condition="Exists('..\..\spec\specification\assets\gherkin\test-flags.json')">
<None Include="..\..\spec\specification\assets\gherkin\test-flags.json" Link="Features\test-flags.json">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
</ItemGroup>

</Project>
124 changes: 42 additions & 82 deletions test/OpenFeature.E2ETests/Steps/BaseStepDefinitions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
using System.Text.Json;
using OpenFeature.E2ETests.Utils;
using OpenFeature.Model;
using OpenFeature.Providers.Memory;
Expand All @@ -17,43 +18,69 @@ public BaseStepDefinitions(State state)
[Given(@"a stable provider")]
public async Task GivenAStableProvider()
{
var memProvider = new InMemoryProvider(E2EFlagConfig);
var options = new JsonSerializerOptions()
{
PropertyNameCaseInsensitive = true
};
options.Converters.Add(new FlagDictionaryJsonConverter());

var json = File.ReadAllText(Path.Combine("Features", "test-flags.json"));
var flags = JsonSerializer.Deserialize<Dictionary<string, Flag>>(json, options)
?? new Dictionary<string, Flag>();

var memProvider = new InMemoryProvider(flags);
await Api.Instance.SetProviderAsync(memProvider).ConfigureAwait(false);
this.State.Client = Api.Instance.GetClient("TestClient", "1.0.0");
}

[Given(@"a Boolean-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a Boolean-flag with key ""(.*)"" and a fallback value ""(.*)""")]
[Given(@"a boolean-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a boolean-flag with key ""(.*)"" and a fallback value ""(.*)""")]
Copy link
Contributor

@chrfwow chrfwow Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need this twice?

Copy link
Member

@askpt askpt Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As far as I remember there were a couple of steps where the type was written as Boolean and boolean. So both scenarios, even they are the "same" code wise, we need to write twice. Correct me if I'm wrong @kylejuliandev

Copy link
Contributor Author

@kylejuliandev kylejuliandev Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah exactly that. Reqnroll is case sensitive when it comes to step definitions, so I needed to declare the permutations (upper and lower case, as the gherkin files have a mix)

Copy link
Member

@aepfli aepfli Oct 31, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sidequestion, can we also handle this like we do it in java/python etc. and create a mapping on the first "type"-string? like

[Given(@"a "(.*)"-flag with key ""(.*)"" and a fallback value ""(.*)""")]
public void GivenAFlagWithKeyAndADefaultValue(string type, string key, string defaultType)
{
    var flagType = ComputeFlagType(type);
    var flagState = new FlagState(key, defaultType, flagType);
    this.State.Flag = flagState;
}

and ComputeFlagType could take the string, lowercase it, and handle this like a switch case?

but anyways, we should normalize the gherkin files, and create a ticket for this, or pr, to make everything capitalized, i think it is just the "hooks.feature"

Copy link
Contributor Author

@kylejuliandev kylejuliandev Nov 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 56bf219

I needed a seperate StepArgumentTransformation to handle the permutations of <flagtype>-flag, but this is definitley much more readable

public void GivenABoolean_FlagWithKeyAndADefaultValue(string key, string defaultType)
{
var flagState = new FlagState(key, defaultType, FlagType.Boolean);
this.State.Flag = flagState;
}

[Given(@"a Float-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a Float-flag with key ""(.*)"" and a fallback value ""(.*)""")]
[Given(@"a float-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a float-flag with key ""(.*)"" and a fallback value ""(.*)""")]
public void GivenAFloat_FlagWithKeyAndADefaultValue(string key, string defaultType)
{
var flagState = new FlagState(key, defaultType, FlagType.Float);
this.State.Flag = flagState;
}

[Given(@"a Integer-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a Integer-flag with key ""(.*)"" and a fallback value ""(.*)""")]
[Given(@"a integer-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a integer-flag with key ""(.*)"" and a fallback value ""(.*)""")]
public void GivenAnInteger_FlagWithKeyAndADefaultValue(string key, string defaultType)
{
var flagState = new FlagState(key, defaultType, FlagType.Integer);
this.State.Flag = flagState;
}

[Given(@"a String-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a String-flag with key ""(.*)"" and a fallback value ""(.*)""")]
[Given(@"a string-flag with key ""(.*)"" and a default value ""(.*)""")]
[Given(@"a string-flag with key ""(.*)"" and a fallback value ""(.*)""")]
public void GivenAString_FlagWithKeyAndADefaultValue(string key, string defaultType)
{
var flagState = new FlagState(key, defaultType, FlagType.String);
this.State.Flag = flagState;
}

[Given(@"a Object-flag with key ""(.*)"" and a fallback value ""(.*)""")]
[Given(@"a object-flag with key ""(.*)"" and a fallback value ""(.*)""")]
public void GivenAObject_FlagWithKeyAndADefaultValue(string key, string defaultType)
{
var flagState = new FlagState(key, defaultType, FlagType.Object);
this.State.Flag = flagState;
}

[Given("a stable provider with retrievable context is registered")]
public async Task GivenAStableProviderWithRetrievableContextIsRegistered()
{
Expand Down Expand Up @@ -111,18 +138,28 @@ public async Task WhenTheFlagWasEvaluatedWithDetails()
{
case FlagType.Boolean:
this.State.FlagEvaluationDetailsResult = await this.State.Client!
.GetBooleanDetailsAsync(flag.Key, bool.Parse(flag.DefaultValue)).ConfigureAwait(false);
.GetBooleanDetailsAsync(flag.Key, bool.Parse(flag.DefaultValue), this.State.EvaluationContext)
.ConfigureAwait(false);
break;
case FlagType.Float:
this.State.FlagEvaluationDetailsResult = await this.State.Client!
.GetDoubleDetailsAsync(flag.Key, double.Parse(flag.DefaultValue)).ConfigureAwait(false);
.GetDoubleDetailsAsync(flag.Key, double.Parse(flag.DefaultValue), this.State.EvaluationContext)
.ConfigureAwait(false);
break;
case FlagType.Integer:
this.State.FlagEvaluationDetailsResult = await this.State.Client!
.GetIntegerDetailsAsync(flag.Key, int.Parse(flag.DefaultValue)).ConfigureAwait(false);
.GetIntegerDetailsAsync(flag.Key, int.Parse(flag.DefaultValue), this.State.EvaluationContext)
.ConfigureAwait(false);
break;
case FlagType.String:
this.State.FlagEvaluationDetailsResult = await this.State.Client!.GetStringDetailsAsync(flag.Key, flag.DefaultValue)
this.State.FlagEvaluationDetailsResult = await this.State.Client!
.GetStringDetailsAsync(flag.Key, flag.DefaultValue, this.State.EvaluationContext)
.ConfigureAwait(false);
break;
case FlagType.Object:
var defaultStructure = JsonStructureLoader.ParseJsonValue(flag.DefaultValue);
this.State.FlagEvaluationDetailsResult = await this.State.Client!
.GetObjectDetailsAsync(flag.Key, new Value(defaultStructure), this.State.EvaluationContext)
.ConfigureAwait(false);
break;
}
Expand Down Expand Up @@ -176,83 +213,6 @@ private void InitializeContext(string level, EvaluationContext context)
}
}

private static readonly IDictionary<string, Flag> E2EFlagConfig = new Dictionary<string, Flag>
{
{
"metadata-flag", new Flag<bool>(
variants: new Dictionary<string, bool> { { "on", true }, { "off", false } },
defaultVariant: "on",
flagMetadata: new ImmutableMetadata(new Dictionary<string, object>
{
{ "string", "1.0.2" }, { "integer", 2 }, { "float", 0.1 }, { "boolean", true }
})
)
},
{
"boolean-flag", new Flag<bool>(
variants: new Dictionary<string, bool> { { "on", true }, { "off", false } },
defaultVariant: "on"
)
},
{
"string-flag", new Flag<string>(
variants: new Dictionary<string, string>() { { "greeting", "hi" }, { "parting", "bye" } },
defaultVariant: "greeting"
)
},
{
"integer-flag", new Flag<int>(
variants: new Dictionary<string, int>() { { "one", 1 }, { "ten", 10 } },
defaultVariant: "ten"
)
},
{
"float-flag", new Flag<double>(
variants: new Dictionary<string, double>() { { "tenth", 0.1 }, { "half", 0.5 } },
defaultVariant: "half"
)
},
{
"object-flag", new Flag<Value>(
variants: new Dictionary<string, Value>()
{
{ "empty", new Value() },
{
"template", new Value(Structure.Builder()
.Set("showImages", true)
.Set("title", "Check out these pics!")
.Set("imagesPerPage", 100).Build()
)
}
},
defaultVariant: "template"
)
},
{
"context-aware", new Flag<string>(
variants: new Dictionary<string, string>() { { "internal", "INTERNAL" }, { "external", "EXTERNAL" } },
defaultVariant: "external",
(context) =>
{
if (context.GetValue("fn").AsString == "Sulisław"
&& context.GetValue("ln").AsString == "Świętopełk"
&& context.GetValue("age").AsInteger == 29
&& context.GetValue("customer").AsBoolean == false)
{
return "internal";
}
else return "external";
}
)
},
{
"wrong-flag", new Flag<string>(
variants: new Dictionary<string, string>() { { "one", "uno" }, { "two", "dos" } },
defaultVariant: "one"
)
}
};

public class BeforeHook : Hook
{
private readonly EvaluationContext context;
Expand Down
14 changes: 14 additions & 0 deletions test/OpenFeature.E2ETests/Steps/EvaluationStepDefinitions.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -255,4 +255,18 @@ public void Giventhereasonshouldindicateanerrorandtheerrorcodeshouldindicateatyp
Assert.Equal(Reason.Error, result?.Reason);
Assert.Equal(errorCode, result?.ErrorType.GetDescription());
}

[When(@"a non-existent string flag with key ""(.*)"" is evaluated with details and a fallback value ""(.*)""")]
public async Task WhenANon_ExistentStringFlagWithKeyIsEvaluatedWithDetailsAndAFallbackValue(string flagKey, string defaultValue)
{
this.State.Flag = new FlagState(flagKey, defaultValue.ToString(), FlagType.String);
this.State.FlagEvaluationDetailsResult = await this.State.Client!.GetStringDetailsAsync(flagKey, defaultValue).ConfigureAwait(false);
}

[When(@"a string flag with key ""(.*)"" is evaluated as an integer, with details and a fallback value (.*)")]
public async Task WhenAStringFlagWithKeyIsEvaluatedAsAnIntegerWithDetailsAndAFallbackValue(string flagKey, int defaultValue)
{
this.State.Flag = new FlagState(flagKey, defaultValue.ToString(), FlagType.Integer);
this.State.FlagEvaluationDetailsResult = await this.State.Client!.GetIntegerDetailsAsync(flagKey, defaultValue).ConfigureAwait(false);
}
Copy link
Member

@aepfli aepfli Oct 31, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those are part of the deprecated feature files, annotated with @deprecated-tag. As we are not planning to support them in the future, and they are covered with other newer gherkin tests, should we care about those, or should we simply ignore those steps for now?

Copy link
Contributor Author

@kylejuliandev kylejuliandev Nov 1, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed these in d0e3580. I think when I was trying to get the e2e tests initially updated I went through and added all the step definitions. I suspect I added these before I added the exclusion on @deprecated.

}
Loading
Loading