Skip to content

Commit d504da2

Browse files
ErikSchierboomErikSchierboom
authored
Add generator based on templates (#2350)
* Add initial structure for new generators * More work * Work on template * More work * Fix ordering * Fix quotes * Fix * Command-line arg * Allow "overloading" * Simplify template * Fix * Switch to handlebars * Remove deprecated generators * Fix * Workaround: IDEs use a different current directory than dotnet run * Sync prob-specs-repo * Auto-quote strings * Add pangram * Support helpers * WOrk on transformer * Add transformer * Another fix * Add solution * Minor refactoring * Refactor * Refactor * Include exercises as content * Include prob-specs * Introduce vars * Add sum-of-multiples * Add more generators * Add two-fer * add space-age * Add square-root * Add sieve * Add hamming * Allow helper without prefix * Add eliuds-eggs * Use newtonsoft.system.json * Use invariant culture * Remove file * Refactoring * Minor refactoring * Try refactor * Move to expandoobject * Extract naming * Remove unused * Add literal formatting * Refactor * Simplify * Add rotational-cipher * Fix test naming * Update method names * Fix skips * Fix rot cipher * Fix imports * Build both generators * Deprecate old one * Update add exercise [no important files changed]
1 parent f2d0ff6 commit d504da2

File tree

202 files changed

+982
-521
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

202 files changed

+982
-521
lines changed

‎bin/add-practice-exercise.ps1‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -38,25 +38,28 @@ $project = "${exerciseDir}/${ExerciseName}.csproj"
3838
# Update project packages
3939
& dotnet remove $project package coverlet.collector
4040
& dotnet add $project package Exercism.Tests --version 0.1.0-beta1
41-
& dotnet add $project package xunit.runner.visualstudio --version 2.4.3
42-
& dotnet add $project package xunit --version 2.4.1
43-
& dotnet add $project package Microsoft.NET.Test.Sdk --version 16.8.3
41+
& dotnet add $project package xunit.runner.visualstudio --version 3.0.1
42+
& dotnet add $project package xunit --version 2.8.1
43+
& dotnet add $project package Microsoft.NET.Test.Sdk --version 17.12.0
4444

4545
# Remove and update files
4646
Remove-Item -Path "${exerciseDir}/UnitTest1.cs"
4747
(Get-Content -Path ".editorconfig") -Replace "\[\*\.cs\]", "[${exerciseName}.cs]" | Set-Content -Path "${exerciseDir}/.editorconfig"
4848

4949
# Add and run generator (this will update the tests file)
50-
$generator = "generators/Exercises/Generators/${ExerciseName}.cs"
50+
$generator = "${exerciseDir}/.meta/Generator.tpl"
5151
Add-Content -Path $generator -Value @"
52-
using System;
52+
using Xunit;
5353
54-
using Exercism.CSharp.Output;
55-
56-
namespace Exercism.CSharp.Exercises.Generators;
57-
58-
internal class ${exerciseName} : ExerciseGenerator
54+
public class ${exerciseName}Tests
5955
{
56+
{{#test_cases}}
57+
[Fact{{#unless @first}}(Skip = "Remove this Skip property to run this test"){{/unless}}]
58+
public void {{test_method_name}}()
59+
{
60+
// TODO: implement the test
61+
}
62+
{{/test_cases}}
6063
}
6164
"@
6265
& dotnet run --project generators --exercise $Exercise

‎bin/generate-tests.ps1‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
<#
2+
.SYNOPSIS
3+
Generate the tests for exercises
4+
.DESCRIPTION
5+
Generate the tests for exercises that have a template.
6+
The tests are generated from canonical data.
7+
.PARAMETER Exercise
8+
The slug of the exercise to generate the tests for (optional).
9+
.EXAMPLE
10+
The example below will generate the tests for exercises with a template
11+
PS C:\> ./test.ps1
12+
.EXAMPLE
13+
The example below will generate the tests for the specified exercise
14+
PS C:\> ./test.ps1 acronym
15+
#>
16+
17+
[CmdletBinding(SupportsShouldProcess)]
18+
param (
19+
[Parameter(Position = 0, Mandatory = $false)]
20+
[string]$Exercise
21+
)
22+
23+
$ErrorActionPreference = "Stop"
24+
$PSNativeCommandUseErrorActionPreference = $true
25+
26+
if ($Exercise) {
27+
dotnet run --project generators --exercise $Exercise
28+
} else {
29+
dotnet run --project generators
30+
}

‎bin/test.ps1‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -130,6 +130,7 @@ function Parse-Exercises {
130130
function Build-Generators {
131131
Write-Output "Build generators"
132132
& dotnet build generators
133+
& dotnet build generators.deprecated
133134
}
134135

135136
function Test-Refactoring-Exercise-Default-Implementations {

‎docs/GENERATORS.md‎

Lines changed: 31 additions & 239 deletions
Original file line numberDiff line numberDiff line change
@@ -1,262 +1,54 @@
11
# Test generators
22

3-
Test generators allow tracks to generate tests automatically without having to write them ourselves. Each test generator reads from the exercise's `canonical data`, which defines the name of the test, its inputs, and outputs. You can read more about exercism's approach to test suites [here](https://github.com/exercism/problem-specifications#test-data-canonical-datajson).
3+
The C# track uses a [test generator](https://exercism.org/docs/building/tooling/test-generators) to auto-generate practice exercise tests.
4+
It uses the fact that most exercises defined in the [problem-specifications repo](https://github.com/exercism/problem-specifications/) also have a `canonical-data.json` file, which contains standardized test inputs and outputs that can be used to implement the exercise.
45

5-
Generating tests automatically removes any sort of user error when creating tests. Furthermore, we want the tests to be accurate with respect to its canonical data. Test generation also makes it much easier to keep tests up to date. As the canonical data changes, the tests will be automatically updated when the generator for that test is run.
6+
## Steps
67

7-
An example of a canonical data file can be found [here](https://github.com/exercism/problem-specifications/blob/master/exercises/bob/canonical-data.json)
8+
To generate a practice exercise's tests, the test generator:
89

9-
## Common terms
10+
1. Reads the exercise's test cases from its `canonical-data.json` file
11+
2. Uses `tests.toml` file to omit and excluded test cases
12+
3. Renders the test cases using the exercise's generator template
13+
4. Format the rendered template using Roslyn
14+
5. Writes the formatted template to the exercise's test file
1015

11-
When looking through the canonical data and the generator code base, we use a lot of common terminology. This list hopefully clarifies what they represent.
16+
### Step 1: read `canonical-data.json` file
1217

13-
- Canonical Data - Represents the entire test suite.
14-
- Canonical Data Case - A representation of a single test case.
15-
- Description - The name of the test.
16-
- Property - The method to be called when running the test.
17-
- Input - The input for the test case.
18-
- Expected - The expected value when running the test case.
18+
The test generator parses the test cases from the exercise's `canonical-data.json` using the [JSON.net library](https://www.newtonsoft.com/json).
1919

20-
## Adding a simple generator
20+
Since some canonical data uses nesting, the parsed test case includes an additional `path` field that contains the `description` properties of any parent elements, as well as the test case's own `description` property.
2121

22-
Adding a test generator is straightforward. Simply add a new file to the `Exercises/Generators` folder with the name of the exercise (in PascalCase), and create a class that extends the `GeneratorExercise` class.
22+
Note: the JSON is parsed to an `ExpandoObject` instance, which makes dealing with dynamic data easier.
2323

24-
An example of a simple generator would be the Bob exercise. The source is displayed below, but you can freely view it in the repository [here](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/Bob.cs).
24+
### Step 2: omit excluded tests from `tests.toml` file
2525

26-
```csharp
27-
namespace Exercism.CSharp.Exercises.Generators
28-
{
29-
public class Bob : GeneratorExercise
30-
{
31-
}
32-
}
33-
```
26+
Each exercise has a `tests.toml` file, in which individual tests can be excluded/disabled.
27+
The test generator will remove any test cases that are marked as excluded (`include = false`).
3428

35-
This is a fully working generator, no other code needs to be written! However, it's simplicity stems from the fact that the test suite and the program itself are relatively trivial.
29+
### Step 3: render the test cases
3630

37-
## Adding a complex generator
31+
The (potentially transformed) test cases are then passed to the `.meta/Generator.tpl` file, which defines how the tests should be rendered based on those test cases.
3832

39-
When the generator's default output is not sufficient, you can override the `GeneratorExercise` class' virtual methods to override the default behavior.
33+
### Step 4: format the rendered template using Roslyn
4034

41-
### Method 1: UpdateTestMethod(TestMethod testMethod)
35+
The rendered template is then formatted using [Roslyn](https://github.com/dotnet/roslyn).
36+
This has the following benefits:
4237

43-
Update the test method that described the test method being generated. When you are required to customize a test generator, overriding this method is virtually always what you want to do.
38+
- Exercises are formatted consistently
39+
- You're not required to worry much about whitespace and alignment when writing templates
4440

45-
There are many things that can be customized, of which we'll list the more common usages.
41+
### Step 5: write the rendered template to the exercise's test file
4642

47-
#### Customize test data
43+
Finally, the output of the rendered template is written to the exercise's test file.
4844

49-
It is not uncommon that a generator has to transform its input data or expected value to a different value/representation.
45+
## Templates
5046

51-
An example of this is the [matching-brackets](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/MatchingBrackets.cs) generator, which has a `"value"` input value, which is of type `string`. However, this `string` value contains a backslash, which needs to escaped in order for it to be rendered correctly:
47+
The templates are rendered using the [Handlebars.Net library](https://github.com/Handlebars-Net/Handlebars.Net), which supports [handlebars syntax](https://handlebarsjs.com/).
5248

53-
```csharp
54-
protected override void UpdateTestMethod(TestMethod testMethod)
55-
{
56-
testMethod.Input["value"] = testMethod.Input["value"].Replace("\\", "\\\\");
57-
// [...]
58-
}
59-
```
49+
## Command-line interface
6050

61-
Another common use case is to handle empty arrays. If an array is empty, its type will default to `JArray`, which doesn't have any type information. To allow the generator to output a correctly typed array, we have to convert the `JArray` to an array first.
51+
There are two ways in which the test generator can be run:
6252

63-
An example of this is the [proverb](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/Proverb.cs) generator, which converts the `JArray` to an empty `string` array:
64-
65-
```csharp
66-
protected override void UpdateTestMethod(TestMethod testMethod)
67-
{
68-
// [...]
69-
70-
if (testMethod.Input["strings"] is JArray)
71-
testMethod.Input["strings"] = Array.Empty<string>();
72-
73-
if (testMethod.Expected is JArray)
74-
testMethod.Expected = Array.Empty<string>();
75-
}
76-
```
77-
78-
#### Output test data as variables
79-
80-
Sometimes, it might make sense to not define a test method's data inline, but as variables.
81-
82-
An example of this is the [crypto-square](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/CryptoSquare.cs) generator, which indicates that both the test method input as well as the expected value, should be stored in variables:
83-
84-
```csharp
85-
protected override void UpdateTestMethod(TestMethod testMethod)
86-
{
87-
testMethod.UseVariablesForInput = true;
88-
testMethod.UseVariableForExpected = true;
89-
}
90-
```
91-
92-
#### Custom tested method type
93-
94-
By default, the generator will test a static method. However, you can also test for instance methods, extension methods, properties and constructors.
95-
96-
An example of this is the [roman-numerals](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/RomanNumerals.cs) generator, which indicates that it tests an extensions method:
97-
98-
```csharp
99-
protected override void UpdateTestMethod(TestMethod testMethod)
100-
{
101-
testMethod.TestedMethodType = TestedMethodType.ExtensionMethod;
102-
testMethod.TestedMethod = "ToRoman";
103-
}
104-
```
105-
106-
#### Change names used
107-
108-
As we saw in the previous example, you can also customize the name of the tested method. You are also allowed to customize the tested class' name and the test method name.
109-
110-
An example of this is the [triangle](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/Triangle.cs) generator, which by default generates duplicate test method names (which will be a compile-time error), but instead uses the `TestMethodNameWithPath` to use the full path as the test method name (effectively making the test method name unique):
111-
112-
```csharp
113-
protected override void UpdateTestMethod(TestMethod testMethod)
114-
{
115-
// [...]
116-
testMethod.TestMethodName = testMethod.TestMethodNameWithPath;
117-
// [...]
118-
}
119-
```
120-
121-
#### Test for an exception being thrown
122-
123-
Some test methods want to verify that an exception is being thrown.
124-
125-
An example of this is the [rna-transcription](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/RnaTranscription.cs) generator, which defines that some of its test methods should throw an `ArgumentException`:
126-
127-
```csharp
128-
protected override void UpdateTestMethod(TestMethod testMethod)
129-
{
130-
if (testMethod.Expected is null)
131-
testMethod.ExceptionThrown = typeof(ArgumentException);
132-
}
133-
```
134-
135-
Note that `ArgumentException` type's namespace will be automatically added to the list of namespaces used in the test class.
136-
137-
#### Custom input/constructor parameters
138-
139-
In some cases, you might want to override the parameters that are used as input parameters.
140-
141-
An example of this is the [two-fer](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/TwoFer.cs) generator, which does not use any input parameters when the `"name"` input parameter is set to `null`:
142-
143-
```csharp
144-
protected override void UpdateTestMethod(TestMethod testMethod)
145-
{
146-
// [...]
147-
148-
if (testMethod.Input["name"] is null)
149-
testMethod.InputParameters = Array.Empty<string>();
150-
}
151-
```
152-
153-
If a test method tests an instance method, you can also specify which parameters to use as constructor parameters (the others will be input parameters, unless specified otherwise).
154-
155-
An example of this is the [matrix](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/Matrix.cs) generator, which specifies that the `"string"` parameter should be passed as a constructor parameter:
156-
157-
```csharp
158-
protected override void UpdateTestMethod(TestMethod testMethod)
159-
{
160-
testMethod.TestedMethodType = TestedMethodType.InstanceMethod;
161-
testMethod.ConstructorInputParameters = new[] { "string" };
162-
}
163-
```
164-
165-
#### Custom arrange/act/assert code
166-
167-
Although this should be used as a last resort, some generators might want to skip the default generation completely and control which arrange, act or assert code the test method should contain.
168-
169-
An example of this is the [run-length-encoding](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/RunLengthEncoding.cs) generator, which uses a custom assertion for one specific property:
170-
171-
```csharp
172-
protected override void UpdateTestMethod(TestMethod testMethod)
173-
{
174-
// [...]
175-
176-
if (testMethod.Property == "consistency")
177-
testMethod.Assert = RenderConsistencyToAssert(testMethod);
178-
}
179-
180-
private string RenderConsistencyToAssert(TestMethod testMethod)
181-
{
182-
var expected = Render.Object(testMethod.Expected);
183-
var actual = $"{testMethod.TestedClass}.Decode({testMethod.TestedClass}.Encode({expected}))";
184-
return Render.AssertEqual(expected, actual);
185-
}
186-
```
187-
188-
Note that the `Render` instance is used to render the assertion and the expected value.
189-
190-
### Method 2: UpdateNamespaces(ISet<string> namespaces)
191-
192-
Allows additional namespaces to be added to the test suite.
193-
194-
All tests use the `Xunit` framework, so each test class will automatically include the `Xunit` namespace. However, some test classes may require additional namespaces.
195-
196-
An example of this is the [gigasecond](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/Gigasecond.cs) generator, which uses the `DateTime` class in its test methods, and thus adds its namespace to the list of namespaces:
197-
198-
```csharp
199-
protected override void UpdateNamespaces(ISet<string> namespaces)
200-
{
201-
namespaces.Add(typeof(DateTime).Namespace);
202-
}
203-
```
204-
205-
Note that as mentioned before, the namespace of any thrown exception types are automatically added to the list of namespaces.
206-
207-
### Method 3: UpdateTestClass(TestClass testClass)
208-
209-
This method allows you to customize the output of the test class. Only in rare cases would you want to override this method. The most common use case to override this method, is to add additional (helper) methods to the test suite.
210-
211-
An example of this is the [tournament](https://github.com/exercism/csharp/blob/main/generators/Exercises/Generators/Tournament.cs) generator, which adds a helper method to the test suite:
212-
213-
```csharp
214-
protected override void UpdateTestClass(TestClass testClass)
215-
{
216-
AddRunTallyMethod(testClass);
217-
}
218-
219-
private static void AddRunTallyMethod(TestClass testClass)
220-
{
221-
testClass.AdditionalMethods.Add(@"
222-
private string RunTally(string input)
223-
{
224-
var encoding = new UTF8Encoding();
225-
226-
using (var inStream = new MemoryStream(encoding.GetBytes(input)))
227-
using (var outStream = new MemoryStream())
228-
{
229-
Tournament.Tally(inStream, outStream);
230-
return encoding.GetString(outStream.ToArray());
231-
}
232-
}");
233-
}
234-
```
235-
236-
Additional methods will be added to the bottom of the test suite.
237-
238-
## Updating Existing Files
239-
240-
It is possible that an existing exercise does not match the canonical data. It is OK to update the exercise stub and/or the exercise example to follow the canonical data! An example might be that an exercise is named SumOfMultiples, but the SumOfMultiples.cs and Example.cs files both use `Multiples` as the name of the class.
241-
242-
Also, if you find an issue with one of the existing generators or test suites simply open up the generator that you would like to update, make your changes, and then run the generators.
243-
244-
## Running The Generators
245-
246-
This repository is coded against [.NET Core](https://www.microsoft.com/net/core). To run the generators all you need to do is run the following command in the generators directory:
247-
248-
`dotnet run`
249-
250-
This command will take all of the generators that are in the `Exercises` folder, and generate all of the test cases for that exercise. We use reflection to get all of the exercises, so if you are adding a new test, the test will be automatically included when running the generator.
251-
252-
If you only need to run a single generator, you can do so by running the following command:
253-
254-
`dotnet run -e <exercise>`
255-
256-
Once the generator has been run, you can view the output of your generation by navigating to the test file for that exercise. As an example, the test suite for the Bob exercise can be found at:
257-
258-
`exercises/bob/BobTests.cs`
259-
260-
## Submitting A Generator
261-
262-
If you are satisfied with the output of your generator, we would love for you to submit a pull request! Please include your generator, updated test suite, and any other corresponding files that you may have changed.
53+
1. `bin/generate-tests.ps1`: generate the tests for all exercises that have a generator template
54+
2. `bin/generate-tests.ps1 -e <slug>`: generate the tests for the specified exercise, if it has a generator template

0 commit comments

Comments
 (0)