Skip to content

Commit f4dd574

Browse files
ahoppenahoppen
authored
Merge pull request #1993 from ahoppen/ahoppen/update-contributing
Improve CONTRIBUTING.md
2 parents 11b9042 + e8fc865 commit f4dd574

File tree

2 files changed

+56
-130
lines changed

2 files changed

+56
-130
lines changed

‎CONTRIBUTING.md

Lines changed: 28 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -1,109 +1,54 @@
11
# Contributing
22

3-
This guide refers to Swift Development Snapshots. These can be downloaded from [swift.org/download](https://swift.org/download/#snapshots).
4-
If you are seeing issues, it is generally a good idea to re-try with the latest Swift Development Snapshot.
3+
## Building & Testing
54

6-
## Building
5+
swift-syntax is a SwiftPM package, so you can build and test it using anything that supports packages - opening in Xcode, Visual Studio Code with [Swift for Visual Studio Code](https://github.com/swift-server/vscode-swift) installed, or through the command line using `swift build` and `swift test`.
76

8-
The easiest option to build SwiftSyntax is to open it in Xcode.
7+
> 💡 Tip: Running swift-syntax’s self-parse tests takes the majority of testing time. If you want to iterate quickly, you can skip these tests:
8+
> - In Xcode
9+
> 1. Product -> Scheme -> Edit Scheme…
10+
> 2. Select the Arguments tab in the Run section
11+
> 3. Add a `SKIP_LONG_TESTS` environment variable with value `1`
12+
> - On the command line: Set the `SKIP_LONG_TESTS` environment variable to `1` when running tests, e.g by running `SKIP_LONG_TESTS=1 swift test`
913
10-
Alternatively you can also build it from the command line using `build-script.py`. To do this, perform the following steps
11-
12-
1. Check `swift-syntax` and `swift-argument-parser` out side by side:
13-
```
14-
- (enclosing directory)
15-
- swift-argument-parser
16-
- swift-syntax
17-
```
18-
2. Execute the following command
19-
```bash
20-
swift-syntax/build-script.py build --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr
21-
```
14+
> 💡 Tip: If you are only modifying the `SwiftParser` module, you can run only the parser tests by selecting the `SwiftParserTest` target.
15+
> - In Xcode: Select the `SwiftParserTest` scheme. If you can’t find it in your Schemes, you need to manually add it using Product -> Scheme -> New Scheme…
16+
> - On the command line: Run `swift test --test-product SwiftParserTests`
2217
2318
## Formatting
2419

25-
SwiftSyntax is being formatted using [swift-format](http://github.com/apple/swift-format) to ensure a consistent style.
26-
27-
To format your changes run `format.py` at the root of this repository. If you have a `swift-format` executable ready, you can pass it to `format.py`. If you do not, `format.py` will build its own copy of `swift-format` in /tmp/swift-format.
28-
29-
CI will ensure that all hand-written source code is correctly formatted. Generated source code is not formatted to make it easier to spot changes when re-running code generation.
30-
31-
## Testing
32-
33-
Because of SwiftSyntax’s integration with the Swift compiler project, testing certain parts of the project is a little bit more involved than others.
34-
35-
The `SwiftParser` tests have no dependencies, other `XCTests` require you to use a matching Swift Development Snapshot, `lit`-based test require a compiler build.
36-
37-
Run the tests that you believe are necessary locally. CI will always run all test before allowing the PR to be merged.
38-
39-
### SwiftParser
20+
swift-syntax is formatted using [swift-format](http://github.com/apple/swift-format) to ensure a consistent style.
4021

41-
If you are only modifying the `SwiftParser` module, you can just run the tests using Xcode by testing the `SwiftParserTest` target.
22+
To format your changes run `format.py` at the root of this repository. If you have a `swift-format` executable ready, you can pass it to `format.py`. If you do not, `format.py` will build its own copy of `swift-format` in `/tmp/swift-format`.
4223

43-
If you can’t find it in your Schemes, you need to manually add it using Product -> Scheme -> New Scheme…
24+
If you are seeing surprising formatting results, you most likely have a `swift-format` installed on your system that’s not the most recent version built from the `main` branch. To fix this, clone [swift-format](http://github.com/apple/swift-format), build it using `swift build` and pass the freshly built executable to `format.py` as `--swift-format path/to/swift-format/.build/debug/swift-format`. Alternatively, you can uninstall `swift-format` on your system and `format.py` will build it from scratch.
4425

45-
### XCTests
26+
Generated source code is not formatted to make it easier to spot changes when re-running code generation.
4627

47-
To run the tests in Xcode, select the SwiftSyntax-Package scheme and hit Product -> Test.
48-
49-
You can also run the tests from the command line using
50-
```bash
51-
./build-script.py test --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr --skip-lit-tests
52-
```
28+
## Generating Source Code
5329

54-
If you are seeing issues regarding a mismatched parser library, try the following
55-
1. Update your Swift Development Snapshot
56-
2. Revert your swift-syntax checkout to the date of your Swift Development Snapshot
57-
3. Wait for a new Swift Development Snapshot
58-
4. If the above options are not possible, build your own Swift toolchain locally and use that toolchain as the `--toolchain` parameter for SwiftSyntax’s `build-script.py` invocations
59-
- Note: Building your own toolchain will take more than 1 hour and even longer if you are running into any issues. We do not recommend building your own Swift toolchain unless you are familiar with Swift compiler development.
30+
If you want to modify the generated files, open the [CodeGeneration](CodeGeneration) package and run the `generate-swift-syntax` executable.
6031

61-
Tip: Running SwiftSyntax’s self-parse tests takes the majority of testing time. If you want to iterate quickly, you can skip these tests using the following steps:
62-
1. Product -> Scheme -> Edit Scheme…
63-
2. Select the Arguments tab in the Run section
64-
3. Add a `SKIP_LONG_TESTS` environment variable with value `1`
32+
## Additional Verification
6533

66-
### Additional Verification
34+
swift-syntax has additional verification methods (see the sections below) that provide more extensive validation. They have a significant runtime impact on swift-syntax and are thus not enabled by default when building swift-syntax, but are enabled in CI. If CI fails and you are unable to reproduce the failure locally, make sure that `SKIP_LONG_TESTS` is not set and try enabling these validations.
6735

68-
swift-syntax has two additional verification methods (see the two sections below) that provide more extensive validation. They have significant runtime impact on swift-syntax and are thus not enabled by default when building swift-syntax, but are enabled in CI. If CI fails and you are unable to reproduce the failure locally, make sure that `SKIP_LONG_TESTS` is not set and try enabling these validations.
36+
### RawSyntax Validation
6937

70-
#### RawSyntax Validation
38+
When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax, SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that
39+
1. every child of a syntax node has the correct kind, which should be guaranteed by the Swift type system in most cases
40+
2. each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package.
7141

72-
When the `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` environment variable is set while building swift-syntax (or the check for that variable has been changed to always return `true` in Package.swift), SwiftSyntax will perform additional validation that the layout of the syntax tree is correct. It validates that every child of a syntax node has the correct kind (which should be guaranteed by the Swift type system in most cases) and, more importantly, validates that each token only has one of the token kinds that is specified in the syntax tree layout of the `CodeGeneration` package.
42+
If this validation hits an assertion failure that a token is not accepted at a certain position in the syntax tree, double-check if the token kind that is being stored in the syntax tree actually makes sense here. If it does not, check if there is a parser bug or whether you need to remap the token kind. If it does make sense, add the token kind to `.token(choices:)` of the syntax node in CodeGeneration, re-generate that source code and run tests again.
7343

74-
If this validation hits an assertion failure that a token is not accepted at a certain position in the syntax tree, double check if the token kind that is being stored in the syntax tree actually makes sense here. If it does not, check if there is a parser bug or whether you need to remap the token kind. If it does make sense, add the token kind to `.token(choices:)` of the syntax node in CodeGeneration, re-generate that source code and run tests again.
44+
> 💡 Tip: If you want to run tests in Xcode with RawSyntax validation enabled, you can also modify Package.swift and replace the check for `SWIFTSYNTAX_ENABLE_RAWSYNTAX_VALIDATION` with `true`.
7545
76-
#### Test Case Mutation
46+
### Test Case Mutation
7747

78-
When the `SWIFTPARSER_ENABLE_ALTERNATE_TOKEN_INTROSPECTION` environment variable is set while building swift-syntax (or the check for that variable has been changed to always return `true` in Package.swift), SwiftParser records alternative tokens that the parser was looking for at specific offsets in the source file (e.g. whether it also checked for a `struct` keyword when the source code contained a `class` keyword). It will then use that information to mutate the test case by e.g. substituting `class` by `struct`.
48+
When the `SWIFTPARSER_ENABLE_ALTERNATE_TOKEN_INTROSPECTION` environment variable is set while building swift-syntax, SwiftParser records alternative tokens that the parser was looking for at specific offsets in the source file, e.g. whether it also checked for a `struct` keyword when the source code contained a `class` keyword. It will then use that information to mutate the test case by e.g. substituting `class` with `struct`.
7949

8050
When testing finds one of these failures, it will show you the syntax tree that produced the failure. Create a new test case with the source code the failure gives you and fix the failure.
8151

82-
### `lit`-based Tests
83-
84-
A few tests are based LLVM’s `lit` and `FileCheck` tools.
85-
To run these, build `FileCheck`, e.g. by building the Swift compiler and run the tests using the following command:
86-
```bash
87-
./build-script.py test --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr --filecheck-exec /path/to/build/Release+Asserts/llvm-macosx-x86_64/bin/FileCheck
88-
```
89-
90-
## Generating Source Code
91-
92-
If you want to modify the code-generated files, perform the following steps:
93-
94-
1. Check out `swift` next to `swift-syntax`
95-
```
96-
- (enclosing directory)
97-
- swift
98-
- swift-argument-parser
99-
- swift-syntax
100-
```
101-
2. Run the following command
102-
```bash
103-
swift-syntax/build-script.py generate-source-code --toolchain /Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-<recent date>.xctoolchain/usr
104-
```
105-
3. The new source-generated file will be written into your `Sources` directory.
106-
10752
## Swift Version
10853

109-
We require that SwiftSyntax builds with the latest released compiler and the previous major version (e.g. with Swift 5.8 and Swift 5.7).
54+
We require that swift-syntax builds with the latest released compiler and the previous major version (e.g. with Swift 5.8 and Swift 5.7).

‎Sources/SwiftSyntax/Documentation.docc/Contributing/ChangingSwiftSyntax.md

Lines changed: 28 additions & 47 deletions
Original file line numberDiff line numberDiff line change
@@ -12,31 +12,27 @@ structures in sync, this project generates code using the Swift package
1212
## Regenerating Files
1313

1414
To re-generate the files after changing `CodeGeneration` run the `generate-swiftsyntax`
15-
target of `CodeGeneration` and pass `path/to/swift-syntax/Sources` as the argument.
15+
target of `CodeGeneration`.
1616

1717
On the command line, this would be
1818
```bash
19-
swift run --package-path CodeGeneration generate-swiftsyntax Sources
19+
swift run --package-path CodeGeneration generate-swiftsyntax
2020
```
2121

22-
Or if you open the `CodeGeneration` package in Xcode, you can add the
23-
`generate-swift syntax ../Sources` arguments using Product -> Scheme -> Edit Scheme…
24-
2522
## Adding and Removing Syntax Nodes
2623

2724
The files containing the definition of all of the syntax nodes are available
28-
under the [SyntaxSupport][SyntaxSupport] directory. These files
25+
under the [SyntaxSupport](SyntaxSupport) directory. These files
2926
are roughly divided according to broad syntactic categories in the Swift
3027
programming language. That is, the syntax nodes for classes, structs, and actors
31-
are defined in `DeclNodes.py`, while the syntax nodes for string literals,
32-
arrays, and tuples is defined in `ExprNodes.py`.
28+
are defined in `DeclNodes.swift`, while the syntax nodes for string literals,
29+
arrays, and tuples is defined in `ExprNodes.swift`.
3330

3431
To add a node to these files, it can be helpful to copy an existing node and
3532
alter its definition to suit the needs of the new syntax being defined. A syntax
36-
node consists of a name, a kind, and a list of child nodes. The name of a node
37-
not only identifies it uniquely to Swift Syntax, it also provides a handle that
38-
other nodes can use to refer to it. The kind of a syntax node defines the
39-
class of syntax the node belongs to. All nodes are at least of the `Syntax`
33+
node consists of a kind (which also defines the node’s name), a base kind, and a list of
34+
child nodes. The base kind of a syntax node defines the class of syntax the node belongs to.
35+
All nodes are at least of the `Syntax`
4036
kind, though some nodes may have a more specific kind like `Stmt` for
4137
statements or `Expr` for expressions. The SwiftSyntax library expands these
4238
kinds into protocol conformances and allows for casting strongly typed syntax
@@ -45,33 +41,29 @@ nodes among these categories for easier processing.
4541
The node for a source file is reproduced below:
4642

4743
```swift
48-
Node(
49-
name: "SourceFile",
50-
nameForDiagnostics: "source file",
51-
kind: "Syntax",
52-
traits: [
53-
"WithStatements"
54-
],
55-
parserFunction: "parseSourceFile",
56-
children: [
57-
Child(
58-
name: "Statements",
59-
kind: .collection(kind: "CodeBlockItemList", collectionElementName: "Statement")
60-
),
61-
Child(
62-
name: "EOFToken",
63-
kind: .node(kind: "EOFToken")
64-
),
65-
]
66-
),
67-
44+
Node(
45+
kind: .sourceFile,
46+
base: .syntax,
47+
nameForDiagnostics: "source file",
48+
parserFunction: "parseSourceFile",
49+
traits: ["WithStatements"],
50+
children: [
51+
Child(
52+
name: "Statements",
53+
kind: .collection(kind: .codeBlockItemList, collectionElementName: "Statement")
54+
),
55+
Child(
56+
name: "EndOfFileToken",
57+
deprecatedName: "EOFToken",
58+
kind: .token(choices: [.token(tokenKind: "EndOfFileToken")])
59+
),
60+
]
61+
)
6862
```
6963

7064
## Committing Changes
7165

72-
When it is time to commit changes to the Swift Syntax repository, most cases
73-
of adding syntax will require a single PR to swift-syntax. Added
74-
syntactic elements will require corresponding changes to the included
66+
Added syntactic elements will require corresponding changes to the included
7567
SwiftParser library. For an introduction on parsing Swift nodes, see
7668
[the article on Parsing Basics][ParserBasics].
7769

@@ -82,21 +74,10 @@ SwiftParser library. For an introduction on parsing Swift nodes, see
8274

8375
When updating nodes, certain clients of SwiftSyntax that are relying upon those
8476
nodes will need to be changed in tandem. For example, the
85-
[swift-stress-tester][swift-stress-tester] uses SwiftSyntax, and the CI/CD
77+
[swift-stress-tester][swift-stress-tester] uses SwiftSyntax, and the CI
8678
system will not allow changes to SwiftSyntax that break `swift-stress-tester`
8779
without a paired change to that repository.
8880

89-
When adding nodes or removing existing nodes, it is often the case that you
90-
will need a paired PR to the [Swift Compiler repository][Swiftc] as well. This
91-
step currently involves some amount of duplication with the SwiftParser library
92-
and is a much more involved process. [The Swift Forums][CompilerDev] are a
93-
great source for guidance on how to extend the C++ compiler to accommodate new
94-
syntax and semantics for the Swift programming language.
95-
96-
[LiterateProgramming]: https://en.wikipedia.org/wiki/Literate_programming
97-
[SwiftToolchains]: https://www.swift.org/download/
9881
[SyntaxSupport]: https://github.com/apple/swift-syntax/tree/main/CodeGeneration/Sources/SyntaxSupport
9982
[swift-stress-tester]: https://github.com/apple/swift-stress-tester
100-
[Swiftc]: https://github.com/apple/swift
10183
[ParserBasics]: https://github.com/apple/swift-syntax/tree/main/Sources/SwiftParser/SwiftParser.docc/ParsingBasics.md
102-
[CompilerDev]: https://forums.swift.org/c/development/compiler/

0 commit comments

Comments
 (0)