Pass 1 of SilkTouch Scraper - Joint Infrastructure and XML Generation (#559)

Author: Perksey

.gitmodules

@@ -1,9 +1,3 @@
-[submodule "build/submodules/SDL"]
-	path = build/submodules/SDL
-	url = https://github.com/libsdl-org/SDL
-[submodule "build/submodules/Vulkan-Headers"]
-	path = build/submodules/Vulkan-Headers
-	url = https://github.com/KhronosGroup/Vulkan-Headers
-[submodule "build/submodules/GLFW"]
-	path = build/submodules/GLFW
-	url = https://github.com/glfw/glfw.git
+[submodule "src/submodules/Vulkan-Headers"]
+	path = src/submodules/Vulkan-Headers
+	url = https://github.com/KhronosGroup/Vulkan-Headers.git

Silk.NET.sln

@@ -0,0 +1,13 @@
+# SilkTouch
+
+SilkTouch is the Silk.NET project's C# native interoperability code generation stack. It contains useful generators which together generate the majority of the library's codebase.
+
+It does this by splitting the generation process into multiple phases, all anchored around the concept of a "shared infrastructure" - abstractions for SilkTouch generators which are agnostic of where they're running. Currently the only places SilkTouch can run is in a Roslyn Source Generator and using a `dotnet tool` Command Line Interface, but the idea is so long as no incompatibilities are introduced by each individual phase (such as being locked to .NET 6 instead of .NET Standard 2.0) the generator should be able to run in whatever form factor is desirable - the Shared Infrastructure will provide everything a generator could need for consuming a JSON configuration with C# input code and producing additional generated C# code as a result.
+
+Learn more about each individual cornerstone of the SilkTouch Stack:
+- [Shared Infrastructure](shared-infrastructure.md)
+- [Emitter](emitter.md)
+- [Overloader](overloader.md)
+- [Scraper](scraper.md)
+
+TODO: Add a link to the original proposal once the proposals folder has been reorganised according to the Software Development Plan.

documentation/for-contributors/generators/README.md

@@ -0,0 +1,3 @@
+# Emitter
+
+The Emitter.

documentation/for-contributors/generators/emitter.md

@@ -0,0 +1,3 @@
+# Overloader
+
+The Overloader.

documentation/for-contributors/generators/overloader.md

@@ -0,0 +1,39 @@
+# Scraper
+
+The SilkTouch Scraper is responsible for walking through the abstract syntax tree (AST) of a C/C++ header and all the referenced headers therein to generate C# bindings to as much of this as possible where the configuration indicates this is desirable. Only certain headers will have bindings generated for them, which can be specified by providing the paths to the headers to "traverse" in the JSON configuration.
+
+## ClangSharp
+
+To walk through the C/C++ AST, ClangSharp's P/Invoke Generator is used, which is a proven tool for generating C# bindings to C/C++ APIs published by Microsoft. It does this by using the clang compiler to walk through the AST.
+
+ClangSharp's P/Invoke Generator is capable of producing C# bindings on its own, however these are raw bindings that use DllImport and don't aim to lift the bindings to be more C#-friendly. However, as part of Scraper development work was done to add an "XML output mode" to ClangSharp which instead of producing pure C# code, it outputs XML which roughly represents what the C# code would've looked like.
+
+## The Subagent
+
+ClangSharp uses libclang, which has funny rules over multithreading. To maintain a degree of parallelism during generation, the invocation of ClangSharp was moved to be done in a separate process. The original intention was to invoke the `ClangSharpPInvokeGenerator` tool as shipped publicly, but unfortunately `dotnet tool`s cannot be referenced by other libraries or `dotnet tool`s. As such, we created our own shim which mimics the behaviour of and duplicates a lot of code from the `ClangSharpPInvokeGenerator` tool, but within the same CLI executable.
+
+### The `ISubagent` abstraction
+
+`ISubagent` is responsible for launching the ClangSharp tool with certain parameters, inputs, and configuration. This is an abstraction such that the main Scraper assembly remains easily portable to another form factor should we enable the Scraper to run in another form factor in the future, as well as to prevent the library assembly knowing too much about the assembly in which its executing.
+
+The only implementation of this interface is the `ClangSharpSubagent` class in the `SilkTouch` assembly (CLI implementation of the Shared Infrastructure) which gathers information on the currently executing assembly, and relaunches it with `clangsharp` as the first argument and a JSON-encoded record as the second.
+
+### The `silktouch clangsharp` command
+
+This is the command that invokes ClangSharp as described in the previous section. This deserializes the JSON-encoded record and passes the data held therein to the `PInvokeGenerator` class within ClangSharp. This will generate the XML dump and then quit, possibly with some ClangSharp-generated diagnostics and a non-zero error code if those diagnostics are severe. The implementation of this is in the `ClangSharpHandoff` class.
+
+## XML Mods
+
+Once an initial XML dump has been generated by The Subagent, the XML dump will be parsed by the `Silk.NET.SilkTouch.ClangSharp.Xml` library and produce a tree of records. This is done in the `ScraperGenerator` ahead of the XML mods stage.
+
+The purpose of the XML mods stage is to walk this tree of records, and modify it depending on the project configuration (for example changing all names to be PascalCase, or adding custom attributes to cause the Overloader to generate certain overloads). Each mod must be specifically enabled, and has free range over the XML. As such, they execute in the order in which they were specified in the project configuration.
+
+Each XML mod has access to the `ScraperJobConfiguration.ModOptions` and the full XML tree. It does not have any access to resources beyond that, such as the SilkTouch Context.
+
+TODO: Write words here about each XML mod once XML mods have been implemented
+
+## Transformation
+
+After the XML mods have been applied, the transformation stage is responsible for converting the bindings represented by the XML tree back into C# code that can be output via the SilkTouch Context (and added to the project compilation). Unlike ClangSharp, this stage will output bindings which are compatible with/invoke the Emitter and Overloader instead of using DllImport directly.
+
+TODO: Write more words here once this stage is implemented.

documentation/for-contributors/generators/scraper.md

@@ -0,0 +1,39 @@
+# Shared Infrastructure
+
+The Shared Infrastructure contains common code and abstractions for bootstrapping and running generators within the SilkTouch Stack. It aims to ensure that SilkTouch generators are easily portable, and have a unified interface over multiple generation mediums such that no extra code is required in individual generators to accomodate new form factors.
+
+## "Form Factor"
+
+A "Form Factor", referred to throughout the documentation, is simply a distinct type of application which invokes the SilkTouch Stack via the Shared Infrastructure (i.e. "it's a place where SilkTouch runs"). Distinct types such as Roslyn Source Generators and Command Line Interfaces, which are the two first-party form factors supported today.
+
+## Configuration
+
+All JSON configuration structures are represented by C# records in the Silk.NET.SilkTouch.Configuration namespace (in the Configuration folder of Silk.NET.SilkTouch.Common).
+
+### Project Configuration
+
+Each project has its own JSON configuration file which defines what each generator does for that particular project. This file should be added by the user as an `AdditionalFiles` file, with the name `silktouch.json` unless the `silktouch_config_file` `.editorconfig` option is used to override this with a new file name or file path (either absolute or relative). 
+
+Each generator has its own specific configuration structure, which is outlined in XML documentation in the `Config.cs` file as well as the markdown documents for each individual generator.
+
+The project configuration also provides a (relative or absolute) path to a file containing the global configuration.
+
+### Global Configuration
+
+The global configuration contains variables which aren't specific to a particular project or generation process/job. Stuff like the License Header, which doesn't need to be duplicated across every single project configuration.
+
+## Generation
+
+### SilkTouch "Context"
+
+The SilkTouch "Context" (represented by the `SilkTouchContext` class in the `Silk.NET.SilkTouch.Generation` namespace, or Generation folder of Silk.MET.SilkTouch.Common) encapsulates all inputs and outputs passed into and retrieved from a generator running atop the Shared Infrastructure. It exists to isolate each individual generator from one another, and provide one unified data structure containing everything a generator might reasonably need to access about a project. 
+
+Some SilkTouch Contexts have access to outputs generated by other generators in the `AdditionalSyntaxTrees` property. To ease context consumption, there is a `AllSyntaxTrees` property which will contain both the Roslyn `Compilation`'s syntax trees as well as the newly-added syntax trees. Currently this is only the Emitter and Overloader if the Scraper has ran before them, though it is intended that the Shared Infrastructure will be expanded to have a full-blown dependency resolution mechanism to allow more generator modularity and dynamic definition of a generator's soft or hard dependencies.
+
+### The Generator Object
+
+The Generator Object (represented by the `SilkTouchGenerator` class) is responsible for the consumption of basic project information, and morphing/storing this information in a way that allows for Contexts to be created from the generator object. After a context is created, it will be consumed by a generator, and then once again "ingested" by the generator object. This generator object forwards the results to bindable events, which are subscribed to by the form factor.
+
+The intention of this is to reduce duplicated code between the two form factors we maintain, and provide an interface to ease the bootstrapping of SilkTouch code generation - this class significantly reduces the amount of work required to implement another form factor.
+
+Currently there are no methods on `SilkTouchGenerator` which can create a `SilkTouchContext` directly, nor are there APIs which nicely facilitate the passing of generated outputs to a stage depending on them. Both of which are things we'd like to address in the future. 

documentation/for-contributors/generators/shared-infrastructure.md

@@ -18,6 +18,11 @@
         <SilkDocumentationDir>$(MSBuildThisFileDirectory)../documentation</SilkDocumentationDir>
         <SilkTestsDir>$(MSBuildThisFileDirectory)../documentation</SilkTestsDir>
     </PropertyGroup>
+    <!-- Compiler Options -->
+    <PropertyGroup>
+        <Nullable>enable</Nullable>
+        <SilkDocumentationWarningsAsErrors>CS1591;CS1573</SilkDocumentationWarningsAsErrors>
+    </PropertyGroup>
     <!-- NuGet -->
     <PropertyGroup>
         <PackageIcon>logo.png</PackageIcon>

src/Directory.Build.props

@@ -1,4 +1,4 @@
-<Project>
+<Project InitialTargets="PostImportDefinitions">
     <!-- NuGet Stuff -->
     <ItemGroup>
         <Content Include="$(MSBuildThisFileDirectory)..\documentation\branding\logo.png"
@@ -9,4 +9,10 @@
             <CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
         </Content>
     </ItemGroup>
+    <!-- Force Documentation -->
+    <Target Name="PostImportDefinitions">
+        <PropertyGroup Condition="'$(SilkDocumentationExempt)' != 'true'">
+            <WarningsAsErrors>$(SilkDocumentationWarningsAsErrors);$(WarningsAsErrors)</WarningsAsErrors>
+        </PropertyGroup>
+    </Target>
 </Project>

src/Directory.Build.targets

@@ -0,0 +1,7 @@
+<Project>
+    <ItemGroup>
+        <None Remove="$(MSBuildProjectDirectory)/silktouch.json" />
+        <AdditionalFiles Include="$(MSBuildProjectDirectory)/silktouch.json" />
+    </ItemGroup>
+    <Import Project="$(MSBuildThisFileDirectory)../Directory.Build.targets" />
+</Project>