From 17438daa2db66e462145587d09266557206192a5 Mon Sep 17 00:00:00 2001 From: Youness KAFIA Date: Mon, 22 Dec 2025 12:30:18 +0100 Subject: [PATCH 1/5] creating a new blog post --- ...ting-spirv-for-the-shader-system-part-3.md | 337 ++++++++++++++++++ 1 file changed, 337 insertions(+) create mode 100644 posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md diff --git a/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md b/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md new file mode 100644 index 0000000..665b523 --- /dev/null +++ b/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md @@ -0,0 +1,337 @@ +--- +title: "Investigating SPIR-V for the shader system - Part 3" +author: youness +popular: false +image: /images/spir.png +tags: ['.NET', 'Shaders'] +--- +In this third part we're going to design a parser for the SDSL language! +--- + +Table of Contents: + +[[TOC]] + +So many things happened since the [the last blog post](/blog/investigating-spirv-for-the-shader-system-part-2/) but none were written in a blog post format, fortunately for me, with the holidays I have time to tell more about it. + +## Preface + +Last time we've left on a fun design for a SPIR-V assembler/parser. This time instead of working on the backend side of the compiler, we're going to work on the frontend side, the parser. + +We're also keeping the same principles that this project started with : + +- Fast +- Low heap allocation +- Easy to update/modify + +Only this time we might break some rules since we're not following any C# churches :D. + +This post will show much more code, most of it will serve as an example, it won't always compile and might need some tweeking but can be used as a base to understand how the new SDSL parser works. + +## Why writing a parser from scratch and not use a library ? + +This is very simple to answer : because we can! + +Parsing a programming language is one of the easiest part of this whole ordeal, there are multiple ways to do it each with they pro and cons. + +The first thing we've tried was using 2 ready made parsing library, all of them were very powerful and easy to use but sacrificing on speed and heap allocation. + +The third experiment with parsing SDSL was with a handmade recursive descent parser, it was the easiest and fastest to implement and apparently used in major compiler frontends (gcc and clangs) but don't quote me on that. + +I haven't had the chance of finding a proper (read *idiot proof*) explanation of how to implement a recurisve descent parser but I'd like to mention [Kay Lack's video](https://www.youtube.com/watch?v=ENKT0Z3gldE) which is the clearest explanation I could ever found on this subject. + + +## The 3rd experiment + +It started as an expression/statement parser to compare with the current parser. In our case statements are what takes the most processing in SDSL, a low hanging fruit that we can reach fast. + +To avoid allocation, we will avoid creating classes as much as we can except for the abstract syntax tree (AST). + +### The tree + +We'll define some basic nodes for our AST, the first will be an abstract node and derive everything from it. + +```csharp + +// This struct will retain data of which part of text a node in our AST will represent +public record struct TextLocation(string Text, Range Range); + +// Each node will have to store location +public abstract class ASTNode(TextLocation location) +{ + TextLocation Location { get; set; } = location; +} + +// Expressions are values or a combinations of them with +public abstract class Expression(TextLocation location) : ASTNode(location); + +// Literals will contain the basic values we'll be dealing with in expressions +public abstract class Literal(TextLocation location) : Expression(location); + +// Such as : + +public class IntegerLiteral(long value, TextLocation location) : Literal(location); +public class FloatLiteral(double value, TextLocation location) : Literal(location); + +// And others but this will be enough, I'm also leaving implementation details out for simplicity + +// We'll first focus on binary operations, but there are unary and ternary operations too + +public enum Operators +{ + None, + Multiply, + Divide, + Moderand, + Add, + Subtract, + LeftShift, + RightShift, + //... + +} + + +public class BinaryExpression(Expression left, Operator operator, Expression right, TextLocation location) : Expression(location); + + + +``` + +### Scanning code + +For our parser we need a construct that will keep track of the position of the parser in the text. Then we'll need to define our rules which, when matched, will advance our position in the text and when not, will reset the position. + +For that we create a scanner as a `ref struct` and our rules will be written in plain function. We don't need any abstractions (or maybe a little), functions are reusable enough and we don't allocate anything on the heap by calling a function. + + +```csharp +public ref struct Scanner(string code) +{ + public string Code { get; } = code; + public int Position { get; set; } = 0; + + // Sometimes we'd like to peek one or more characters to reject rules early + public int Peek() => (int)Code[Position]; + public ReadOnlySpan Peek(int size) => Code[Position..(Position+size)]; + + public bool Match(string matchString, bool caseSensitive) + { + // The implementation is fairly straightforward :) + } +} +``` + + +### Building blocks + +Our rules will be functions returning a boolean, taking as parameters the scanner and an `out` reference to a node of the tree. + +Here's an example for the integer parser : + +```csharp + +// A utility function that checks if the char is a digit with a specific value or in a range + +bool IsDigitValue(char c, int value) + => char.IsDigit(c) && (c - '0' == value); + +bool IsDigitRange(char c, Range range); + + +bool ParseIntegerLiteral(ref Scanner scanner, out IntegerLiteral literal) +{ + // We keep track of the position before we match anything + // We can also create copies of the struct instead by storing it in different variables + // but I prefer copying the least amount of data possible + var start = scanner.Position; + // If we get a zero, that's our value :D, integer literals never start with a zero + if(IsDigitValue(scanner.Peek(), 0)) + { + scanner.Position += 1; + literal = new(0, new((int)(scanner.Peek() - '0'), new(scanner.Code, position..scanner.Position))); + return true; + } + // We keep scanning until there is no digits + else if(char.IsDigit(scanner.Peek())) + { + while(char.IsDigit(scanner.Peek())) + scanner.Position += 1; + + literal = new(0, new(int.Parse(scanner.Code[position..scanner.Position]), new(scanner.Code, position..scanner.Position))); + return true; + } + // If we don't match any digit then we have to go back in the starting position + // This is not necessary here so we can omit this line of code + // In some cases this line of code will prove very relevant + scanner.Position = start; + + // Sometimes in the parse we never allocate any data for the node + // This is perfect for reducing allocation in our case + literal = null!; + return false; +} + +``` + +Of course this implementation is lacking for the sake of simplicity, I leave the details to those who want to implement it themselves! + +We can derive this integer parser into a float parser quite easily, each language has its own rules of writing floating point numbers and it's up to the author of the language to decide which one is best. + + +### Abstractions + +After implementing the floating point parser we can abstract both those parsers into one encompassing everything very simply. + + +```csharp + +bool ParseNumberLiteral(ref Scanner scanner, out NumberLiteral number) +{ + var start = scanner.Position; + if(ParseIntegerLiteral(ref scanner, out var integerLiteral)) + { + number = (NumberLiteral)integerLiteral; + return true; + } + else if(ParseFloatLiteral(ref scanner, out var floatLiteral)) + { + number = (NumberLiteral)floatLiteral; + return true; + } + // Same as before :) + scanner.Position = start; + number = null!; + return false; +} + +``` + + +We can even go one step higher and use this number parser to create our first binary operation parser : + +```csharp + + +// A utility function that resets the position. We can modify it to catch errors +bool ExitParser(ref Scanner scanner, out TNode node, int position) where TNode : ASTNode; + +// A utility function that looks ahead by skipping some white space. +// Looking ahead is something that happens quite often when parsing SDSL. + +bool FollowedByAny(ref Scanner scanner, string chars, out char value, bool withSpaces = false, bool advance = false); + + +bool Spaces(ref Scanner scanner, int min = 0) +{ + var start = scanner.Position; + while(char.IsWhiteSpace(scanner.Peek())) + scanner.Position += 1; + if(scanner.Position - start >= min) + return true; + else + { + scanner.Position = start; + return false; + } +} + +bool Mul(ref Scanner scanner, out Expression expression) +{ + // Mulitplicative expressions can be stringed together, so we should use a loop + char op = '\0'; + expression = null!; + var position = scanner.Position; + do + { + Spaces(ref scanner, 0); + // If we have reached an operator and have already parsed a first expression + if (op != '\0' && expression is not null) + { + // We nest the multiplications by creating a binary expression containing the binary expression we just finished parsing + if (ParseNumberLiteral(ref scanner, out var number)) + expression = new BinaryExpression(expression, op.ToOperator(), number, scanner[position..scanner.Position]); + else return ExitParser(ref scanner, out expression, position); + } + // In the case we haven't reached an operator + else if (expression is null && op == '\0') + { + // Our number becomes our result expression + if (ParseNumberLiteral(ref scanner, result, out var number)) + expression = number; + else return ExitParser(ref scanner, out expression, position); + } + } + // We keep parsing until we don't see any multiplicative operators + while (FollowedByAny(ref scanner, "*/%", out op, withSpaces: true, advance: true)); + if (expression is not null) + return true; + else return ExitParser(ref scanner, result, out expression, position, orError); +} + +``` + +We can also write the addition/subtraction parser, following the operator precedence : + + +```csharp +bool Add(ref Scanner scanner, out Expression expression) +{ + char op = '\0'; + expression = null!; + var position = scanner.Position; + do + { + Spaces(ref scanner, 0); + if (op != '\0' && expression is not null) + { + if (Add(ref scanner, out var mul)) + expression = new BinaryExpression(expression, op.ToOperator(), mul, scanner[position..scanner.Position]); + else return ExitParser(ref scanner, out expression, position); + } + else if (expression is null && op == '\0') + { + if (Mul(ref scanner, result, out var prefix)) + expression = prefix; + else return ExitParser(ref scanner, out expression, position); + } + } + while (FollowedByAny(ref scanner, "+-", out op, withSpaces: true, advance: true)); + if (expression is not null) + return true; + else return ExitParser(ref scanner, result, out expression, position, orError); +} + +``` + +And that's the gist of it! The rest is about adding all the other elements of the language. + +## The difficult parts + +There have been little bumps here and there while writing this parser + +### C-like languages + +SDSL was originally created as an extension of HLSL, a higher level of HLSL that includes mixin operators making it possible to mix some shader modules together. HLSL is itself very inspired from C/C++ in its syntax, some of the syntax in C were put in HLSL and kept in SDSL and have made it a bit more complicated to parse. + +A prime example is the "Declaration follows usage" principles. In C, declaring an array is written the same way it is used (eg : `int myArray[5];` and `myArray[0] = 2;`). + +This is confusing when coming from C# and creates some frictions when developping a game with Stride since we're using both C# and SDSL. There is no need to stick to C's syntax and HLSL/SDSL don't have the same memory semantic as C so the new parser supports both the C# and C syntax for array declarations and hopefully in the future SDSL will drop the C syntax and closely ressemble C# while being consistent with SPIR-V/HLSL memory semantic. + +### Recursion problem + +This was a problem we didn't see through the bits of code I've showed in this blog post but the original experiment had recursion instead of loops for binary operators. This lead the new parser to perform as fast as the previous one in cases where expressions were complex and varied, eg. `a + b * c + d - (3 * (5 + (e / 12)) * 4 * 1 / f)`. + +This is due to the parser doing lots of backtracking and the solution was to implement [precedence climing](https://eli.thegreenplace.net/2012/08/02/parsing-expressions-by-precedence-climbing), the while loop present in our `Add` and `Multiply` parsing function we saw a bit earlier in our example 😊. + + +### The ambiguity of SDSL + +The ideal programming language grammar is context free, all information is gathered through the parsing +SDSL has introduced some ambiguity since mixins are considered as types, it's harder to know which identifier corresponds to which type or function. Lots of programming languages are not context free and SDSL is no different, so part of the work was offloaded to the backend + +## Conclusions + +Writing this SPIR-V library was very fun, I've learned a lot about SPIR-V and some of the possible use cases for it in the context of Stride. As you might have imagined, this was the easy part of this shader system rewrite. In the next installment we'll see the little adventure I went through to make the most of our shader front-end! + +Thank you for reading! From 26dd05a118a776938752369dfb440f38fcc96e9f Mon Sep 17 00:00:00 2001 From: KAFIA Youness Date: Tue, 23 Dec 2025 00:53:08 +0100 Subject: [PATCH 2/5] updated the blog post --- ...ting-spirv-for-the-shader-system-part-3.md | 48 +++++++++++-------- 1 file changed, 27 insertions(+), 21 deletions(-) diff --git a/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md b/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md index 665b523..776d05d 100644 --- a/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md +++ b/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md @@ -16,7 +16,7 @@ So many things happened since the [the last blog post](/blog/investigating-spirv ## Preface -Last time we've left on a fun design for a SPIR-V assembler/parser. This time instead of working on the backend side of the compiler, we're going to work on the frontend side, the parser. +Last time we went through SPIR-V and how to parse/assemble SPIR-V using C#, this time we'll see how the new SDSL parser was built from the ground up. I'll go through the implementation of a small parser to illustrate the design chosen for SDSL. We're also keeping the same principles that this project started with : @@ -28,22 +28,23 @@ Only this time we might break some rules since we're not following any C# church This post will show much more code, most of it will serve as an example, it won't always compile and might need some tweeking but can be used as a base to understand how the new SDSL parser works. -## Why writing a parser from scratch and not use a library ? +## Why not using a library ? -This is very simple to answer : because we can! +Why should we ? 🤔 -Parsing a programming language is one of the easiest part of this whole ordeal, there are multiple ways to do it each with they pro and cons. +SDSL is not going to change much and we're only maintaining one language so we can write a simple parser that will only work for SDSL. -The first thing we've tried was using 2 ready made parsing library, all of them were very powerful and easy to use but sacrificing on speed and heap allocation. +Parsing a programming language is easy, the difficult part comes after parsing a language. -The third experiment with parsing SDSL was with a handmade recursive descent parser, it was the easiest and fastest to implement and apparently used in major compiler frontends (gcc and clangs) but don't quote me on that. +I've experimented twice with very powerful libraries, they were easy to use and would have us maintain fewer lines of code by sacrificing on speed and heap allocation. -I haven't had the chance of finding a proper (read *idiot proof*) explanation of how to implement a recurisve descent parser but I'd like to mention [Kay Lack's video](https://www.youtube.com/watch?v=ENKT0Z3gldE) which is the clearest explanation I could ever found on this subject. +The third experiment with parsing SDSL was with a handmade recursive descent parser, it was the easiest and fastest to implement and apparently used in major compiler frontends [like clang](https://discourse.llvm.org/t/where-can-i-download-the-ebnf-syntax-description-document-for-c-syntax-parser-by-clang/87614). +At the time, I hadn't had the chance of finding a proper (read *idiot proof*) explanation of how to implement a recurisve descent parser but I'd like to mention [Kay Lack's video](https://www.youtube.com/watch?v=ENKT0Z3gldE) which is the clearest explanation I could ever found on this subject. I recommend you give a watch to have a better understanding of what will be written. ## The 3rd experiment -It started as an expression/statement parser to compare with the current parser. In our case statements are what takes the most processing in SDSL, a low hanging fruit that we can reach fast. +It started as an expression/statement parser to compare with the current parser. In our case statements are what takes the most processing in SDSL, a low hanging fruit that we can reach fast and benchark against our current shader system. To avoid allocation, we will avoid creating classes as much as we can except for the abstract syntax tree (AST). @@ -52,7 +53,6 @@ To avoid allocation, we will avoid creating classes as much as we can except for We'll define some basic nodes for our AST, the first will be an abstract node and derive everything from it. ```csharp - // This struct will retain data of which part of text a node in our AST will represent public record struct TextLocation(string Text, Range Range); @@ -100,9 +100,11 @@ public class BinaryExpression(Expression left, Operator operator, Expression rig ### Scanning code -For our parser we need a construct that will keep track of the position of the parser in the text. Then we'll need to define our rules which, when matched, will advance our position in the text and when not, will reset the position. +For our parser we need a construct that will keep track of the position of the parser in the text. Then we'll need to define our rules which, when matched, will advance our position in the text and when not, will reset the position. For that we create a scanner object with two parameters, the code being scanned and the position we're at. + +Our parser rules will be written as plain function, they don't need to be abstracted any other ways as functions are reusable enough and we don't allocate anything on the heap by calling a function. -For that we create a scanner as a `ref struct` and our rules will be written in plain function. We don't need any abstractions (or maybe a little), functions are reusable enough and we don't allocate anything on the heap by calling a function. +We should still be mindful of the limitations of recursion! ```csharp @@ -125,12 +127,11 @@ public ref struct Scanner(string code) ### Building blocks -Our rules will be functions returning a boolean, taking as parameters the scanner and an `out` reference to a node of the tree. +For our parser, the rules will be functions returning a boolean, taking as parameters the scanner and an `out` reference to a node of the tree. Here's an example for the integer parser : ```csharp - // A utility function that checks if the char is a digit with a specific value or in a range bool IsDigitValue(char c, int value) @@ -149,6 +150,7 @@ bool ParseIntegerLiteral(ref Scanner scanner, out IntegerLiteral literal) if(IsDigitValue(scanner.Peek(), 0)) { scanner.Position += 1; + literal = new(0, new((int)(scanner.Peek() - '0'), new(scanner.Code, position..scanner.Position))); return true; } @@ -157,7 +159,7 @@ bool ParseIntegerLiteral(ref Scanner scanner, out IntegerLiteral literal) { while(char.IsDigit(scanner.Peek())) scanner.Position += 1; - + literal = new(0, new(int.Parse(scanner.Code[position..scanner.Position]), new(scanner.Code, position..scanner.Position))); return true; } @@ -166,7 +168,7 @@ bool ParseIntegerLiteral(ref Scanner scanner, out IntegerLiteral literal) // In some cases this line of code will prove very relevant scanner.Position = start; - // Sometimes in the parse we never allocate any data for the node + // In some cases we never allocate the node in the heap because we haven't matched our rules // This is perfect for reducing allocation in our case literal = null!; return false; @@ -174,7 +176,6 @@ bool ParseIntegerLiteral(ref Scanner scanner, out IntegerLiteral literal) ``` -Of course this implementation is lacking for the sake of simplicity, I leave the details to those who want to implement it themselves! We can derive this integer parser into a float parser quite easily, each language has its own rules of writing floating point numbers and it's up to the author of the language to decide which one is best. @@ -185,7 +186,6 @@ After implementing the floating point parser we can abstract both those parsers ```csharp - bool ParseNumberLiteral(ref Scanner scanner, out NumberLiteral number) { var start = scanner.Position; @@ -308,7 +308,7 @@ And that's the gist of it! The rest is about adding all the other elements of th ## The difficult parts -There have been little bumps here and there while writing this parser +There have been little bumps here and there while writing this parser. This is the first time I've implemented one, especially one that fits in the context of a compiler. ### C-like languages @@ -328,10 +328,16 @@ This is due to the parser doing lots of backtracking and the solution was to imp ### The ambiguity of SDSL The ideal programming language grammar is context free, all information is gathered through the parsing -SDSL has introduced some ambiguity since mixins are considered as types, it's harder to know which identifier corresponds to which type or function. Lots of programming languages are not context free and SDSL is no different, so part of the work was offloaded to the backend +SDSL has introduced some ambiguity since mixins are considered as types, it's harder to know which identifier corresponds to which type or function. Lots of programming languages are not context free and SDSL is no different, so part of the work was offloaded to the backend. + +This is something I've personally struggled while implementing the compiler, knowing which parts can go to syntax analysis, semantic analysis or the assembler itself is challenge in itself for first timers 😅. ## Conclusions -Writing this SPIR-V library was very fun, I've learned a lot about SPIR-V and some of the possible use cases for it in the context of Stride. As you might have imagined, this was the easy part of this shader system rewrite. In the next installment we'll see the little adventure I went through to make the most of our shader front-end! +Writing a handmade parser for your programming language is almost always the best decision you could make. There are many ways to write one but recursive descent parsers can be a very helpful start until you have a very specific need for your language. + +I'm excited to show more of SDSL's new compiler progress and writing that parser was a huge endeavor, going through two different implementations before setting to what it is now. + +In the next blog post we'll see how bridging the parser with the SPIR-V assembler was done, hopefully with an example of a working version of the compiler. -Thank you for reading! +Thanks for reading! From 6029a3b64afa54681bcb5b30df9571049b9d4321 Mon Sep 17 00:00:00 2001 From: KAFIA Youness Date: Tue, 23 Dec 2025 00:54:20 +0100 Subject: [PATCH 3/5] rename the file --- ...025-12-23-investigating-spirv-for-the-shader-system-part-3.md} | 0 1 file changed, 0 insertions(+), 0 deletions(-) rename posts/{2025-12-21-investigating-spirv-for-the-shader-system-part-3.md => 2025-12-23-investigating-spirv-for-the-shader-system-part-3.md} (100%) diff --git a/posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md b/posts/2025-12-23-investigating-spirv-for-the-shader-system-part-3.md similarity index 100% rename from posts/2025-12-21-investigating-spirv-for-the-shader-system-part-3.md rename to posts/2025-12-23-investigating-spirv-for-the-shader-system-part-3.md From 86a8f252f5219095627cde6b24879ce36a9ff808 Mon Sep 17 00:00:00 2001 From: Youness KAFIA Date: Wed, 24 Dec 2025 12:41:53 +0100 Subject: [PATCH 4/5] last update of the website --- ...ing-spirv-for-the-shader-system-part-3.md} | 91 +++++++++++++++---- 1 file changed, 71 insertions(+), 20 deletions(-) rename posts/{2025-12-23-investigating-spirv-for-the-shader-system-part-3.md => 2025-12-24-investigating-spirv-for-the-shader-system-part-3.md} (76%) diff --git a/posts/2025-12-23-investigating-spirv-for-the-shader-system-part-3.md b/posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md similarity index 76% rename from posts/2025-12-23-investigating-spirv-for-the-shader-system-part-3.md rename to posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md index 776d05d..21955f8 100644 --- a/posts/2025-12-23-investigating-spirv-for-the-shader-system-part-3.md +++ b/posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md @@ -24,8 +24,6 @@ We're also keeping the same principles that this project started with : - Low heap allocation - Easy to update/modify -Only this time we might break some rules since we're not following any C# churches :D. - This post will show much more code, most of it will serve as an example, it won't always compile and might need some tweeking but can be used as a base to understand how the new SDSL parser works. ## Why not using a library ? @@ -44,13 +42,13 @@ At the time, I hadn't had the chance of finding a proper (read *idiot proof*) ex ## The 3rd experiment -It started as an expression/statement parser to compare with the current parser. In our case statements are what takes the most processing in SDSL, a low hanging fruit that we can reach fast and benchark against our current shader system. +It started as an expression/statement parser to compare with the current parser. In our case statements are what takes the most processing in SDSL, a low hanging fruit that we can reach fast and benchark against our current shader system. It ended up working so well it became the main parser for SDSL ! To avoid allocation, we will avoid creating classes as much as we can except for the abstract syntax tree (AST). ### The tree -We'll define some basic nodes for our AST, the first will be an abstract node and derive everything from it. +We'll define some basic nodes for our AST, the first will be an abstract node and we'll derive everything from it. ```csharp // This struct will retain data of which part of text a node in our AST will represent @@ -304,39 +302,92 @@ bool Add(ref Scanner scanner, out Expression expression) ``` -And that's the gist of it! The rest is about adding all the other elements of the language. +The sharpest ones will notice I'm not using recursion for everything, i'm using a while loop. This is a small optimisation called [precedence climbing](https://eli.thegreenplace.net/2012/08/02/parsing-expressions-by-precedence-climbing) to avoid to many recursions when parsing long and complex expressions. + +And that's the gist of it! The rest is about adding all the other elements of the language. For SDSL that was about parsing a subset of HLSL, add the SDSL extra features and digging through all of the source code to find any quirks of the language and make sure we can handle them. + +In the future SDSL will be less reliant on HLSL syntax and follow one more similar to C# to avoid any confusion when switching between the two languages. + +## Benchmarks -## The difficult parts +It wouldn't mean anything without benchmarks, so let's have a typical SDSL shader to benchmark for. -There have been little bumps here and there while writing this parser. This is the first time I've implemented one, especially one that fits in the context of a compiler. -### C-like languages +```hlsl +namespace MyNameSpace +{ + shader Parent + { + + struct MyStruct { + int a; + }; -SDSL was originally created as an extension of HLSL, a higher level of HLSL that includes mixin operators making it possible to mix some shader modules together. HLSL is itself very inspired from C/C++ in its syntax, some of the syntax in C were put in HLSL and kept in SDSL and have made it a bit more complicated to parse. + stream int a; -A prime example is the "Declaration follows usage" principles. In C, declaring an array is written the same way it is used (eg : `int myArray[5];` and `myArray[0] = 2;`). + stage abstract void DoSomething(); -This is confusing when coming from C# and creates some frictions when developping a game with Stride since we're using both C# and SDSL. There is no need to stick to C's syntax and HLSL/SDSL don't have the same memory semantic as C so the new parser supports both the C# and C syntax for array declarations and hopefully in the future SDSL will drop the C syntax and closely ressemble C# while being consistent with SPIR-V/HLSL memory semantic. + void Method() + { + int a = 0; + float4 buffer = float4(1,3, float2(1,2)); + float4x4 a = float4x4( + float4(1,2,3,4), + float4(1,2,3,4), + float4(1,2,3,4), + float4(1,2,3,4) + ); + for(;;) + { + if(i == 5) + { + Print(i); + } + } + int b = (a - 10 / 3 ) * 32 +( streams.color.Normalize() + 2); + if(a == 2) + { + } + } + }; +} +``` -### Recursion problem +After some text crunching we get these numbers! -This was a problem we didn't see through the bits of code I've showed in this blog post but the original experiment had recursion instead of loops for binary operators. This lead the new parser to perform as fast as the previous one in cases where expressions were complex and varied, eg. `a + b * c + d - (3 * (5 + (e / 12)) * 4 * 1 / f)`. +DRUM ROLL 🥁 ... -This is due to the parser doing lots of backtracking and the solution was to implement [precedence climing](https://eli.thegreenplace.net/2012/08/02/parsing-expressions-by-precedence-climbing), the while loop present in our `Add` and `Multiply` parsing function we saw a bit earlier in our example 😊. +```pwsh +AMD Ryzen 7 3700X 3.60GHz, 1 CPU, 16 logical and 8 physical cores +.NET SDK 10.0.101 + [Host] : .NET 10.0.1 (10.0.1, 10.0.125.57005), X64 RyuJIT x86-64-v3 + DefaultJob : .NET 10.0.1 (10.0.1, 10.0.125.57005), X64 RyuJIT x86-64-v3 +| Method | Mean | Error | StdDev | Gen0 | Gen1 | Allocated | +|--------------- |---------:|---------:|---------:|--------:|-------:|----------:| +| ParseShaderOld | 151.3 us | 1.490 us | 1.16 us | 22.2168 | 5.1270 | 183.17 KB | +| ParseShaderNew | 32.28 us | 0.295 us | 0.276 us | 1.6479 | | 13.78 KB | +``` -### The ambiguity of SDSL +We run 5 times faster and allocate 20 times less memory than before. Objects allocated are not promotted to Gen1. -The ideal programming language grammar is context free, all information is gathered through the parsing -SDSL has introduced some ambiguity since mixins are considered as types, it's harder to know which identifier corresponds to which type or function. Lots of programming languages are not context free and SDSL is no different, so part of the work was offloaded to the backend. +That's exciting news to me! We could theoritically parse 4 more shaders while in a single core without worrying too much about GC pauses. This will surely improve startup times for the Stride engine! -This is something I've personally struggled while implementing the compiler, knowing which parts can go to syntax analysis, semantic analysis or the assembler itself is challenge in itself for first timers 😅. ## Conclusions -Writing a handmade parser for your programming language is almost always the best decision you could make. There are many ways to write one but recursive descent parsers can be a very helpful start until you have a very specific need for your language. +Let's check our goals again ... + +- ✅ Fast + 5x faster allows me to stamp that parser with ✨*Blazingly fast*🚀 +- ✅ Low heap allocation + 20x less allocation is going to help us a lot, especially during runtime compilation +- ✅ Easy to update/modify + Going the functional way has only made it easier to make changes. We also own all our parsing code, nothing is hidden behind API which is great! + +I could leave with a very nonchalant "Code speaks for itself" but I have to convince you that writing a handmade parser for your programming language is almost always the best decision you could make, there are many ways to write one but recursive descent parsers can be a very helpful start until you have a very specific need for your language. -I'm excited to show more of SDSL's new compiler progress and writing that parser was a huge endeavor, going through two different implementations before setting to what it is now. +I'm excited to show more of SDSL's new compiler progress and writing that parser was a huge endeavor, espcially going through two different implementations before settling to what it is now. In the next blog post we'll see how bridging the parser with the SPIR-V assembler was done, hopefully with an example of a working version of the compiler. From 7151787e8c6edd9570ca97447572a798b3030877 Mon Sep 17 00:00:00 2001 From: Youness KAFIA Date: Wed, 24 Dec 2025 17:39:01 +0100 Subject: [PATCH 5/5] Adding some back and forward links --- ...23-11-10-investigating-spirv-for-the-shader-system.md | 7 +++++-- ...0-investigating-spirv-for-the-shader-system-part-2.md | 4 ++++ ...4-investigating-spirv-for-the-shader-system-part-3.md | 9 ++++++++- 3 files changed, 17 insertions(+), 3 deletions(-) diff --git a/posts/2023-11-10-investigating-spirv-for-the-shader-system.md b/posts/2023-11-10-investigating-spirv-for-the-shader-system.md index 99a0d86..97e1f41 100644 --- a/posts/2023-11-10-investigating-spirv-for-the-shader-system.md +++ b/posts/2023-11-10-investigating-spirv-for-the-shader-system.md @@ -8,10 +8,13 @@ tags: ['.NET', 'Shaders'] In this first part of a new series of blog posts, we will learn more about Stride's shader system, its limitations and how to make it better thanks to a very useful shader language called SPIR-V. This will be the first step in implementing a new and better shader system. -Don't forget to checkout [part 2](/blog/investigating-spirv-for-the-shader-system-part-2/)! - --- +If you're interested in the other parts of this blog series : + - [Part 2](/blog/investigating-spirv-for-the-shader-system-part-2/) we parse and assemble some SPIR-V + - [Part 3](/blog/investigating-spirv-for-the-shader-system-part-3/) we write a new parser for SDSL + + Table of Contents: [[TOC]] diff --git a/posts/2024-11-20-investigating-spirv-for-the-shader-system-part-2.md b/posts/2024-11-20-investigating-spirv-for-the-shader-system-part-2.md index 1bbd310..c73edb6 100644 --- a/posts/2024-11-20-investigating-spirv-for-the-shader-system-part-2.md +++ b/posts/2024-11-20-investigating-spirv-for-the-shader-system-part-2.md @@ -43,6 +43,10 @@ In this second part we're going to dive deeper in how the current SDSL compiler } +If you're interested in the other parts of this blog series : + - [Part 1](/blog/investigating-spirv-for-the-shader-system/) An introduction to the project + - [Part 3](/blog/investigating-spirv-for-the-shader-system-part-3/) we write a new parser for SDSL + Table of Contents: [[TOC]] diff --git a/posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md b/posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md index 21955f8..cb4bbe5 100644 --- a/posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md +++ b/posts/2025-12-24-investigating-spirv-for-the-shader-system-part-3.md @@ -5,9 +5,16 @@ popular: false image: /images/spir.png tags: ['.NET', 'Shaders'] --- -In this third part we're going to design a parser for the SDSL language! + +In this blog post, we will focus on how the new SDSL parser has been implemented through writing a prototype expression parser as an example. We will see how this can be possible without sacrificing performance and allocating the least amount of memory possible. And finally see how this improved on the current shader parser system in Stride. + --- +If you're interested in the other parts of this blog series : + - [Part 1](/blog/investigating-spirv-for-the-shader-system/) An introduction to the project + - [Part 2](/blog/investigating-spirv-for-the-shader-system-part-2/) we parse and assemble some SPIR-V + + Table of Contents: [[TOC]]