Proposal: Compile time expressions

[jcouv]

@nvmkpk commented on Mon Jun 27 2016

It would be useful to specify something like "Hello World".Length in some special syntax that gets evaluated at compile time and replaced with 11 (becoming a constant) in the compiled binaries.

---

@choikwa commented on Mon Jun 27 2016

You mean it's not already converted to constant?

---

@nvmkpk commented on Mon Jun 27 2016

JIT compiler might optimize it but I want the language compiler to do it. In the generated IL code the string is constant but the value of its length is not.

---

@nvmkpk commented on Tue Jun 28 2016

I gave string as example. I want to be able to use more expressions to evaluated at compile time. Something like eval(simple expression).

---

@svick commented on Tue Jun 28 2016

What is the reason? Is it about performance? Or something else?

---

@alrz commented on Tue Jun 28 2016

This is also related to #10972 and #9627 (there are similar examples in the latter issue).

---

@nvmkpk commented on Tue Jun 28 2016

The reason is simple. If we know that the value of an expression at run time is exactly same as at compile time, there is no point in deferring evaluation to run time.

---

@Unknown6656 commented on Wed Jun 29 2016

@nvmkpk : I do quite like the idea, I do however see the following problem:

Imagine, that one is writing a (memory) profiler. The profiler would not register the function call int System::String::get_Length(), as the function would be "precompiled" into its result. Thus the application profiling result could be incorrect compared to expected data/values.

My second question/concern: To which degree do you want this "optimization"? Should it apply to each completely deterministic function(-chain) with only constant input?
e.g. should the expression "Hello,World!".ToUpper().Remove(0,1) be precompiled into "ELLO,WORLD!" ? Where do we draw the line?

EDIT: or should one use a pseudo-attribute like [compile], which one could put in front of fields/expressions?

---

@svick commented on Wed Jun 29 2016

@Unknown6656 That already happens, in the JIT compiled machine code, there is no call to get_Length(), since it's inlined.

And you can't put attributes on expressions, at least not without #6671.

---

@Unknown6656 commented on Wed Jun 29 2016

@svick : Interesting, I did not know the fact about get_Length()-inlining.

Concerning the usage of attributes on expressions: During the composition of my previous comment I assumed, that issue #6671 will be implemented in the future C# version, as it has been marked as
[2 - Ready]

---

@choikwa commented on Wed Jun 29 2016

My second question/concern: To which degree do you want this "optimization"?

Hopefully in Release build, all of the optimizations.

Where do we draw the line?

Where-ever compiler can statically determine static or read-only or const-ness and exploit it 👯

---

@nvmkpk commented on Wed Jun 29 2016

That is fine for optimization part but I would like the developer to also call for compile time evaluation by using something like eval(expression). This would be better way than using attributes.

---

@Unknown6656 commented on Wed Jun 29 2016

@choikwa (, @nvmkpk ): So, would the following expression be optimized (assuming, that a array<string> shall be interpreted as constant, and that all functions do not use non-constant internal parameters and are fully deterministic)?

char a = (new string[] { "test", "string" })[1].Reverse().ToArray()[3]) - 'B';
// a now stores the character `0`

My point is: I really do love your proposal, but does too much optimization not take too much time on a big scale? Are arrays defined at compile time also a constant, if they remain unchanged?
Imagine the calculation time for large for-loops, nested conditions etc. which will be precompiled. Does not take this too much compilation time?

---

If anyone is interested, how the character is calculated in the example above:

string[] arr = new string[] { "test", "string" };
char[] elem = arr[1].Reverse().ToArray(); // [g,n,i,r,s,t]
char r = elem[3]; // 'r' (0x72)
char a = r - 'B'; // 0x72 - 0x42 == 0x30 = '0'
// a now stores the character `0`

---

@choikwa commented on Wed Jun 29 2016

would the following expression be optimized?

A compiler worth its salt would optimize. I am hoping CoreCLR is to be such one. Maybe you can help make that happen 😄

does too much optimization not take too much time on a big scale?

C++ already does this with constexpr and template meta. A saving grace is that you only do this once during compilation. In C# context, maybe this makes more sense in AOT compilation or more expensive ReJIT (don't we all want tiered JIT?).

PS. a lot of compilers already neuter optimization if method size is too large, to the detriment of performance just because compilation time would be excessive. Would not surprise me if CoreCLR had to do something similar since it's a VM.

---

@Unknown6656 commented on Wed Jun 29 2016

@choikwa I think that you have me convinced on that point! 😄
I knew that C++ has precompiled templates instead of runtime-generated generics, but I did not yet know about constexpr. I think, that this is an excellent definition of constant expressions/methods (which needs some adaptation for C#)

Maybe you can help make that happen

I will try as much as I can 😉

---

@choikwa commented on Wed Jun 29 2016

FYI this very issue is open in https://github.com/dotnet/coreclr/issues/3633

---

@nvmkpk commented on Wed Jun 29 2016

That is for optimzation. This proposal is about letting developer write general compile time expressions not just for string length.

---

@choikwa commented on Wed Jun 29 2016

Sorry for hijacking the issue. You are correct, and I agree that this should have some dependence on success of https://github.com/dotnet/roslyn/issues/6671

---

@ashmind commented on Tue Aug 23 2016

On the syntax -- I suggested const(...) in #11259 which has a benefit of already being a keyword.

---

@alrz commented on Thu Apr 20 2017

@nvmkpk May you move this to csharplang repo?

[tuespetre]

👍 for @ashmind's suggestion

[alrz]

I don't think there is a need for a new syntax, the compiler could evaluate some properties/methods at compile-time, the resulting expression could be used in a const context as well, e.g.

switch(obj) {
  case string_constant.Length:
  case string_constant.Substring(...): // string_constant could be a `const`, `nameof`, literal, etc.
  case typeof(T).Namespace:
  case typeof(T).Name:
    break;
}

and so on.

[MgSam]

@alrz People have asked for all kinds of stuff to be compile time constants- including mathematical expressions, string manipulation, date manipulation, regex manipulation, etc. A more general feature allowing anything known statically to be evaluated at compile time would solve all of these types of requests rather than waiting for the dev team to add support for each little use case someone might think of.

[alrz]

a new syntax doesn't mean that it would magically enable all expressions to be evaluated at compile-time. there will be a set rules for the said expressions so that does not lessen the amount of work to make this happen. I don't think making it an opt-in feature would be useful either. some expressions are inherently a constant, evaluating them at compile-time has an advantage of lighting up without any source changes.

[alrz]

If you're talking about "const functions" I'll agree with you, being able to write a function that is able to be evaluated at compile-time will be very useful, but even then the compiler needs to validate the requirements of such functions and at the end, there is no need to use a new syntax at the call-site.

[bbarry]

no need to use a new syntax at the call-site.

There may be a small need for some new syntax at the call site in places where the existing syntax would be allowed runtime syntax. I've suggested before to have a (const) "cast" be available for these expressions.

[tannergooding]

Don't we already have 3 or 4 proposals that cover some form of constexpr for C#?

[TonyValenti]

I would suggest that any method (even non-determinstic ones) should be constexprable.

var BuildDate = constexpr(DateTime.Now.ToString());

[tannergooding]

I would suggest that any method (even non-determinstic ones) should be constexprable.

If that were supported, I would fully expect it to fail if you compiled with /deterministic

[alrz]

@bbarry

Can you give an example where it's absolutely needed to explicitly mark an expression as const? If you're suggesting that we allow evaluating arbitrary expressions at compile-time, I don't think that's something constexpr is about. We make guarantees at declaration-site and pass it over to metadata for client code. There is a reason why we don't allow non-constants in case labels: because we don't want undefined behavior depending on implementation detail. For example, if compiler turns a switch statement to a binary tree, you might lose some intended side effects, or worse, they might evaluate in an unexpected order.

[alrz]

Wouldn't new pattern syntax (dotnet/roslyn#22718) block this feature? imagine you have a compile-time function like this:

static const int Add3(reaodnly int a) => a + 3;

then if you want to use it in a switch case:

case Add3(4):

it would be parsed as a pattern after this change, and not an invocation expression.

[Pzixel]

@TonyValenti you're already being answered here: dotnet/roslyn#15079 (comment)

There's a rule in the compiler world: if a compiler evaluates an expression at compile time then it better ensure that the result is identical with the result obtained if the expression is evaluated at runtime. Anything less than that is usually considered a bug, plain and simple.

I totally agree with this.

However, I'd like to have pure methods even if they aren't currently optimized in any way. All public string methods are pure so it would be really nice if we could use them at compile time.

[HaloFour]

@Pzixel

Even with String there could be subtle differences. For example, if the result of two expressions are the same string value they would still be different string instances. But if those expressions were evaluated at compile-time into string literals then they would be the same string instance. Probably a trivial concern, but a change nonetheless.

[Pzixel]

@HaloFour I think we're ok with that. I can't imagine a real case when it would break anything. You want compare strings, but only by reference? Whoa. That doesn't seems to be performed often. And it works with constants only, so it wouldn't affect most of code anyway. It's just a better indicator for developers hey, you are calling a pure method but do not use return value, and source of some optimizations.

It seems much smaller and safer change than change in loop closures from 3 3 3 to 1 2 3.

[CyrusNajmabadi]

All public string methods are pure so it would be really nice if we could use them at compile time.

I'm not sure this is true. For example, string.CompareTo says this:

This method performs a word (case-sensitive and culture-sensitive) comparison using the current culture.

Therefore:

1. If the compiler runs in a different culture than the actual program, this will result in different behavior.
2. If the user were to change the culture at runtime, this would now not be reflected properly.

String, IMO, would be one of the hardest APIs to get right for compile time evaluation unless you were extremely careful to only whitelist the ordinal methods. Even then, i wouldn't be certain that something wouldn't be going wrong :-/

[jrmoreno1]

For anyone questioning whether constant expressions could be useful, I’ suggest watching this video:
https://youtu.be/zBkNBP00wJE. It’s in c++ and not directly applicable, but it shows that this can have a huge impact.

[TonyValenti]

@jrmoreno1 -
I've come up with a solution that kind of gets me to where I want. It is a nasty, nasty hack but it is better than nothing.

1. I have a solution that has my standard, non-constant methods in it.
2. I create a second solution that has T4 files that reference my primary solution. My T4 files invoke my methods and then write the output to constant fields and such.
3. I add a reference to the output.g.cs file that is generated wherever I want to use the constant values.

That type of workflow should be natively part of C# and shouldn't require such a glorious hack.

Below is an easy to understand sample showing what I'm doing (in this case, capturing version and build date info):

<#@ template debug="true" hostspecific="false" language="C#" #>
<#@ output extension=".g.cs" #>
<#@ assembly name="System.Core" #>
<#@ import namespace="System.Linq" #>
<#@ import namespace="System.Text" #>
<#@ import namespace="System.Collections.Generic" #>

<#

	var Date = DateTimeOffset.Now.ToString(System.Globalization.DateTimeFormatInfo.InvariantInfo);

    var Version = String.Format("{0:00}.{1:00}.{2:00}", DateTime.Now.Year % 100, DateTime.Now.Month,DateTime.Now.Day);

#>


using System;

    static partial class InternalAssemblyInfo {

		public const string AssemblyBuildDateString = "<#= Date #>" ;
		public static DateTimeOffset AssemblyBuildDate => DateTimeOffset.Parse(AssemblyBuildDateString, System.Globalization.DateTimeFormatInfo.InvariantInfo);

        public const string Version = "<#= Version #>";

	}

[TonyValenti]

I should be able to just write the following code:

public static partial class InternalAssemblyInfo {
  public const string AssemblyBuildDate = compile(DateTimeOffset.Now.ToString(System.Globalization.DateTimeFormatInfo.InvariantInfo));
//...etc.
}

[CyrusNajmabadi]

The above worries me. It means that each time i build the code i get a different set of IL. That seems to me to be the antithesis of what i would ever want in a programming language.

[jnm2]

@CyrusNajmabadi Imagine code generators! 😱

[CyrusNajmabadi]

@jnm2 Indeed. i would think it was an anti-pattern even for code-generators to not be deterministic.

[TonyValenti]

@CyrusNajmabadi -
In those cases, the reason I make it const is so that the value can be used in Attributes.

Also, while this case does involve generating different IL each time, that is only because it is particularly useful for this case.

[jnm2]

@CyrusNajmabadi I'm not sure why I'd particularly be bothered by it. If there's a particular point where it becomes annoying, you can always just stop using DateTime.Now.

[theunrepentantgeek]

Here's an example of why non-determinism is bad ... admittedly a little contrived, but only a little ...

It's regrettably common for people to screw up date/time calculations - and there are a lot of ways to do it. Here are three examples linked to leap day:

• I've seen systems that failed on 29 February of a leap year because they calculate a date 1 year in the future by doing new DateTime(year+1, month, day);
• Others that failed on 31 December of a leap year because it was day 366 of the year, and therefore out of range;
• And I've seen systems fail on 1 March in the year prior to a leap year because they added 365 days to the date and it wasn't in March.

Combine that with compile time constants like this one (slightly modified from an example above):

public static partial class InternalAssemblyInfo {
    public const DateTime AssemblyBuildDate 
        = compile(DateTimeOffset.Now);
}

What do you now have?

You can build the application, run all the automated tests, do some manual exploratory testing, then push it into production, and everything works fine.

Later, you can build exactly the same source code with exactly the same compiler and exactly the same libraries and it now behaves in a different way.

The very fact that this would be a rare edge case makes it even worse, because practically no developer would think of this as a possible cause for odd behavior when they were trying to work out why the latest update in production is crashing.

(And, yes, I know we can get exactly the same problem right now by generating source code in our build scripts ...)

[Pzixel]

@TonyValenti

I should be able to just write the following code:

public static partial class InternalAssemblyInfo {
  public const string AssemblyBuildDate = compile(DateTimeOffset.Now.ToString(System.Globalization.DateTimeFormatInfo.InvariantInfo));
//...etc.
}

Being said, compiler that produces different output for every call is being considered a buggy one.

---

I already have said it before, but you probably don't listen: #1028 (comment)

[jrmoreno1]

@theunrepentantgeek: that's not bad, that's an external dependency. One that in rare circumstances you'd have to be aware of. But you'd have the same problem if you examined the assembly to get the build date:

https://stackoverflow.com/questions/1600962/displaying-the-build-date

You may not like this, but it happens, and it is doing what people want it to.

[arekbal]

@theunrepentantgeek:

Allowing only [Pure] and enforcing it for compile-time code should root out that kind of code(file access, network, devices, timers, all of these would go away). Taking current Date is non-pure(as in non-determinisitic) in my book.

[julealgon]

Would this allow something like:

[MyCaching(expireIn: TimeSpan.FromMinutes(10))]

?

Working with attributes currently is fairly limited because you can only pass constant values to it.

At the same time, dealing with "durations of time" (in the example above) is usually best handled by leveraging TimeSpan instead of raw int, since it provides better semantic, readability (no need to suffix variables with "seconds" and things like that) and uniformity in usage (no need for conversions to take place).

Similarly, Uri is another common type where it is a good practice to use it to represent arbitrary urls instead of using raw string, but then you cannot actually pass values to it in an attribute since they are not constant.

Would this proposal allow things like TimeSpan.FromX and Uri(string) to be constexpr methods and be used in contexts where const values are required, like attribute declarations?

[TahirAhmadov]

I've been thinking about this a little. While it's easy to understand how to allow certain static methods to be marked as "const" (or "deterministic" - word doesn't matter as much), it gets a little more complicated with struct instance methods. (Obviously classes instance methods cannot be made deterministic by definition - because a heap object by definition is not constant.) If we take something like TimeSpan, we have an immutable struct, and calling it's Add method is not much different from a hypothetical TimeSpanHelper.Add(ts1, ts2) helper method.
We can devise a rule that if a struct is immutable (as defined by readonly keyword usage), then its methods can be optionally marked "deterministic", as well.
Most importantly, this hinges on making it possible in the runtime to treat immutable structs as constants - which would also give you the attribute parameter values scenario.

[xPaw]

Kinda sad to see this discussion partially fell into bike shedding about dates. Simple compile time evaluation would be great for quickly generating things like static/const fields, including frozen collections.

The current solution of using source generators is very clunky and requires a lot of setup, and then manually outputting code as strings.

I wouldn't even care if you called methods with side effects in compile time. If you want it to be deterministic, then don't do these things. Jai for example doesn't limit that.

[CyrusNajmabadi]

Simple being the consumer use as a programmer

I don't think it's simple at all as a consumer. As I mentioned, there isn't even a viable design presented at far that is actually simple (in terms of consumption).

There's a reason why const is defined as it is in the language today. It's something actually reasonably simple to design, implement and consume.

I'd want the same for any new feature. And I think it's telling that no design has actually been presented that accomplishes that.

[xPaw]

I'm not buying your argument that it wouldn't be simple for consumers if you specified a method to be const via an attribute or keyword or any other way.

Determinism is fine, but if you intentionally use non deterministic calls, you can't expect the result to be deterministic. And that's fine.

I'm not sure what proposal you need from people who aren't designing the language here.

This discussion has been open since 2017, are there specific problems that actually have no solutions yet, or is it simply stalled because no one from Microsoft wants to do it?

[tannergooding]

Determinism is fine, but if you intentionally use non deterministic calls, you can't expect the result to be deterministic. And that's fine.

Other languages notably don't really allow this. For example, the language restrictions for C++ don't allow it. There is some onus on the author to not do something like ifdef and provide differing implementations, but that's generally viewed as acceptable since its only rationalized at the point native machine codegen happens.

You can try and hand wave it away all you want, but that simply isn't how C# (or most languages) work, particularly not for constant folding. C# has additional considerations due to not being the final compilation step, which is rather the JIT/AOT compiler instead. -- Which is also why constexpr in C++ requires it all persist in the header, same with templates, because the ecosystem and its compilation semantics fundamentally differ, due to the designs of the ecosystems.

I'm not sure what proposal you need from people who aren't designing the language here.

The language team gets hundreds of proposals for features. They do not have the time, nor would it be a good use of resources, to try and design every niche feature proposed. They have to prioritize ideas that could likely make it into the language based on overall ecosystem need and direction, as well as based on total impact.

Even in languages like C++, constexpr is relatively low impact for real world code. It isn't typical that you have cases that are full static foldable logic. Where those cases do crop up, there are often better and more productive ways to handle them. It's more a "nice to have" feature that can help improve codegen in places that the compiler isn't already handling such folding (where most such folding happens across inlining boundaries).

This discussion has been open since 2017, are there specific problems that actually have no solutions yet, or is it simply stalled because no one from Microsoft wants to do it?

It's not "no one wants to do it", it's that it's a complex space that is much lower priority than the myriad of other proposals that also exist. Things that are much higher impact, have much higher desire from the community, will light up in real world code, etc. It's also an area that often has trivial workarounds for the few cases that do crop up.

Now as for some of the issues that would have to be figured out (non comprehensive):

• determinism
• language constructs that are allowed vs disallowed
  • this includes things that are planned but can't land in v1
• handling of things that are considered UB
• fixing behaviors over time
  • for example, what if the ICU tables change and it impacts string parsing/formatting
  • this includes handling for things like say parse formats that aren't supported today but may be supported in vNext
• representation of the code in metadata
  • this includes in ref assemblies and the build/size impact from that
• how such code is evaluated
  • this includes handling for if evaluation fails at compile time
• whether any bread crumbs should be left behind to indicate the transforms that occurred
• whether it should be done by the C# compiler or the JIT/AOT compiler
• etc

This is a feature that C++ didn't land until C++11, and even thin in a very minimal capacity. It is a feature that is still being evolved in C++ today and which is rarely used for real world code; it's mostly used for some cool microbenchmarks and samples. It is a feature that is still majorly limited in C++ and which can trivially lead to quirks and other problems, including significantly longer build times when misused.

---

The C# language team has hundreds of years of combined experience designing languages. Many have worked on other major (non .NET based) languages throughout their careers as well. They generally know what they're doing here and the best way to drive a language forward, the complexities involved, what it takes to achieve success and especially success over time, etc.

When they choose to do something, to not do something, or to prioritize other feature/design work; it is typically for very good reason and while it may not be obvious to every developer who has an idea on what to add, it is what helps attribute to C#'s 20+ years of success and continued growth.

[xPaw]

With the list of issues you provided, are all of these irrelevant and solved in source generators?

If you want an example of compile time execution that isn't C++, here's Jai: https://github.com/Ivo-Balbaert/The_Way_to_Jai/blob/main/book/26A_Metaprogramming.md#262-running-code-at-compile-time-with-run

We already know that we need to use #run main() to run the whole program at compile-time. In fact, any code (a code-line, a block of code or a procedure) can be run at compile time with #run. #run produces a constant as result.

This approach to me feels well done, where you simply specify a keyword to execute a particular piece of code at runtime, and the result is stored as const.

--

fixing behaviors over time

this feels like an argument that could be applied to any library provided by the runtime, which notably .NET has done breaking changes to behavior that required changing code to fix the behavior, but the compiler did not warn that such breaking changes occurred.

[tannergooding]

With the list of issues you provided, are all of these irrelevant and solved in source generators?

Some similar issues may exist, but overall they are very different due to it being standard source and evaluating at the same time and in the same environment that all the other code is also evaluated in.

This approach to me feels well done, where you simply specify a keyword to execute a particular piece of code at runtime, and the result is stored as const.

I expect most major languages would disagree. This can introduce a number of problems and other bugs

this feels like an argument that could be applied to any library provided by the runtime, which notably .NET has done breaking changes to behavior that required changing code to fix the behavior, but the compiler did not warn that such breaking changes occurred.

The difference is where the issues can arise and how you can track breaking changes. In general, the core of .NET tends to only do bug fixes and feature additions. We rarely take breaking changes outside of bugs or other known problematic behavior and weigh the cost vs benefit.

However, with that being said, within a single app results tend to be deterministic. With constexpr, this can change as something that is constant evaluated can now differ from the result you'd observe at runtime. This can lead to bugs, undefined behavior, and other problems much more trivially. You can of course experience similar issues if you try to serialize state across runs or do other things as well, but those tend to be much rarer and easier to track down.

i.e. With standard code, source generators, and other scenarios you don't typically have to worry about Sin(x) and Sin(5) producing different results (when x == 5) just because one was constant and the other wasn't and some "off by 1ulp" bug was found and fixed between the two. This is an actual concern for something involving constexpr and it is something that C++ has struggled with. It is also why many STL and core runtime APIs are still not constexpr (they are only just planning to mark them in C++26, after significant effort and checking of compliance for existing algorithms provided by major compilers).

[TahirAhmadov]

A general purpose source generator can be devised:

[Const]
static int GetLength(string str) => str.Length;
[GenerateConst]
static int HelloWorldLength() => GetLength("Hello World");
// generated
const int HelloWorldLengthConst = 11;

The difficult, but doable part, would be for SG to compile the static method. It would need to ensure that all dependencies are either methods decorated with [Const], or one of methods in BCL which can be deemed as "const" (like Length property on string, etc.).

I never did anything like this, perhaps people who know the Roslyn API better than me can pitch in with respect to feasibility of this approach. One immediate issue I can think of poor performance of such an SG.

[spydacarnage]

When I asked recently, I was given this link: https://github.com/dotnet/roslyn/blob/main/docs/features/interceptors.md

The important part being:

Interceptors is a C# compiler feature, first shipped experimentally in .NET 8, with stable support in .NET 9.0.2xx SDK and later.

[julealgon]

Ok, which one of you did it?

• https://github.com/sebastienros/comptime

[TonyValenti]

I haven't seen that before but that looks super amazing! I've wanted something like that forever.

[TahirAhmadov]

This SG is great in that it shows how to accomplish the compile time execution of marked methods. The only thing it's missing is actual generation of const int HelloWorldLengthConst = 11; fields, so that downstream code can use the results of these calculations without needing to use this SG globally - if I understand correctly, it has to intercept each place where the values are used, which requires more projects to reference the SG, and doesn't allow a library author to publish a DLL with constants ready to go.

[Blizzardo1]

Ok, which one of you did it?

* https://github.com/sebastienros/comptime

Don't blame me, Blame @sebastienros!
Also how cool this actually might be as it would likely speed up runtime 10-fold for larger datasets. It would be awesome if there was a way to decorate variables such as with a keyword comp for known datasets such as embedded datasets or of a certain length list like in the comptime repo: Constants.GetPrimeNumbers();

I know it's farfetched and likely to never be implemented, but at the end of the day, it does sound like a promising keyword let alone an Function Attribute.