[RFC] Annotations 2.0

[Blacksmoke16]

I wanted to get an issue opened for tracking/discussion regarding the future of annotations. This issue will essentially be a reiteration of my thoughts from https://forum.crystal-lang.org/t/rfc-annotations-metadata-declaration-dsl/1312/7?u=blacksmoke16 and some of my thoughts regarding annotations in general.

Background

The concept of annotations and how they fit into the macro system works quite well at the moment. Annotations can be used to store metadata that can then be acted upon via macro code. However, things aren't as smooth when you want to access that annotation data at runtime. #7694 helped quite a bit in this regard as it is now possible to do ann.named_args.double_splat, where that double_splat could be into the constructor of a struct for example. This is a pattern I use quite a bit. But there is still an area where this pattern lacks: matching the annotation to its corresponding struct.

In the past this hasn't been too big of a problem for me as my use cases were mainly static mappings that are not use configurable. Or have been simple enough where I could get away with abstractions like https://github.com/athena-framework/config/blob/master/src/athena-config.cr#L61-L70. Unfortunately with my recent work on https://github.com/athena-framework/validator, this issue has become harder and harder to work around.

Example problem

The validator implementation uses constraints like NotBlank or EqualTo. These are represented as classes. You can then apply an annotation to a property to have it be validated against that specific constraint, e.x. Assert::NotBlank. The major challenge with this, especially when dealing with custom constraints that are nested, i.e. #9766; is there is not a good way to know which constraint a given annotation should be "handled" by.

Suggested Solution

The best way I can think of to solve this would be to remove the need for there to be two types in the first place, or in other words, make the constraint the annotation itself. I.e. something like:

@[Annotation]
class NotBlank
  def initialize(...); end
end

alternatively:

annotation NotBlank
  # Allow defining initializers/methods/etc within the `annotation` type
  def initialize(...); end
end

In either case the end goal would be to more tightly couple the type that gets applied to something and the logic that annotation should represent at runtime. The initializer (or some other DSL) would define the specific fields that an annotation supports, which if an unexpected one is provided a compile time error would be raised.

I'm personally in favor of the @[Annotation] annotation because it makes annotations less unique and would inherently support methods, inheritance, modules, etc.

I think ideally the current semantics around how annotations work in macro land would remain. I could see there being a new method added to Crystal::Macros::Annotation like #build or something along those lines. This method would essentially be doing like @type.new self.args.splat, self.named_args.double_splat where @type is the type the annotation is related to (assuming the first example). Or maybe just have the annotation expand to that when doing like {{ann}} and remove the need for the method (if that would even be possible).

Implementation

My, probably naive thinking, is that not much would need to change on the compiler side of things; mainly given the macro land representation wouldn't be affected only how an annotation is defined. Then of course the logic around the handling type. Probably another argument on AnnotationDef? I guess my main question would be how to implement the @[Annotation] annotation, but I'm thinking it wouldn't be much different than the other built in annotation types like @[Flags], as it could directly instantiate an AnnotationType?

IMO, this, #8148, #9246, and #8835 would be a game changer in the usability/effectiveness of annotations, and macros in general.

WDYT?

[Blacksmoke16]

After hacking around the compiler a bit I think the main challenge of this will be how to represent a type that is essentially two types. I.e.

@[Annotation]
record MyAnn, id : In32

MyAnn is both something that can be applied via @[MyAnn] and a struct that can be instantiated directly, MyAnn.new 10.

I'm going to take a break from messing around on this until some more discussions can be had.

[erdnaxeli]

What you are proposing, with the @[Annotation], is the possibility to declare a type as an annotation. This is a paradigm shift from today where an annotation is its own kind of object.

I kinda like your idea. You take a type and you add a metadata to say to the compiler "this type can be used as an annotation". Then what I would imagine is that everywhere you can access this annotation, you actually get an instance of the type (in macros or at runtime). And parameters to the annotation (in @[MyAnn("a", "b", "c")]) should be given to initialize like when you instantiate any type.

[Blacksmoke16]

@erdnaxeli Yes exactly. The data could still be accessed at compile time, but could also be used to get an instance of the related type at runtime. For example (from the forum thread):

@[Annotation]
record MyAnn, name : String, active : Bool = true

@[MyAnn(name: "foo")
def foo
  {{ @def.annotation(MyAnn)[:name] }}
end

foo # => "foo"

@[MyAnn(active: false)
def bar 
  "bar"
end

bar # => Compile error because `name` wasn't supplied and it's not nilable nor has a default value

@[MyAnn(name: "baz")]
def baz
  {{ @def.annotation(MyAnn).build }}
end

baz # => MyAnn(@name="baz", @active=false)

This is essentially what PHP8 is doing as well with their new attributes concept: https://stitcher.io/blog/attributes-in-php-8.

[erdnaxeli]

Indeed PHP8 looks great (except for the part where you can annotate a function parameter :s)! Something similar would be very cool.

[Blacksmoke16]

Now that 1.0.0 is released maybe this can/will get more attention? 🙂.

[watzon]

I'd love to see someone get a working POC together, Ary style. I know this isn't super high priority for the core team.

[devnote-dev]

I've been thinking about this a lot recently and would like to give my two cents on this:

 @[Annotation]
 class NotBlank
   def initialize(...); end
 end

I think it's important that Crystal does not follow in the footsteps of other languages that use classes for everything. Ignoring the underlying implementation in the compiler, these are separate types with individual purposes, I don't think that we should take away from that, especially with annotations. At its core, annotations are a very unique construct in Crystal, and should not be treated as just another class with specific functionality.

 annotation NotBlank
   # Allow defining  initializers/methods/etc within the `annotation` type
   def initialize(...); end
 end

This feels like a better approach given that the majority of annotation handling is done via macro methods. I also believe that this would be useful for opening the gateway to using macro methods (or "macro defs") in a macro context (#8835). Take the following example:

annotation Deprecated
  def initialize(message : StringLiteral)
  end

  def initialize(*, since : StringLiteral, use : StringLiteral)
    initialize "since #{since}, use #{use}"
  end
end

@[Deprecated(since: "0.3.0", use: "Bar")]
class Foo
end

class Bar
end

I'm not saying this should be the case for the Deprecated annotation, but it is definitely something that could be experimented with for annotations, and eventually macro defs. The forum post for this highlights a few useful points about this:

Having an annotated macro to specifically handle annotation logic. I believe this should have been available in the language already as the current methods for going about handling annotations can be complicated at times, and does not work in all cases - you can't (effectively) handle annotations on modules or enums, and there's no way to handle annotations on annotations.

Being able to restrict an annotation's target. I don't think this requires having specific DSL logic, or even its own @[Target] annotation, but rather it be declared in the annotation signature:

annotation Foo(:classes, :structs) # only works on classes and structs
end

annotation Bar(:methods) # only works on methods
end

annotation Baz # works on all constructs because there are no restrictions
end

Alternatively, HertzDevil's suggestion about handling it in the annotated macro would also work, but would probably need more code for handling the different targets (which isn't really a bad thing). It would also likely require a few nodes from this long list to be implemented before macro annotated(target, *args, **options) would become stable.

[Blacksmoke16]

To summarize, my ideal new features for annotations would include:

• Limit the targets they can be applied to
• Be more strict in regards to what positional/named arguments they accept
  • Ideally being able to document these too
• Able to be used at both runtime and compile time

I don't have strong feelings on exact syntax/implementation at the moment, but I just don't want for the last bullet to be forgotten. The reasoning for me leaning towards @[Annotation] was to make them less of a special thing that basically just allows reusing the annotated type's name also as an annotation with the added benefit of providing a convenient way to obtain an instance of the type based on the positional/named arguments from the annotation.

In this way, you're not limiting the behavior of the annotation keyword to either that of a class, or struct, or something in between since you could apply the annotation to either a class or struct depending on your use case and it would just tell the compiler to treat that name as if it was also an annotation. Which is how PHP handles it. However based on how Kotlin handles them, another option instead of the @[Annotation] annotation could be something like:

• annotation Foo - Compile time only annotation
• annotation class Foo - Compile time + runtime class semantics
• annotation struct Foo - Compile time + runtime struct semantics

This would make them less unique, and probably easier to parse.

Ultimately tho, there are other ways to accomplish the use case that doesn't involve any of this. E.g. some annotation/syntax to establish the relationships between an annotation and external class/struct type versus treating them as one in the same. Just have to figure out the pros and cons of each approach.

[Blacksmoke16]

I'd also like to add being able inherit annotations like you can classes. I think this could lead to some interesting use cases where some type could check for annotations of some parent type, and inherently handle more specialized versions of it out of the box.

Being able to call the parent annotations constructor/have default values for a specific annotation field would also be 💯 when paired with this.