RFC 0017: Improved Annotations

[Blacksmoke16]

Preview: https://github.com/crystal-lang/rfcs/blob/improved-annotations/text/0017-improved-annotations.md
Proof of concept PR: crystal-lang/crystal#16527

Am planning on play testing this with Athena Validator, but in the meantime looking to gather some feedback :)

[Blacksmoke16]

Open to suggestions here. I guess another option is doing something like what PHP did and go with @[Attribute] to make the distinction more clear.

[crysbot]

This pull request has been mentioned on Crystal Forum. There might be relevant details there:

https://forum.crystal-lang.org/t/rfc-annotations-metadata-declaration-dsl/1312/14

[Blacksmoke16]

I ended up implementing metadata in the initial draft of this just to avoid the case of releasing this feature, someone adds same annotation to same thing multiple times, then we introduce repeatable and have to make it default to true. Having it out of the box allows the default to be the safer false value.

[Blacksmoke16]

Open to ideas on how to handle this. This is straightforward enough but 🤷.

[Blacksmoke16]

Another idea would be like targets: Annotation::Target[:method, :property].

[Blacksmoke16]

This is one of the bigger open questions. I just felt we needed this to be opt-in, otherwise you may always have to deal with getting more than you want, but still need a way to allow including child types too. Not 100% sold on this name tho.

Having the provided type be a module is also kinda interesting. But modules are also stored in the #ancestors array so kinda just get it for free.

[Blacksmoke16]

As discussed on Discord a bit, I think this warrants a bit more thought. Specifically how current annotations can accept any macro ASTNode that isn't really usable within a runtime constructor. As it stands with the PoC implementation you can do stuff like:

@[Annotation]
class Foo
  def self.new(test : Crystal::Macros::Def); end
end

Or how def self.new(*args, **kwargs); end could also be used to get existing annotation semantics that allow anything. But neither of these "feel" right mixing compile time only types with runtime constructors.

---

I'm now thinking it would be better to fully de-couple the constructors of the runtime type and that of the annotation, possibly with some sort of pseudo-overload mechanic. The annotation could still store whatever internally, but the provided args (both named and positional) would have to match one of the annotation constructors. Type validation would be a bit easier for these since they'd be using the macro ASTNode types vs having to map from the runtime types.

For ease of use, could maybe still use runtime constructors as a fallback if no annotation-constructor(s) are present. But if any annotation constructor(s) are present, they must be used and can no longer fallback on the runtime ones.

This would help with the compile-time only/zero-overhead use case that existing annotations currently serve, but with added benefit of input argument validation.

[Blacksmoke16]

Not sold on this name, but just went with this for now.

[BlobCodes]

• I'm not really sold on the idea that annotations should even be exposed to the runtime.

  The referenced PHP implementation (and others alike, ex. Java) all use some form of Reflection API,
  which is just a generic API that allows you to inspect and modify the code while it is being executed.
  That's a perfect fit for VM-based/interpreted languages since this allows meta-programming without reinventing another language just for it. Annotations (or attributes, as PHP calls it) aren't accessible without this Reflection API. Speaking in crystal terms, PHP's getAttributes method would be implemented on the Crystal::Macros::TypeNode, not the Class (it's defined on ReflectionClass and returns a ReflectionAttribute). It just happens that you can access the TypeNode equivalent in normal code.

  As a compiled language with clear separation between compile-time and runtime, I think this would be a very bad fit.
  I think we should orient ourselves more towards pure compile-time annotations, e.g. Rust.

• I think it should be possible to achieve most of what this RFC allows by adding a macro annotated mechanism, which is far simpler than the RFC and feels more crystal-like.

  This RFC also includes a macro annotated mechanism, but in a completely different way than I'd expect it (and as was discussed in the Discord).

  Similar to macro included, macro inherited and macro extended, I would expect a macro annotated to be given both the object the annotation was applied to and the annotation itself. This would allow early parameter validation and encapsulated logic for simpler annotations.

  I think this example from @HertzDevil explains the concept quite well:

  module Annotations
    record NotBlank, allow_nil : Bool, message : String
  end
  
  annotation NotBlank
    macro annotated(ann, node)
      {%
        node.raise ... unless node.is_a?(TypeDeclaration)
        args = ann.named_args
        ann.raise ... if args.has_key?(:allow_nil) && !args[:allow_nil].is_a?(BoolLiteral)
        ann.raise ... if args.has_key?(:message) && !args[:message].is_a?(StringLiteral)
        allow_nil = !!ann[:allow_nil]
        message = ann[:message] || "This value should not be blank."
        CONSTRAINTS << {ann.name, {allow_nil, message}}
      %}
    end
  end
  
  class User
    private CONSTRAINTS = [] of _
  
    @[NotBlank(allow_nil: true)]
    property name : String = "Jim"
  
    def validate!
      {% for v, i in CONSTRAINTS %}
        {% name, args = v %}
        %constraint{i} = ::Annotations::#{name}.new(#{args.splat})
      {% end %}
    end
  end

  The macro annotated mechanism shown in this RFC should probably be called macro constructed since it only accepts the arguments given to construct the annotation, not the annotated object.

• I think the RFC is quite complicated for what it delivers.

  I expect most (nearly all) annotations to remain compile-time-only metadata objects, even if all these features would be added. Still, a new layer of complexity is added to all use-cases.

  Why should a beginner decide between @[Annotation] class and @[Annotation] struct when there is no distinction between the two for most use-cases (compile-time metadata)? It feels like an unnecessary hurdle only to cover very few niche use-cases.

• I think the way the runtime constructors are used as fallback-constructors for macro land is confusing and exposes lots of rough edges.

• The RFC seems to expect features from macro land that just don't exist (yet) like type restrictions inside macros.

[Blacksmoke16]

Thanks for the feedback!

I just want to clarify that this proposal is not allowing annotations to be queried at runtime of course. They still are a compile-time only thing. The proposal is mainly allowing other types to be used as annotations, and exposing a new macro method to generate a Call to instantiate that type with the args stored in the annotation at runtime.

It does add two meanings to a given type, but the runtime and compile-time sides are still separate things. To me it just makes sense that you can use one type as the source of truth. Mainly so you don't have to think about I have to use x to annotate something but y when I'm doing it manually.

I do think being explicit and having the macro be like macro annotation_constructor would be better than the current name.

I would argue that a beginner isn't likely going to be using annotations anyway, so seems reasonable to treat them as more of an intermediate/advanced concept.

I made this RFC as kind of how I'd like to see/use annotations. There is definitely scope that could be cut to make things simpler; e.g. falling back on the runtime constructors, the pseudo-overload mechanic of the macro annotation constructors, using default values from the used overload, etc.

[BlobCodes]

The proposal is mainly allowing other types to be used as annotations, and exposing a new macro method to generate a Call to instantiate that type with the args stored in the annotation at runtime.

I think being able to interact with runtime types from within macro land on a deeper level could be an improvement to the language in general. However, I don't think this should be limited to annotations and I think this requires a more holistic approach.

Why should only annotations gain this special treatment of deducting macro behaviour from runtime behaviour? If it's possible for the compiler to map runtime behaviour to macro behaviour, why shouldn't I be able to instantiate or use structs as values inside macros?

[Blacksmoke16]

Why should only annotations gain this special treatment of deducting macro behaviour from runtime behaviour?

Probably just because this was the first time that the idea was brought up. I'm not really aware of any other discussions around this idea. As I mentioned we could ultimately defer that part of the implementation and handle it as part of a dedicated RFC/track of work. Would really just mean you have to define the macro constructors which would be a bit annoying, but not the end of the world.