Unify nf-lang config scopes with runtime classes (#6271)

---------

Signed-off-by: Ben Sherman <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
Co-authored-by: Paolo Di Tommaso <[email protected]>

Author: bentsherman

docs/developer/config-scopes.md

@@ -0,0 +1,176 @@
+(config-scopes-page)=
+
+# Configuration scopes
+
+This page provides guidance on defining configuration scopes in the Nextflow runtime.
+
+## Overview
+
+The Nextflow configuration is defined as a collection of *scope classes*. Each scope class defines the set of available options, including their name, type, and an optional description for a specific configuration scope.
+
+Scope classes are used to generate a configuration schema, which is in turn used for several purposes:
+
+- Validating config options at runtime (`nextflow run` and `nextflow config`)
+
+- Providing code intelligence in the language server (validation, hover hints, code completion)
+
+- Generating reference documentation (in progress)
+
+Scope classes are also used by the runtime itself as type-safe domain objects. This way, the construciton of domain objects from the configuration map is isolated from the rest of the runtime.
+
+## Definition
+
+### Config scopes
+
+A *config scope* is defined as a class that implements the `ConfigScope` interface. Top-level scope classes must have the `@ScopeName` annotation, which defines the name of the config scope.
+
+For example:
+
+```groovy
+package nextflow.hello
+
+import nextflow.config.schema.ConfigScope
+import nextflow.config.schema.ScopeName
+
+@ScopeName('hello')
+class HelloConfig implements ConfigScope {
+}
+```
+
+A scope class must provide a default constructor, so that it can be instantiated as an extension point. If no such constructor is defined, the config scope will not be included in the schema. In the above example, this constructor is implicitly defined because no constructors were declared.
+
+The fully-qualified class name (in this case, `nextflow.hello.HelloConfig`) must be included in the list of extension points.
+
+### Config options
+
+A *config option* is defined as a field with the `@ConfigOption` annotation. The field name determines the name of the config option.
+
+For example:
+
+```groovy
+    @ConfigOption
+    String createMessage
+```
+
+The `@ConfigOption` annotation can specify an optional set of types that are valid in addition to the field type. For example, the `fusion.tags` option, which accepts either a String or Boolean, is declared as follows:
+
+```groovy
+    @ConfigOption(types=[Boolean])
+    String tags
+```
+
+The field type and any additional types are included in the schema, allowing them to be used for validation.
+
+The field type can be any Java or Groovy class, but in practice it should be a class that can be constructed from primitive values (numbers, booleans, strings). For example, `Duration` and `MemoryUnit` are standard Nextflow types that can each be constructed from an integer or string.
+
+### Nested scopes
+
+A *nested scope* is defined as a field whose type is an implementation of `ConfigScope`. The field name determines the name of the nested scope.
+
+The scope class referenced by the field type defines config options and scopes in the same manner as top-level scope classes. Unlike top-level scopes, nested scope classes do not need to use the `@ScopeName` annotation or provide a default constructor.
+
+See `ExecutorConfig` and `ExecutorRetryConfig` for an example of how a nested scope is defined and constructed.
+
+### Placeholder scopes
+
+A *placeholder scope* is a config scope that applies to a collection of user-defined names.
+
+For example, the `azure.batch.pools` scope allows the user to define a set of named pools, where each pool is configured with a standard set of options such as `autoScale`, `lowPriority`, `maxVmCount`, etc. These options are defined in a placeholder scope with a placeholder name of `<name>`. Thus, the generic name for the `autoScale` option is `azure.batch.pools.<name>.autoScale`.
+
+A placeholder scope is defined as a field with type `Map<String, P>`, where `P` is a nested scope class which defines the scope options. The field should have the `@PlaceholderName` annotation which defines the placeholder name (e.g. `<name>`).
+
+See `AzBatchOpts` and `AzPoolOpts` for an example of how placeholder scopes are defined and constructed.
+
+### Descriptions
+
+Top-level scope classes and config options should use the `@Description` annotation to provide a description of the scope or option. This description is included in the schema, which is in turn used by the language server to provide hover hints.
+
+For example:
+
+```groovy
+@ScopeName('hello')
+@Description('''
+    The `hello` scope controls the behavior of the `nf-hello` plugin.
+''')
+class HelloConfig implements ConfigScope {
+
+    @ConfigOption
+    @Description('''
+        Message to print to standard output when a run is initialized.
+    ''')
+    String createMessage
+}
+```
+
+Nested scopes and placeholder scopes may also use this annotation, but will inherit the description of the top-level scope by default.
+
+### Best practices
+
+The Nextflow runtime adheres the following best practices where appropriate:
+
+- Config options should be declared as public and final, so that the scope class can be used as an immutable domain object.
+
+- Scope classes should define a constructor that initializes each field from a map, casting each map property to the required type and providing default values as needed.
+
+- In cases where an option defaults to an environment variable, the environment map should be provided as an additional constructor argument rather than accessing the system environment directly.
+
+- In cases where an option with a primitive type (e.g., `int`, `float`, `boolean`) can be unspecified without a default value, it should be declared with the equivalent reference type (e.g. `Integer`, `Float`, `Boolean`), otherwise it should use the primitive type.
+
+- In cases where an option represents a path, it should be declared as a `String` and allow clients to construct paths as needed, since path construction may depend on plugins which aren't yet loaded.
+
+For example:
+
+```groovy
+import nextflow.config.schema.ConfigOption
+import nextflow.config.schema.ConfigScope
+import nextflow.config.schema.ScopeName
+
+@ScopeName('hello')
+class HelloConfig implements ConfigScope {
+
+    @ConfigOption
+    final String createMessage
+
+    @ConfigOption
+    final boolean verbose
+
+    HelloConfig() {}
+
+    HelloConfig(Map opts, Map env) {
+        this.createMessage = opts.createMessage ?: env.get('NXF_HELLO_CREATE_MESSAGE')
+        this.verbose = opts.verbose as boolean
+    }
+}
+```
+
+## Usage
+
+### Runtime
+
+Nextflow validates the config map after it is loaded. Top-level config scopes are loaded by the plugin system as extension points and converted into a schema, which is used to validate the config map.
+
+Plugins are loaded after the config is loaded and before it is validated, since plugins can also define config scopes. If a third-party plugin declares a config scope, it must be explicitly enabled in order to validate config options from the plugin. Otherwise, Nextflow will report these options as unrecognized.
+
+Core plugins are loaded automatically based on other config options. Therefore, Nextflow only validates config from a core plugin when that plugin is loaded. Otherwise, any config options from the plugin are ignored -- they are neither validated nor reported as unrecognized.
+
+For example, when the `process.executor` config option is set to `'awsbatch'`, the `nf-amazon` is automatically loaded. In this case, all options in the `aws` config scope will be validated. If the executor is not set to `'awsbatch'`, all `aws` options will be ignored. This way, config files can be validated appropriately without loading additional core plugins that won't be used by the run.
+
+The scope classes themselves can be used to construct domain objects on-demand from the config map. For example, an `ExecutorConfig` can be constructed from the `executor` config scope as follows:
+
+```groovy
+new ExecutorConfig( Global.session.config.executor as Map ?: Collections.emptyMap() )
+```
+
+:::{note}
+In practice, it is better to avoid the use of `Global` and provide an instance of `Session` to the client class instead.
+:::
+
+### JSON Schema
+
+Config scope classes can be converted into a schema with the `SchemaNode` class, which uses reflection to extract metadata such as scope names, option names, types, and descriptions. This schema is rendered to JSON and used by the language server at build-time to provide code intelligence such as code completion and hover hints.
+
+### Documentation
+
+The schema described above can also be rendered to Markdown using the `MarkdownRenderer` class. It produces a Markdown document approximating the {ref}`config-options` page.
+
+This approach to docs generation is not yet complete, and has not been incorporated into the build process yet. However, it can be used to check for discrepancies between the source code and docs when making changes. The documentation should match the `@Description` annotations as closely as possible, but may contain additional details such as version notes and extra paragraphs.

docs/developer/plugins.md

@@ -133,6 +133,9 @@ import nextflow.script.dsl.Description
 ''')
 class MyPluginConfig implements ConfigScope {
 
+    /* required by extension point -- do not remove */
+    MyPluginConfig() {}
+
     MyPluginConfig(Map opts) {
         this.createMessage = opts.createMessage
     }
@@ -143,7 +146,7 @@ class MyPluginConfig implements ConfigScope {
 }
 ```
 
-While this approach is not required to support plugin config options, it allows Nextflow to recognize plugin definitions when validating the config.
+While this approach is not required to support plugin config options, it allows Nextflow to recognize plugin definitions when validating the config. See {ref}`config-scopes-page` for more information.
 
 ### Executors
 

docs/google.md

@@ -2,6 +2,8 @@
 
 # Google Cloud
 
+(google-credentials)=
+
 ## Credentials
 
 Credentials for submitting requests to the Google Cloud Batch API are picked up from your environment using [Application Default Credentials](https://github.com/googleapis/google-auth-library-java#google-auth-library-oauth2-http). Application Default Credentials are designed to use the credentials most natural to the environment in which a tool runs.

docs/index.md

@@ -152,6 +152,7 @@ migrations/index
 
 developer/index
 developer/diagram
+developer/config-scopes
 developer/packages
 developer/plugins
 ```

docs/notifications.md

@@ -220,17 +220,26 @@ mail {
 }
 ```
 
+:::{note}
+Some versions of Java (e.g. Java 11 Corretto) do not default to TLS v1.2, and as a result may have issues with 3rd party integrations that enforce TLS v1.2 (e.g. Azure Active Directory OIDC). This problem can be addressed by setting the following config option:
+
+```groovy
+mail {
+    smtp.ssl.protocols = 'TLSv1.2'
+}
+```
+:::
+
 See the {ref}`mail scope <config-mail>` section to learn more the mail server configuration options.
 
 ### AWS SES configuration
 
 :::{versionadded} 23.06.0-edge
 :::
 
-Nextflow supports [AWS SES](https://aws.amazon.com/ses/) native API as an alternative
-provider to send emails in place of SMTP server.
+Nextflow supports the [AWS Simple Email Service](https://aws.amazon.com/ses/) API as an alternative provider to send emails in place of an SMTP server.
 
-To enable this feature add the following environment variable in the launching environment:
+To enable this feature, set the following environment variable in the launch environment:
 
 ```bash
 export NXF_ENABLE_AWS_SES=true
@@ -242,6 +251,20 @@ Make also sure to add the following AWS IAM permission to the AWS user (or role)
 ses:SendRawEmail
 ```
 
+The following snippet shows how to configure Nextflow to send emails through SES:
+
+```groovy
+mail {
+    smtp.host = 'email-smtp.us-east-1.amazonaws.com'
+    smtp.port = 587
+    smtp.user = '<Your AWS SES access key>'
+    smtp.password = '<Your AWS SES secret key>'
+    smtp.auth = true
+    smtp.starttls.enable = true
+    smtp.starttls.required = true
+}
+```
+
 ## Mail notification
 
 You can use the `sendMail` function with a {ref}`workflow completion handler <metadata-completion-handler>` to notify the completion of a workflow completion. For example: