WIP: Parallel Trial Scheduler

.github/instructions/README.md

@@ -0,0 +1,20 @@
+# Custom Copilot Instructions
+
+This directory contains custom instructions for the Copilot AI assistant.
+The instructions are designed to guide the AI in providing responses that align with specific project needs and preferences.
+
+## Organization
+
+- Language specific instructions go in their own file, with the cross task instructions in the root of this directory.
+
+  (e.g., [python.instructions.md](python.instructions.md), [bash.instructions.md](bash.instructions.md), [markdown.instructions.md](markdown.instructions.md), [json.instructions.md](json.instructions.md)).
+
+- Instructions relevant to all languages go in the `default.instructions.md` file.
+
+  (e.g., [default.instructions.md](default.instructions.md)).
+
+## See Also
+
+- <https://code.visualstudio.com/docs/copilot/copilot-customization>
+- [.vscode/settings.json](../.vscode/settings.json)
+  - Configures which of these instructions are used for each file type.

.github/instructions/bash.instructions.md

@@ -0,0 +1,33 @@
+---
+applyTo: '**/*.sh'
+---
+
+# Bash shell scripting language instructions
+
+- Include instructions from [default.instructions.md](default.instructions.md) for all languages.
+
+- Scripts schould use `set -e` or `set -o errexit` to exit on error.
+
+- Scripts should use use `set -u` or `set -o nounset` to exit on unset variables.
+
+- Scripts should use `set -o pipefail` to exit on errors in pipelines.
+
+- Commands should be checked for non-zero exit codes and either handled or reported.
+
+- Scripts should use portable syntax for MacOS vs. Linux
+
+- Scripts should validate input.
+
+- Scripts should include usage instructions.
+
+- Scripts should be executable (e.g., `chmod +x`).
+
+- Scripts should include a shebang line (e.g., `#!/usr/bin/env bash`).
+
+- Scripts should be well commented.
+
+- Scripts should include documentation updates if needed.
+
+- Scripts should be well formatted.
+
+- `if` `then` statements should be on the same line.

.github/instructions/default.instructions.md

@@ -0,0 +1,16 @@
+---
+applyTo: '**'
+---
+
+# Default language review selection instructions
+
+- PRs should be small and focused on a single change or feature.
+- PRs should have a clear description of what the change is and why it is needed.
+- PRs should have tests that cover the changes made.
+- PRs should try not to include any commented-out code.
+- PRs should not include any unnecessary files or changes to files that are not related to the change being made.
+- PRs should not include any changes to the README or other documentation unless it is directly related to the change being made.
+- PRs should be well commented.
+- PRs should include documentation updates if needed.
+- PRs should try to avoid unnecessary formatting changes or else keep them to their own PR that is just for formatting.
+- PRs that change the CI pipeline or pre-commit hooks should generally be kept to their own PR.

.github/instructions/json.instructions.md

@@ -0,0 +1,15 @@
+---
+applyTo: '**/*.json,**/*.jsonc,**.json5'
+---
+
+# JSON language instructions
+
+- Include instructions from [default.instructions.md](default.instructions.md) for all languages.
+
+- Files with a `.json` extension that are ARM templates or JSON schema files should be well formatted and valid JSON, without any comments, trailing commas, etc.
+
+- Files with a `.json` extension that are VSCode settings files (e.g., inside the [.vscode](../../../.vscode)) or [.devcontainer](../../../.devcontainer) directories) should be well formatted and valid JSON, but may contain comments, trailing commas, etc.
+
+- Files with a `.jsonc` or `.json5` extension should be well formatted and valid JSON5 or JSONC or JSON, and can include comments, trailing commas, etc.
+
+- If a file is an `mlos_bench` config, it should have a `.mlos.jsonc` of `.mlos.json` or `.mlos.json5` extension, and should generally match the schemas defined in the [mlos_bench/configs/schemas/](../../../mlos_bench/mlos_bench/config/schemas/) directory (e.g., [mlos-bench-config-schema.json](../../../mlos_bench/mlos_bench/config/schemas/mlos-bench-config-schema.json)), unless it is a test config under the [tests/configs/schemas](../../../mlos_bench/mlos_bench/tests/configs/schemas/) directory.

.github/instructions/markdown.instructions.md

@@ -0,0 +1,9 @@
+---
+applyTo: '**/*.md'
+---
+
+# Markdown language instructions
+
+- Include instructions from [default.instructions.md](default.instructions.md) for all languages.
+- Documentation should include relative links to other documentation files whenever possible.
+- Markdown files should be well formatted and valid Markdown conforming to markdownlint rules.

.github/instructions/python.instructions.md

@@ -0,0 +1,85 @@
+---
+applyTo: '**/*.py'
+---
+
+# Python language file instructions
+
+- Include instructions from [default.instructions.md](default.instructions.md) for all languages.
+
+- All functions, methods, classes, and attributes should have docstrings.
+
+- Docstrings should include Sphinx style crossref directives for functions, methods, classes, attributes, and data whenever possible using `:py:class:` or `:py:func:` or `:py:meth:` or `:py:attr:` or `:py:data` syntax, respectively,
+
+  See Also <https://www.sphinx-doc.org/en/master/usage/domains/python.html#python-xref-roles>
+
+- Docstrings for modules should include a summary of the module's purpose and any important details about its usage.
+
+  - Module docstrings should also include an executable example of how to use the module, including any important functions or classes or configuration options (especially those derived from a JSON config file) like any of those in `mlos_bench.environments`, `mlos_bench.services`, `mlos_bench.schedulers`, `mlos_bench.optimizers`, and `mlos_bench.storage`.
+
+    For instance:
+
+    ```python
+    '''
+    This is an example module docstring for the mlos_bench.environments.my_special_env module.
+
+    It should include some descriptive text about the module and its purpose.
+
+    Example
+    -------
+    It also includes some executable code examples.
+
+    >>> import json5 as json
+    >>> # Load a JSON config string for a MySpecialEnvironment instance.
+    >>> json_string = """
+    ... {
+    ...     "class": "mlos_bench.environments.my_special_env.MySpecialEnvironment",
+    ...     "name": "MySpecialEnvironment",
+    ...     "config": {
+    ...             "param1": 42,
+    ...             "param2": "foo",
+    ...     },
+    ... }
+    ... """
+    >>> config = json.loads(json_string)
+
+    >>> from mlos_bench.environments.my_special_env import MySpecialEnvironment
+    >>> my_env = MySpecialEnvironment(config=config)
+    >>> print(my_env)
+    MySpecialEnvironment(param1=42, param2='foo')
+    '''
+    ```
+
+  - Docstrings for classes can refer to their module docstring with `:py:mod:` cross-references for usage examples to allow easier browser navigation of generated documentation.
+
+    For instance:
+
+    ```python
+    class MySpecialEnv:
+        """
+        This is class docstring for MySpecialEnv.
+
+        It should include some descriptive text about the class and its purpose.
+
+        Example
+        -------
+        Refer to to :py:mod:`mlos_bench.environments.my_special_env` for usage examples.
+        """
+    ```
+
+- If not all arguments to a function or method fit on the same line, then they should each be on their own line.
+
+  Adding a trailing comma to the last argument is optional, but recommended for consistency whenever a single line is insufficient.
+
+- Code should be formatting using `black`.
+
+- Code should be type checked using `mypy`.
+
+  - All function and method parameters should be type annotated.
+
+- Code should be linted using `pylint`.
+
+- Tests should be included for all new code and should be run using `pytest`.
+
+- Tests should be organized roughly the same way as the code they are testing (e.g., `tests/some/test_module.py` for `some/module.py`).
+
+- Test fixtures that setup the resources for the tests (e.g., Environments, Services, Storage, Optimizer, Scheduler, etc.) should be included in a `conftest.py` file in the same directory as the tests or else a `fixtures.py` file in the same directory as the tests.

.github/instructions/rst.instructions.md

@@ -0,0 +1,11 @@
+---
+applyTo: '**/*.rst'
+---
+
+# RST markup language instructions
+
+- Include instructions from [default.instructions.md](default.instructions.md) for all languages.
+
+- Documentation should include sphinx crossref directives for functions, methods, and classes whenever possible.
+
+  See Also: <https://www.sphinx-doc.org/en/master/usage/referencing.html>

.github/prompts/README.md

@@ -0,0 +1,29 @@
+# Custom `.prompt.md` files for Github Copilot
+
+This directory contains custom `.prompt.md` files for Github Copilot.
+
+These files are used to customize the behavior of Github Copilot when generating code.
+
+The can be invoked with the `/custom-prompt-file-name-prefix` command in the Copilot Chat view (generally when in Agent mode).
+
+For instance:
+
+```txt
+/add-sphinx-crossref-to-docstrings
+```
+
+will invoke the [`add-sphinx-crossref-to-docstrings.prompt.md`](./add-sphinx-crossref-to-docstrings.prompt.md) file.
+
+Some prompts take additional arguments to help Copilot understand the context of the code being generated or other action to take.
+
+## Types of Custom Prompts
+
+There are two types of custom prompts:
+
+1. Those for MLOS developers (e.g. `add-sphinx-crossref-to-docstrings.prompt.md`).
+1. Those for MLOS users (e.g., `generate-mlos-configuration-file.prompt.md`).
+
+## See Also
+
+- <https://code.visualstudio.com/docs/copilot/copilot-customization#_prompt-files-experimental>
+- [TODO.md](./TODO.md)

.github/prompts/TODO.md

@@ -0,0 +1,19 @@
+# Custom Prompt TODOs
+
+- MLOS
+
+  - [ ] Create experiment configs
+    - [ ] Create (or reuse) environments configs
+      - [ ] Use `include` directives with `CompositeEnv` to help structure and nest things.
+    - [ ] Create (or reuse) services configs
+      - [ ] Create (or reuse) ARM templates
+    - [ ] Create (or reuse) storage configs
+    - [ ] Create (or reuse) scheduler configs
+
+- mlos-autotuning-template
+
+  - [ ] Create new X config
+    - [ ] `include` config to do Y
+  - Add `globals` variables to configure X
+  - Create (or reuse) ARM template for X
+  - Make sure scripting commands are idempotent

.github/prompts/add-config-examples-to-module-docstring.prompt.md

@@ -0,0 +1,54 @@
+# Custom Prompt: Add config examples to module docstrings
+
+Let's add config examples to module docstrings in this mlos_bench module.
+
+- Docstrings for modules should include a summary of the module's purpose and any important details about its usage.
+  - Module docstrings should also include an executable example of how to use the module, including any important functions or classes or configuration options (especially those derived from a JSON config file) like any of those in `mlos_bench.environments`, `mlos_bench.services`, `mlos_bench.schedulers`, `mlos_bench.optimizers`, and `mlos_bench.storage`.
+
+    For instance:
+
+    ```python
+    '''
+    This is an example module docstring for the mlos_bench.environments.my_special_env module.
+
+    It should include some descriptive text about the module and its purpose.
+
+    Example
+    -------
+    It also includes some executable code examples.
+
+    >>> import json5 as json
+    >>> # Load a JSON config string for a MySpecialEnvironment instance.
+    >>> json_string = """
+    ... {
+    ...     "class": "mlos_bench.environments.my_special_env.MySpecialEnvironment",
+    ...     "name": "MySpecialEnvironment",
+    ...     "config": {
+    ...             "param1": 42,
+    ...             "param2": "foo",
+    ...     },
+    ... }
+    ... """
+    >>> config = json.loads(json_string)
+
+    >>> from mlos_bench.environments.my_special_env import MySpecialEnvironment
+    >>> my_env = MySpecialEnvironment(config=config)
+    >>> print(my_env)
+    MySpecialEnvironment(param1=42, param2='foo')
+    '''
+    ```
+
+    - Configuration options for these modules should be derived from a JSON, included as a string in the module docstring so users reading the documentation can easily copy/paste, but generally they are loaded from a separate `.mlos.jsonc` config file.
+
+    - The JSON config string should be formatted using `json5` to allow for comments and trailing commas.
+
+    - The JSON config options should conform to the relevant JSON schema for the module, usually defined in the [mlos_bench/configs/schemas](../../mlos_bench/mlos_bench/config/schemas/) directory.
+      For instance:
+
+      - For an `mlos_bench.environments` module, the JSON config options should conform to the [mlos_bench/configs/schemas/environments](../../mlos_bench/mlos_bench/config/schemas/environments/environment-schema.json) schema file.
+      - For an `mlos_bench.services` module, the JSON config options should conform to the [mlos_bench/configs/schemas/services](../../mlos_bench/mlos_bench/config/schemas/services/service-schema.json) schema file.
+      - For an `mlos_bench.schedulers` module, the JSON config options should conform to the [mlos_bench/configs/schemas/schedulers](../../mlos_bench/mlos_bench/config/schemas/schedulers/scheduler-schema.json) schema file.
+      - For an `mlos_bench.storage` module, the JSON config options should conform to the [mlos_bench/configs/schemas/storage](../../mlos_bench/mlos_bench/config/schemas/storage/storage-schema.json) schema file.
+      - For an `mlos_bench.optimizers` module, the JSON config options should conform to the [mlos_bench/configs/schemas/optimizers](../../mlos_bench/mlos_bench/config/schemas/optimizers/optimizer-schema.json) schema file.
+
+    - The other options that the config can take can often also be found in the parsing of the `config` argument in the `__init__` method body of the class, but they should be included in the module docstring as well.

.github/prompts/add-sphinx-crossref-to-docstrings.prompt.md

@@ -0,0 +1,52 @@
+# Custom Prompt: Add Sphinx Crossref Links to Python Docstrings
+
+Add Sphinx cross-references to python docstrings referencing classes or functions or methods or attributes or data in this file using `:py:class:` or `:py:func:` or `:py:meth:` or `:py:attr:` or `:py:data` syntax, respectively.
+\- See Also <https://www.sphinx-doc.org/en/master/usage/domains/python.html#python-xref-roles>
+
+We don't need to do this for the parameter types listed in the Parameters or Returns sections of the docstring though.
+
+For example:
+
+```python
+def example_function(param1: MyClass, param2: MyOtherClass) -> SomeOtherType:
+    """
+    Example function working on an instance of MyClass and MyOtherClass.
+
+    Parameters
+    ----------
+    param1 : MyClass
+        An instance of MyClass.
+    param2 : MyOtherClass
+        An instance of MyOtherClass.
+
+    Returns
+    -------
+    SomeOtherType
+        An instance of SomeOtherType.
+    """
+    pass
+```
+
+should be changed to:
+
+```python
+def example_function(param1: MyClass, param2: MyOtherClass) -> SomeOtherType:
+    """
+    Example function working on an instance of :py:class:`MyClass` and :py:class:`MyOtherClass`.
+
+    Uses the :py:meth:`MyClass.method_name` method and the :py:attr:`MyOtherClass.attribute_name` attribute.
+
+    Parameters
+    ----------
+    param1 : MyClass
+        An instance of :py:class:`MyClass`.
+    param2 : MyOtherClass
+        An instance of :py:class:`MyOtherClass`.
+
+    Returns
+    -------
+    SomeOtherType
+        An instance of :py:class:`SomeOtherType`.
+    """
+    pass
+```

.github/prompts/split-large-change-branch-to-separate-change-branches.prompt.md

@@ -0,0 +1,8 @@
+# Custom Prompt: Split A Large Change Branch into Separate Change Branches
+
+The branch I have has too many changes and I want to start splitting them out to separate PRs. Can you help me do the following:
+
+1. Run `git diff main > /tmp/git-diff-main.patch` to get the differences between the `main` branch and the current branch and then use `/tmp/git-diff-main.patch` to summarize the changes branch into distinct groups. You can use the description from the Github PR associated with this branch to see what else was done and needs to be split out.
+1. Propose a sequence of changes to stage each of those sets of changes as different self contained and testable PRs.
+1. Help me create those PRs by applying that group's changes to new and different branches. We don't need to retain the history of this branch, just the changes. For instance, we could use `git checkout -b new-branch && git diff -- some/list of/files | git apply -` to make a new branch with some selected changes applied and the use stage only the relevant changes to each file for that particular PR.
+   Show me the plan and pause to check in with me between each step.

.vscode/settings.json

@@ -1,4 +1,4 @@
-// vim: set ft=jsonc:
+// vim: set ft=json5:
 {
     "makefile.extensionOutputFolder": "./.vscode",
     "files.exclude": {
@@ -147,7 +147,7 @@
     "[python]": {
         "editor.codeActionsOnSave": {
             "source.organizeImports": "explicit",
-            "source.unusedImports": "explicit"
+            //"source.unusedImports": "explicit"
         },
         "editor.defaultFormatter": "ms-python.black-formatter",
         "editor.formatOnSave": true,
@@ -170,5 +170,11 @@
     "python.testing.unittestEnabled": false,
     "debugpy.debugJustMyCode": false,
     "python.analysis.autoImportCompletions": true,
-    "python.analysis.supportRestructuredText": true
+    "python.analysis.supportRestructuredText": true,
+    "makefile.configureOnOpen": false,
+
+    "githubPullRequests.experimental.chat": true,
+    "github.copilot.chat.codesearch.enabled": true,
+    "github.copilot.chat.copilotDebugCommand.enabled": true,
+    "github.copilot.chat.reviewSelection.enabled": true
 }