Update configure.rst with custom components and named resource configuration (#55)

Summary:
Pull Request resolved: #55

Update configure.rst with custom components and named resource configuration

Reviewed By: kiukchung

Differential Revision: D29018082

fbshipit-source-id: 53f6556d185b43deff4b4716ff3e0ee8c3483054

Author: aivanou

docs/source/configure.rst

@@ -65,7 +65,25 @@ enumerate t-shirt sized resource specs for the containers as:
 
 And refer to the resources by their string names.
 
-<COMING SOON>
+Torchx supports custom resource definition and reference by string
+representation:
+
+::
+
+ [torchx.schedulers]
+ gpu_x_2 = my_module.resources:gpu_x_2
+
+
+The named resource after that can be used in the following manner:
+
+.. code-block:: python
+
+  # my_module.component
+  from torchx.specs import AppDef, Role
+  from torchx.components.base import torch_dist_role
+
+  app = AppDef(name="test_app", roles=[torch_dist_role(.., resource="gpu_x_2", ...)])
+
 
 Registering Custom Components
 -------------------------------
@@ -79,4 +97,16 @@ when they run
 
  $ torchx builtins
 
-<COMING SOON>
+Custom components can be registered via the following endpoint:
+
+::
+
+ [torchx.components]
+ custom_component = my_module.components:my_component
+
+
+Custom components can be executed in the following manner:
+
+.. code-block:: shell-session
+
+ $ torchx run --scheduler local --scheduler_args image_fetcher=...,root_dir=/tmp custom_component -- --name "test app"

torchx/components/__init__.py

@@ -43,6 +43,20 @@
 Create component as function. Each component represents a function that accepts
  arbitrary arguments and returns ``specs.AppDef``.
 
+The function should
+have the following properties:
+
+* All arguments of the function must be annotated
+* Current supported types:
+    + Primitives: int, float, str
+    + Optional primitives: Optional[int], Optional[float], Optional[str]
+    + Dict: Dict[Primitive_key, Primitive_value]
+    + List: List[Primitive_value]
+    + Optional[List], Optional[Dict]
+* The function should have well defined description in
+    https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html format
+
+
 Unit tests. Write unit tests that use ``torchx.specs.file_linter`` to validate
 the component's structure,  similar to ``torchx.components.tests.distributed_test.py``.
 

torchx/specs/__init__.py

@@ -12,6 +12,34 @@
 def named_resource(resource: str) -> Resource:
     """
     Gets resource object based on the string definition registered via entrypoints.txt.
+
+    Torchx implements named resource registration mechanism, which consists of
+    the following steps:
+
+    1. Create a module and define your resource retrieval function:
+
+    .. code-block:: python
+
+     # my_module.resources
+     from typing import Dict
+     from torchx.specs import Resource
+
+     def gpu_x_1() -> Dict[str, Resource]:
+         return Resource(cpu=2, memMB=64 * 1024, gpu = 2)
+
+    2. Register resource retrieval in the entrypoints section:
+
+    ::
+
+     [torchx.schedulers]
+     gpu_x_1 = my_module.resources:gpu_x_1
+
+    The ``gpu_x_1`` can be used as string argument to this function:
+
+    ::
+
+     resource = named_resource("gpu_x_1")
+
     """
     try:
         resource_fn = load("torchx.resources", name=resource.lower())