(torchx/workspace) Support multi-project/directory workspace #1114

kiukchung · 2025-09-12T02:40:40Z

Summary:
Usage:

In your .torchxconfig file specify multi-project workspace as follows:

[cli:run]
workspace = 
  /home/$USER/github/myproj: 
  /home/$USER/github/verl: verl
  /home/$USER/.torchxconfig: verl/.torchxconfig

Then the workspace is built as:

/home/$USER/github/myproj/** -> $REMOTE_ROOT/**
/home/$USER/github/verl/** -> $REMOTE_ROOT/verl/**
/home/$USER/.torchxconfig -> $REMOTE_ROOT/verl/.torchxconfig

Notes:

The mappings are basically like BUCK python_library's resoruce attribute. Mapping to an empty string, or simply leaving the target blank (as we did for myproj in the example above) copies the contents of the src dir directly into the remote workspace root
Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example:

[cli:run]
workspace = 
  /home/$USER/github/verl: verl
  /home/$USER/.torchxconfig: verl

copies .torchxconfig into $REMOTE_ROOT/verl/.torchxconfig. However if we switching the order of the projects won't work:

[cli:run]
workspace = 
  /home/$USER/.torchxconfig: verl
  /home/$USER/github/verl: verl

This would copy .torchxconfig to the file called verl in $REMOTE_ROOT/verl, then github/verl would fail to copy.

Differential Revision: D82169554

facebook-github-bot · 2025-09-12T02:40:50Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

facebook-github-bot · 2025-09-12T02:51:09Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-12T20:34:35Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-12T20:46:40Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-12T20:47:50Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-12T20:51:11Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-12T20:52:40Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-12T23:54:21Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-15T17:42:03Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-15T22:58:32Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-16T16:22:05Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-16T16:24:48Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-16T19:19:07Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-16T19:37:32Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-16T19:39:11Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

Summary: Usage: In your `.torchxconfig` file specify multi-project workspace as follows (notice the indent after `workspace =`, this uses INI file's multiline strings and reads the multiline contents as YAML - at some point we should make .torchxconfig a toml or YAML but that's a BC breaking change): ``` [cli:run] workspace = /home/$USER/github/myproj: /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl/.torchxconfig ``` Then the workspace is built as: 1. `/home/$USER/github/myproj/**` -> `$REMOTE_ROOT/**` 1. `/home/$USER/github/verl/**` -> `$REMOTE_ROOT/verl/**` 1. `/home/$USER/.torchxconfig` -> `$REMOTE_ROOT/verl/.torchxconfig` Notes: * The mappings are basically like BUCK `python_library`'s `resoruce` attribute. Mapping to an empty string, or simply leaving the target blank (as we did for `myproj` in the example above) copies the contents of the src dir directly into the remote workspace root * To copy the project dir contents directly into the remote root, map it to `""` (empty string) or leave the target blank ``` [cli:run] workspace = /home/$USER/github/myproj: # same as /home/$USER/github/myproj: "" # same as /home/$USER/github/myproj: '' ``` * Files can be specified as src. The target of a file can be a directory, and if so, the file will be copied into the directory. For example: ``` [cli:run] workspace = /home/$USER/github/verl: verl /home/$USER/.torchxconfig: verl ``` copies `.torchxconfig` into `$REMOTE_ROOT/verl/.torchxconfig`. * However switching the order of the projects won't work: ``` [cli:run] workspace = # copies .torchxconfig -> verl (verl is a file) /home/$USER/.torchxconfig: verl # ERROR trying to copy github/verl/** into verl/** (since verl is a file) /home/$USER/github/verl: verl ``` Reviewed By: highker Differential Revision: D82169554

facebook-github-bot · 2025-09-16T20:10:19Z

@kiukchung has exported this pull request. If you are a Meta employee, you can view the originating diff in D82169554.

meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Sep 12, 2025

facebook-github-bot added fb-exported meta-exported labels Sep 12, 2025

facebook-github-bot force-pushed the export-D82169554 branch from 2683a61 to 1a366c7 Compare September 12, 2025 02:51

facebook-github-bot force-pushed the export-D82169554 branch from 1a366c7 to 0846364 Compare September 12, 2025 20:34

facebook-github-bot force-pushed the export-D82169554 branch from 0846364 to 096868a Compare September 12, 2025 20:46

kiukchung force-pushed the export-D82169554 branch from 096868a to a7c3daf Compare September 12, 2025 20:47

facebook-github-bot force-pushed the export-D82169554 branch from a7c3daf to 4e97e31 Compare September 12, 2025 20:51

facebook-github-bot force-pushed the export-D82169554 branch from 4e97e31 to 4581fed Compare September 12, 2025 20:52

highker approved these changes Sep 12, 2025

View reviewed changes

facebook-github-bot force-pushed the export-D82169554 branch from 4581fed to 9c3fb4c Compare September 12, 2025 23:54

facebook-github-bot force-pushed the export-D82169554 branch from 9c3fb4c to 9593260 Compare September 15, 2025 17:41

kiukchung force-pushed the export-D82169554 branch from 9593260 to 33ea153 Compare September 15, 2025 22:58

kiukchung force-pushed the export-D82169554 branch from 33ea153 to ec62c8e Compare September 16, 2025 16:21

facebook-github-bot force-pushed the export-D82169554 branch from ec62c8e to 18cb679 Compare September 16, 2025 16:24

kiukchung force-pushed the export-D82169554 branch from 18cb679 to 14e1e0d Compare September 16, 2025 19:18

facebook-github-bot force-pushed the export-D82169554 branch from 14e1e0d to 8fb543a Compare September 16, 2025 19:37

facebook-github-bot force-pushed the export-D82169554 branch from 8fb543a to 663483c Compare September 16, 2025 19:39

facebook-github-bot force-pushed the export-D82169554 branch from 663483c to e8181b6 Compare September 16, 2025 20:10

facebook-github-bot merged commit f685fbe into main Sep 17, 2025
20 of 24 checks passed

facebook-github-bot deleted the export-D82169554 branch September 17, 2025 01:09

(torchx/workspace) Support multi-project/directory workspace #1114

(torchx/workspace) Support multi-project/directory workspace #1114

Uh oh!

Conversation

kiukchung commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 12, 2025

Uh oh!

facebook-github-bot commented Sep 15, 2025

Uh oh!

facebook-github-bot commented Sep 15, 2025

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

facebook-github-bot commented Sep 16, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants