You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[](https://github.com/FAIRDataPipeline/FAIR-CLI/actions/workflows/fair-cli.yaml)
| The following document is largely conceptual and therefore does *not* represent a manual for the final interface. Statements within the following are likely to change, further details of possible changes are given throughout. Please either open an issue or pull request on the [source repository](https://github.com/FAIRDataPipeline/FAIR-CLI) raising any changes/issues. |
10
-
11
-
FAIR-CLI forms the main interface for synchronising changes between your local and shared remote FAIR Data Pipeline registries, it is also used to instantiate model runs/data submissions to the pipeline.
12
-
13
-
The project is still under development with many features still to be implemented and checked. Available commands are summarised below along with their usage.
3
+
FAIR-CLI forms the main interface for synchronising changes between local and shared remote FAIR Data Pipeline registries, it is also used to instantiate model runs/data submissions to the pipeline. Full documentation of the FAIR Data Pipeline can be found on the project [website](https://www.fairdatapipeline.org/).
14
4
15
5
## Installation
16
6
17
-
The project makes use of [Poetry](https://python-poetry.org/) for development which allows quick and easy mangement of dependencies, and provides a virtual environment exclusive to the project. Ultimately the project will be built into a pip installable module (using `poetry build`) meaning users will not need Poetry. You can access this environment by installing poetry:
18
-
19
-
```sh
20
-
pip install poetry
21
-
```
22
-
23
-
and, ensuring you are in the project repository, running:
24
-
25
-
```sh
26
-
poetry install
27
-
```
28
-
29
-
which will setup the virtual environment and install requirements. You can then either launch the environment as a shell using:
30
-
31
-
```sh
32
-
poetry shell
33
-
```
34
-
35
-
or run commands within it externally using:
7
+
The package is installed using Pip:
36
8
37
9
```sh
38
-
poetry run <command>
10
+
pip install fair-cli
39
11
```
40
12
41
-
## Structure
42
-
43
-
The layout of FAIR-CLI on a simplified system looks like this:
44
-
45
-
```sh
46
-
$HOME
47
-
├── .fair
48
-
│ ├── cli
49
-
│ │ ├── cli-config.yaml
50
-
│ │ └── sessions
51
-
│ ├── data
52
-
│ │ └── jobs
53
-
│ └── $REGISTRY_HOME
54
-
│
55
-
└─ Documents
56
-
└─ my_project
57
-
├── config.yaml
58
-
└── .fair
59
-
├── cli-config.yaml
60
-
├── logs
61
-
└── staging
62
-
```
63
-
64
-
### Global and Local Directories
65
-
66
-
FAIR-CLI stores information for projects in two locations. The first is a *global* directory stored in the user's home folder in the same location as the registry itself `$HOME/.fair/cli`, and the second is a *local* directory which exists within the model project itself `$PROJECT_HOME/.fair`.
67
-
68
-
The CLI holds metadata for the user in it's own configuration file (not to be confused with the user modifiable `config.yaml`), `cli-config.yaml`, the *global* version of which is initialised during first use. In a manner similar to `git`, FAIR-CLI has repositories which allow the user to override these *global* configurations, this then forming a *local* variant.
69
-
70
-
### Data Directory
71
-
72
-
The directory `$HOME/.fair/data` is the default data store initialised by FAIR-CLI. During setup an alternative can be provided and this can be later changed on a per-run basis if the user so desires. The subdirectory `$HOME/data/jobs` contains timestamped directories of jobs.
73
-
74
-
### Sessions Directory
75
-
76
-
The directory `$HOME/.fair/sessions` is used to keep track of ongoing queries to the registry as a safety mechanism to ensure the registry is not shutdown whilst processes are still occuring.
77
-
78
-
### Logs Directory
79
-
80
-
The directory `$PROJECT/.fair/logs` stores `stdout` logs for jobs also giving information on who launched the job and how long it lasted.
81
-
82
-
### Staging File
83
-
84
-
The staging file, `$PROJECT/.fair/staging`, contains information of what jobs are being tracked, by default all jobs are added to this file after completion and are set to "unstaged". Simply contains a dictionary of booleans where items for sync (staged) are marked true `True` and those to be held only locally `False`. The file uses paths relative to the *local*`.fair` folder as keys, to behave in a manner identical to `git` staging.
85
-
86
-
### `config.yaml`
13
+
## The User Configuration File
14
+
Job runs are configured via `config.yaml` files. Upon initialisation of a project, FAIR-CLI automatically generates a starter configuration file with all requirements in place. To execute a process (e.g. perform a model run from a compiled binary/script) an additional key of either `script` or `script_path` must be provided. Alternatively the command `fair run bash` can be used to append the key and run a command directly.
87
15
88
-
This is the main file the user will interact with to customise their run. FAIR-CLI automatically generates a starter version of this file with everything in place. The only addition required is setting of either `script` or `script_path` (with the exception of running using `fair run bash` - see [below](#run)) under `run_metadata`.
| This should be a command callable by a shell for running a model/submitting data to the registry. This script is saved to a file prior to execution. |
93
-
||
94
-
|**`script_path`**|
95
-
| This is a direct path to an existing script to use for submission. |
96
-
97
-
By default the shell used will be `sh` or `pwsh` for UNIX and Windows systems respectively, however this can be overwritten with the optional `shell` key which recognises the following values (where `{0}` is the script file):
16
+
By default the shell used to execute a process is `sh` or `pwsh` for UNIX and Windows systems respectively. This can be overwritten by assigning the optional `shell` key with one of the following values (where `{0}` is the script file):
| This layout is subject to possible change depending on whether or not multiple aliases for the same user will be allowed in the registry itself. The main reason for having a *local* version is to support separate handling of multiple projects. |
115
-
116
-
## Registry Interaction
31
+
A full description of `config.yaml` files can be found [here](https://www.fairdatapipeline.org/docs/interface/config/).
117
32
118
-
Currently `FAIR-CLI` sets up the write data storage location on the local registry if it does not exist. Entries are created for the YAML file type, current user as an author, and object for a given run.
119
33
120
-
## Command Line Usage
121
-
122
-
As mentioned, all of the subcommands within FAIR-CLI are still under review with many still serving as placeholders for future features. Running `fair` without arguments or `fair --help` will show all of these.
34
+
## Available Commands
123
35
124
36
### `init`
125
37
126
-
Initialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, although setup will ask if you want to use an alternative location. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml`.
127
-
128
-
The first time this command is launched the *global* CLI configuration will be populated. In subsequent calls the *global* will provide default suggestions towards creating the CLI configuration for the repository (*local*).
38
+
Initialises a new FAIR repository within the given directory. This should ideally be the same location as the `.git` folder for the current project, however during setup an option is given to specify an alternative. The command will ask the user a series of questions which will provide metadata for tracking run authors, and also allow for the creation of a starter `config.yaml` file. Initialisation will also configure the CLI itself.
129
39
130
-
A repository directory matching the structure above will be placed in the current location and a starter `config.yaml` file will be generated (see below).
131
-
132
-
#### Example: First call to `fair init`
133
-
134
-
This example shows the process of setting up for the first time. Note the default suggestions for each prompt, in the case of `Full name` and `Default output namespace` this is the hostname of the system and an abbreviated version of this name.
135
-
136
-
```sh
137
-
$ fair init
138
-
Initialising FAIR repository, setup will now ask for basic info:
139
-
140
-
Checking forlocal registry
141
-
Local registry found
142
-
Remote Data Storage Root [http://data.scrc.uk/data/]:
143
-
Remote API Token File: $HOME/scrc_token.txt
144
-
Local API URL [http://localhost:8000/api/]:
145
-
Local registry is offline, would you like to start it? [y/N]: y
146
-
Default Data Store: [/home/joebloggs/.fair/data]:
147
-
Email: jbloggs@noreply.uk
148
-
ORCID [None]:
149
-
Full Name: Joe Bloggs
150
-
Default input namespace [None]: SCRC
151
-
Default output namespace [jbloggs]:
152
-
Project description: Test project
153
-
Local Git repository [/home/joebloggs/Documents/AnalysisProject]:
154
-
Git remote name [origin]:
155
-
Using git repository remote 'origin': git@notagit.com:jbloggs/AnalysisProject.git
156
-
Initialised empty fair repository in /home/joebloggs/Documents/AnalysisProject/.fair
40
+
#### Custom CLI Configuration
41
+
After setup is complete, the current CLI configuration can also be saved using the command:
157
42
```
158
-
159
-
#### Example: Subsequent runs
160
-
161
-
In subsequent runs the first time setup will provide further defaults.
162
-
163
-
```sh
164
-
$ fair init
165
-
Initialising FAIR repository, setup will now ask for basic info:
166
-
167
-
Project description: Test Project
168
-
Local Git repository [/home/joebloggs/Documents/AnalysisProject]:
169
-
Git remote name [origin]:
170
-
Using git repository remote 'origin': git@nogit.com:joebloggs/AnalysisProject.git
171
-
Remote API URL [http://data.scrc.uk/api/]:
172
-
Remote API Token File [/home/kristian/scrc_token.txt]:
173
-
Local API URL [http://localhost:8000/api/]:
174
-
Default output namespace [jbloggs]:
175
-
Default input namespace [SCRC]:
176
-
Initialised empty fair repository in /home/joebloggs/Documents/AnalysisProject/.fair
the user then only needs to add a `script` or `script_path` entry to execute a code run. This is only required for `run`.
192
-
193
-
#### Advanced usage
194
-
195
-
CLI configuration can be read directly from a file which should contain the following:
45
+
the created file can then be re-read at a later point during setup. Alternatively, if creating a configuration from scratch the YAML file should contain the following information:
196
46
197
47
```yaml
198
48
namespaces:
@@ -218,24 +68,24 @@ git:
218
68
remote: origin
219
69
description: Testing Project
220
70
```
221
-
222
-
this file is then read during initialisation:
71
+
this file is then read during the initialisation:
223
72
224
73
```sh
225
74
fair init --using <cli-config.yaml file>
226
75
```
227
76
228
-
For the purposes of CI runs, the initialisation can be "skipped" by running:
77
+
For integration into a CI workflow, the setup can be skipped by running:
229
78
230
79
```sh
231
80
fair init --ci
232
81
```
233
82
234
-
which will create temporary directories for some locations.
83
+
which will create temporary directories for some of the required location paths.
84
+
235
85
236
86
### `run`
237
87
238
-
The purpose of `run` is to execute a model/submission run to the local registry. The command fills any specified template variables of the form `${{ VAR }}` to match those outlined [below](#template-variables). Outputs of a run will be stored within the `coderun` folder in the directory specified under the `data_store` tag in the `config.yaml`, by default this is `$HOME/.fair/data/coderun`.
88
+
The purpose of `run` is to execute a model/submission run and submit results to the local registry. Outputs of a run will be stored within the `coderun` folder in the directory specified under the `data_store` tag in the `config.yaml`, by default this is `$HOME/.fair/data/coderun`.
239
89
240
90
```sh
241
91
fair run
@@ -247,7 +97,7 @@ If you wish to use an alternative `config.yaml` then specify it as an additional
247
97
fair run /path/to/config.yaml
248
98
```
249
99
250
-
You can also launch a bash command directly which will then be automatically written into the `config.yaml` for you:
100
+
You can also launch a bash command directly, this will be automatically written into the `config.yaml`:
251
101
252
102
```sh
253
103
fair run --script "echo \"Hello World\""
@@ -257,84 +107,23 @@ note the command itself must be quoted as it is a single argument.
257
107
258
108
### `pull`
259
109
260
-
Currently `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. For example:
and register all required objects into the local registry.
110
+
Currently `pull` will update any entries within the `config.yaml` under the `register` heading creating `external_object` and `data_product` objects on the registry and downloading the data to the local data storage. Any data required for a run is downloaded and stored within the local registry.
322
111
323
112
### `purge`
324
113
325
-
Removes the local `.fair` (FAIR repository) folder by default so the user can reinitialise:
114
+
The `purge` command removes setup of the current project so it can bereinitialised:
326
115
327
116
```sh
328
117
fair purge
329
118
```
330
119
331
-
You can remove the global configuration and start again entirely by running:
120
+
To remove all configurations entirely (including those global to all projects) run:
332
121
333
122
```sh
334
123
fair purge --global
335
124
```
336
125
337
-
and also the data directory by running:
126
+
Finally the data directory itself can be removed by running:
338
127
339
128
```sh
340
129
fair purge --data
@@ -386,7 +175,7 @@ Date: Wed Jun 30 09:09:30 2021
0 commit comments