You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Swarm Learning is a decentralized, privacy-preserving Machine Learning framework. This framework utilizes the computing power at, or near, the distributed data sources to run the Machine Learning algorithms that train the models. It uses the security of a blockchain platform to share learnings with peers in a safe and secure manner. In Swarm Learning, training of the model occurs at the edge, where data is most recent, and where prompt, data-driven decisions are mostly necessary. In this completely decentralized architecture, only the insights learned are shared with the collaborating ML peers, not the raw data. This tremendously enhances data security and privacy.
|`SL_WAIT_FOR_FULL_QUORUM_SECONDS`|Sets maximum time to wait for full quorum before an SL node, designated as leader node, decides to use minPeers nodes.|
13
+
|`SL_WAIT_FOR_FULL_QUORUM_SECONDS`|Sets the maximum time for an SL leader node to wait for full quorum after minPeers are ready for merge. This parameter lets you to maximize the number of peers participating in the merge process.<br>Default value: 30 secs|
14
+
|`SWCI_TASK_MAX_WAIT_TIME`|Specifies a maximum timeout value for the completion of a task.<br>This value must be set in minutes, and the default is 120 mins (2 hours).
14
15
|`SWCI_MODE`| Enables SWCIs web interface instead of command line interface. Allowed values are CLI and WEB.<br> Default value: CLI<br> |
15
16
|`SWCI_STARTUP_SCRIPT`|This is a default start script of SWCI.|
16
17
|`SWCI_WEB_PORT`|Default port on which SWCI-WEB starts server.|
17
18
|`SWOP_PROFILE`|Indicates default profile for SWOP.|
19
+
|`SWOP_KEEP_CONTAINERS`|By default, SL and ML containers spawned by SWOP are removed. This option can be enabled to retain the stopped containers for debugging.<br>Default value: False|
18
20
|`SWARM_ID_CACERT`|Indicates user CA certificates file.|
19
21
|`SWARM_ID_CAPATH`|Indicates user CA certificates directory.|
20
22
|`SWARM_ID_CERT`|Indicates user certificates file.|
1. After [completing installation of license server and downloading of the Swarm Learning installation files](/docs/Install/Install_the_License_Server.md), you must run one of the Swarm Learning installer based on your platform (Linux, Windows, or Mac as listed below).
3
+
1. After [completing installation of license server and downloading of the Swarm Learning installation files](/docs/Install/Install_the_License_Server.md), you must
4
+
run one of the Swarm Learning installer based on your platform (Linux, Windows, or Mac as listed below).
The Swarm Learning Web App is launched in a web browser.
10
-
11
-
<blockquote>
12
-
IMPORTANT: HPE recommends you to run the downloaded Swarm Learning installer from the terminal window only.
13
-
</blockquote>
14
-
15
-
The installer has a few configurable options. To change the default options, run the installer from a command prompt. Use the following optional flags to customize the configuration or behavior of the installer:
16
-
17
-
-port
18
-
: Defines the port for the application to run. The default value is 30302.
19
9
20
-
Example, `-port 30355`
10
+
2.The Swarm Learning Web App is launched in a web browser.
21
11
22
-
-edition
23
-
: Configure the Swarm Learning edition that must be installed. The following are the available options:
12
+
<blockquote>
13
+
IMPORTANT: HPE recommends you to run the downloaded Swarm Learning installer from the terminal window only.
14
+
</blockquote>
24
15
25
-
eval
26
-
: This option installs the community edition \(free edition\) of the Swarm Learning.
16
+
The installer has a few configurable options. To change the default options, run the installer from a command prompt. Use the following optional flags to
17
+
customize the configuration or behavior of the installer:
27
18
28
-
Example, `-edition eval`
19
+
**-port**
20
+
: Defines the port for the application to run. The default value is 30302.<br> Example, `-port 30355`
29
21
30
-
-logs
31
-
: If enabled, displays the detail message on the CLI during the installation. To enable, use the command, `-logs verbose`.
22
+
**-logs**
23
+
: If enabled, displays the detail message on the CLI during the installation.<br> To enable, use the command, `-logs verbose`.
32
24
33
-
-version
34
-
: This option defines the version of docker images that must be installed. The default value is 1.0.0. Example, `-version 0.3.0`
25
+
**-version**
26
+
: This option defines the version of docker images that must be installed. The default value is 1.1.0.<br> Example, `-version 0.3.0`
35
27
36
-
-timeoutDuration
37
-
: Defines installer timeout duration for individual installation tasks. The default value is 300 seconds.
38
-
39
-
Example, `-timeoutDuration 600`
28
+
**-timeoutDuration**
29
+
: Defines installer timeout duration for individual installation tasks. The default value is 300 seconds.<br> Example, `-timeoutDuration 600`
5. If there is an error message, click **Click here for more info**. Close the error message dialog, **Retry** or **Configure** the host, and click **Next**.
60
-
6. A success message is displayed for all installed hosts. Click **Next**.
61
-
62
-
<blockquote>
51
+
6. A success message is displayed for all installed hosts. Click **Next**.
52
+
53
+
<blockquote>
63
54
NOTE: Unless you configure all the hosts successfully, you cannot go to the next screen.
64
-
</blockquote>
55
+
</blockquote>
65
56
66
57
7. Review **Next Steps** and click **Next**.
67
58
8. Review the **Summary** screen, which displays all the installed hosts. Click **Finish**.
Copy file name to clipboardExpand all lines: docs/Install/Running_Swarm_Learning.md
+4-2Lines changed: 4 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -65,8 +65,10 @@ NOTE: These options do not apply to the `swarm-learning/bin/stop-swarm` script.
65
65
|`--keep`|Same as `--no-rm`. Request Docker to preserve the container after it exits.||
66
66
|`--no-keep`|Same as `--rm`. Request Docker to automatically remove the container when it exits.||
67
67
|`-h, --help`|This \(helpful\) message.||
68
-
|`--apls-ip <IP address or DNS name>`|The IP address on which APLS is serving license requests.|172.1.1.1|
69
-
|`--apls-port <port number>`|The port number on which APLS is serving license requests.|5814|
68
+
|`--primary-apls-ip <IP address or DNS name>`|The IP address on which the primary Autopass License Server is serving license requests.|None|
69
+
|`--secondary-apls-ip <IP address or DNS name>`|The IP address on which the secondary Autopass License Server is serving license requests.|None|
70
+
|`--primary-apls-port <port numberw>`|The port number on which the primary Autopass License Server is serving license requests.|5814|
71
+
|`--secondary-apls-port <port number>`|The port number on which the secondary Autopass License Server is serving license requests.|The value assigned to --primary-apls-port|
70
72
|`--apls-pdf <path to license PD file>`|The path to the license PD file to be used.|None|
71
73
|`--cacert <path to certificates file>`|The path to the file containing the list of CA certificates.|None|
72
74
|`--capath <path to certificates directory>`|The path to the directory containing CA certificate files.|None|
Copy file name to clipboardExpand all lines: docs/Install/Starting_SWCI_nodes.md
+11-1Lines changed: 11 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,4 +23,14 @@ The run-swci script accepts the following parameters:
23
23
|`--usr-dir <dir>`|The host directory that must be used as the user directory by this SWCI node.|None|
24
24
|`--init-script-name <swci-init file>`*|Name of the init script file that has SWCI commands to be executed at the start of SWCI. <br> This file must be located inside the user directory at the top level.<br>|`swci-init`|
25
25
26
-
*If init option is provided, all SWCI commands within this script file are processed before it enters the interactive mode and waits for users commands. Users can simulate a non-interactive SWCI run by having a bunch of SWCI commands and an SWCI `EXIT` command at the end of the `swci-init` file. This could be used for automation.
26
+
*If init script option is provided, all SWCI commands within this script file are processed before it enters the interactive mode and waits for users commands. Users can simulate a non-interactive SWCI run by having a bunch of SWCI commands and an SWCI `EXIT` command at the end of the `swci-init` file. This could be used for automation.
27
+
28
+
<blockquote>
29
+
30
+
NOTE:
31
+
- If you need to use the swci-init script file as-is (default), --usr-dir option must be specified and SWCI looks for this default script file under this user directory.
32
+
- If you want to run a script file with a different filename, you must explicitly specify the --init-script-name
33
+
and --usr-dir.
34
+
- If the --usr-dir is not specified, the SWCI runs in an interactive mode.
# <aname="GUID-2E350669-7E5A-47BC-AB15-58AC4CFAD9C1"/> Upgrading from earlier evaluation version
2
+
The current HPE Swarm Learning release is not compatible with the Eval or community 0.3.0 release. **You must** uninstall the 0.3.0 release before installing the current HPE Swarm Learning version. See [Uninstalling Swarm Learning](Uninstalling_the_Swarm_Learning_package.md).
2
3
3
-
The current Swarm Learning release is not compatible with the earlier Eval.0.3.0 release. **You must** uninstall the earlier version before installing the current HPE Swarm Learning version.
4
+
To upgrade to the latest version, see [Install HPE Swarm Learning](Install_HPE_Swarm_Learning.md).
5
+
<br>The installer deletes existing Swarm Learning files and directories (`docs`, `examples`, `lib`, and `scripts`) from the location and copies updated Swarm Learning files and directories.
6
+
<br>Any directory or file created by users outside of the Swarm Learning directories is preserved during the upgrade process.
Copy file name to clipboardExpand all lines: docs/User/Frequently_asked_questions.md
+13Lines changed: 13 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -170,3 +170,16 @@ Yes. New nodes can be added in the network at any point in the training. Just li
170
170
171
171
Swarm Learning uses averaging as the merge algorithm. Currently, users cannot specify the merge algorithm. This will be supported in a later release.
172
172
173
+
## Before enabling Swarm Learning, how to confirm the standalone user application has no issues and runs?
174
+
175
+
Run the user container with `SWARM_LOOPBACK` set to `TRUE`, this bypasses Swarm Learning to help you quickly develop, integrate, and test your model code with Swarm Learning package. If your code runs to completion and saves the local model it would indicate that the ML application may not have any issues.
176
+
177
+
If `SWARM_LOOPBACK` is set to TRUE, all Swarm functionality is bypassed, except parameter validation.
178
+
179
+
This can help you to verify and test integration of the model code with Swarm without spawning any Swarm Learning
180
+
containers.
181
+
182
+
## How to run user container as non-root?
183
+
184
+
By default, when user ML container is run through SWOP or using the `run-sl` script, the user ML container is run with current user's UID and GID of the host machine. If the current user on the host is non-root, the user container also runs as non-root.
|`syncFrequency`|Specifies the number of batches of local training to be performed between two swarm sync rounds. If adaptive sync enabled, this is the frequency to be used at the start.|
40
40
|`minPeers`|Specifies the minimum number of SL peers required during each synchronization round for Swarm Learning to proceed further.|
41
-
|`useAdaptiveSync`|Modulate the next syncFrequency value post each synchronization round based on performance on validation data.|
41
+
|`useAdaptiveSync`|Modulate the next syncFrequency value post each synchronization round based on performance on validation data. The default value is false.<br>**Note**: As of now, this option is implemented only for KERAS platform.|
42
42
|`adsValData`|Specifies the dataset for generating metrics for adaptive sync logic. It can be either an \(x\_val, y\_val\) tuple or a generator.|
43
43
|`adsValBatchSize`|Specifies the batch size for `adsValData`. This is used when `useAdaptiveSync` is turned ON.|
44
44
|`checkinModelOnTrainEnd`|Specifies the merge behavior of a SL node after it has achieved stopping criterion and it is waiting for all other peers to complete their training. During this period this SL node does not train the model with local data. This parameter decides the nature of the weights that this SL node contributes to the merge process.Allowed values are:<br>`inactive`: Node does not contribute its weights in the merge process but participates as non-contributing peer in the merge process.<br>`snapshot`: Node always contributes the weights that it had when it reached the stopping criterion, it does not accept merged weights.<br>`active`: Node behaves as if it is in active training, but it does not train merged model with local data as mentioned above.<br>`snapshot` is the default value.<br>|
45
45
|`trainingContract`|Training contract associated with this learning. It is a user-defined string. Default value is `defaultbb.cqdb.sml.hpe`. <br> **NOTE**: This parameter enables a user to run <strong>concurrent</strong> swarm learning trainings, within the same swarm network. User must create this training contract using SWCI and then use it as the parameter value.|
46
46
|`nodeWeightage`|A number between 1–100 to indicate the relative importance of this node compared with others during the parameter merge process.By default, all nodes are equal and have the same weight-age of one.|
47
-
|`mlPlatform`|Specifies ML platform. Allowed values are either TF, KERAS or PYTORCH.|
47
+
|`mlPlatform`|Specifies ML platform. Allowed values are either TF, KERAS or PYTORCH. If TF platform is used, the default value is KERAS. If PYTORCH platform is used, the default value is PYTORCH.|
48
48
|`logger`|Provides information about Python logger. `SwarmCallback` class invokes info, debug, and error methods of this logger for logging. If no logger is passed, then `SwarmCallback` class creates its own logger from basic python logger. If required, user can get hold of this logger instance to change the log level as follows: <br> `import logging` <br> `from swarmlearning.tf import SwarmCallback` <br> `swCallback = SwarmCallback(syncFrequency=128, minPeers=3)` <br> `swCallback.logger.setLevel(logging.DEBUG)`|
49
49
50
50
3. Use the `SwarmCallback` object for training the model.
Copy file name to clipboardExpand all lines: docs/User/SWCI_APIs.md
+36Lines changed: 36 additions & 0 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -43,6 +43,42 @@ The python3 program needs to import the SWCI class, and then use the below APIs.
43
43
|`registerTask()`|This method registers a task into the SN network and finalizes it, if the task is valid.|`yamlFileName, finalize=True`|
44
44
|`resetTaskRunner()`|This method resets the state of the taskrunner contract to an uninitialized state.<br><strong>WARNING:</strong>This action cannot be undone, reset only completed Taskrunner contracts. Resetting the active taskrunner contract can result in unexpected behavior.|`trName='defaulttaskbb.taskdb.sml.hpe'`|
45
45
|`resetTrainingContract()`|This method resets the state of the training contract to an uninitialized state.<br><strong>WARNING:</strong>This action cannot be undone, reset only completed Swarm Learning contracts. Resetting the active contracts can result in unexpected behavior.|`ctName='defaultbb.cqdb.sml.hpe'`|
46
+
|`sleep()`|This method sleeps for a specified time before executing the subsequent commands.<br> <br>For example, in between a `WAIT FOR TASKRUNNER` and `RESET TASKRUNNER`, one can use a `SLEEP 10`, to give a grace time of 10 secs, before the `RESET` command cleans up the SL and user container.<br> <br>This would be required to allow the user ML code to save the model or do any inference of the model, after the Swarm training is over.<br> <br>For more information, see the example SWCI scripts in the `swarmlearning/examples/` directory.|`time in seconds`|
46
47
|`setLogLevel()`|This method sets the logging level for the SWCI container.|logLv One of `{ logging.CRITICAL, logging.ERROR, logging.WARNING, logging.INFO, logging.DEBUG }`<br>|
47
48
|`uploadTaskDefintion()`|This method uploads the local task definition file to the SWCI container.|`taskFilePath`|
0 commit comments