From b4639cd2ee9a3406d7f522eb2c6ffe844e2c65ee Mon Sep 17 00:00:00 2001 From: jdorsch Date: Wed, 19 Mar 2025 15:36:58 +0100 Subject: [PATCH 1/7] added version 2 and refactory --- docs/services/firecrest.md | 354 +++++++++++++++++++++++++++++-------- 1 file changed, 280 insertions(+), 74 deletions(-) diff --git a/docs/services/firecrest.md b/docs/services/firecrest.md index d78b369f..8a64001a 100644 --- a/docs/services/firecrest.md +++ b/docs/services/firecrest.md @@ -1,8 +1,11 @@ [](){#ref-firecrest} # FirecREST -FirecREST is a RESTful API for managing High-Performance Computing resources, developed at CSCS. -Scientific platform developers can integrate Firecrest into web-enabled portals and applications, allowing them to securely access authenticated and authorized CSCS services such as job submission and data transfer on HPC systems. +FirecREST is a RESTful API for programatically accessing High-Performance Computing resources, developed at CSCS. + +Users can make use of FirecREST to automate access to HPC, enabling [CI/CD pipelines](https://github.com/eth-cscs/firecrest/tree/master/examples/CI-pipeline), [Worflow Managers](https://github.com/eth-cscs/firecrest/tree/master/examples/airflow-operators), and other tools agains HPC resources. + +Additionally, scientific platform developers can integrate FirecREST into [web-enabled portals](https://firecrest-ui-hpc.cscs.ch) and [applications](https://github.com/eth-cscs/firecrest/tree/master/examples/UI-client-credentials), allowing them to securely access authenticated and authorized CSCS services such as job submission and data transfer on HPC systems. Users can make HTTP requests to perform the following operations: @@ -10,28 +13,74 @@ Users can make HTTP requests to perform the following operations: * actions against the Slurm workload manager (submit, query, and cancel jobs of the user) * internal (between CSCS systems) and external (to/from CSCS systems) data transfers -For a full feature set, have a look at the latest [FirecREST API specification](https://firecrest-docs.v1.svc.cscs.ch/) deployed at CSCS. +## FirecREST versions + +Beginning 2025, CSCS has introduced a new version of the API: [FirecREST version 2](https://eth-cscs.github.io/firecrest-v2). + +Version 2 is faster, easier to use, and more efficient in resource management than its predecesor, therefore, if you are new to FirecREST start directly using **version 2** (if available for your platform). + +If you're using **version 1**, we recommend you to port your applications to use the new version. + +### Version 2 + +!!!Warning + Documentation for version 2 is still work in progress + +For a full feature set, have a look at the latest [FirecREST version 2 API specification](https://eth-cscs.github.io/firecrest-v2/openapi) deployed at CSCS. + +Please refer to the [FirecREST-v2 documentation](https://eth-cscs.github.io/firecrest-v2/use) for detailed documentation. + +### Version 1 + +For a full feature set, have a look at the latest [FirecREST version 1 API specification](https://firecrest-docs.v1.svc.cscs.ch/) deployed at CSCS. + +Please refer to the [FirecREST-v1 documentation](https://firecrest.readthedocs.io/en/latest/) for detailed documentation. -Please refer to the [FirecREST documentation](https://firecrest.readthedocs.io/en/latest/) for detailed documentation. ## FirecREST Deployment on Alps -FirecREST is available for all three major [Alps platforms][ref-alps-platforms], with a different API endpoint for each platform. +FirecREST is available for all three major [Alps platforms][ref-alps-platforms], with a different API endpoint and versions for each platform. + + + + + + + + +
PlatformVersionAPI EndpointClusters
HPC Platformv1https://api.cscs.ch/hpc/firecrest/v1Daint, Eiger
v2https://api.cscs.ch/hpc/firecrest/v2
ML Platformv1https://api.cscs.ch/ml/firecrest/v1Bristen, Clariden
CW Platformv1https://api.cscs.ch/cw/firecrest/v1Santis
v2https://api.cscs.ch/cw/firecrest/v2
+ -| Platform | API Endpoint | Clusters | -|----------|-------------|-----------| -| HPC Platform | https://api.cscs.ch/hpc/firecrest/v1 | [Daint][ref-cluster-daint], [Eiger][ref-cluster-eiger] | -| ML Platform | https://api.cscs.ch/ml/firecrest/v1 | [Bristen][ref-cluster-bristen], [Clariden][ref-cluster-clariden] | -| CW Platform | https://api.cscs.ch/cw/firecrest/v1 | [Santis][ref-cluster-santis] | +## Accessing FirecREST -## Developer Portal: Creating FirecREST clients +### Clients and Access tokens -The [Developer Portal](https://developer.cscs.ch) is an application based on [WSO2 API Manager](https://wso2.com/api-manager/) that facilitates CSCS Users to manage subscriptions to an API at CSCS (such as FirecREST). +A **client application** that makes requests to FirecREST will not be using directly the credentials of the user for the authentication, but an **access token** instead. The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and it's valid for a short time (5 minutes). Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. -Start by navigating to [developer.cscs.ch](https://developer.cscs.ch), then sign in by clicking the "SIGN-IN" button on the top right hand corner of the page. +Every client will have a client ID (Consumer Key) and a secret (Consumer Secret) that will be used to get a short-lived access token with an HTTP request. + +!!! example "curl call to fetch the access token" + ``` + curl -s -X POST https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token \ + --data "grant_type=client_credentials" \ + --data "client_id=" \ + --data "client_secret=" + ``` + +You can manage your client application on the [CSCS Developer Portal][ref-devportal]. + +[](){#ref-devportal} +### CSCS Developer Portal + +The [Developer Portal](https://developer.cscs.ch) facilitates CSCS Users to manage subscriptions to an API at CSCS (such as FirecREST v1/v2). + +Start by browsing to [developer.cscs.ch](https://developer.cscs.ch), then sign in by clicking the "SIGN-IN" button on the top right hand corner of the page. Once logged in, you will see a list of APIs that are available to your user. +!!! Warning + You might not see version 1 or version 2 of some API. You will be able to see all the versions when you *subscribe* your Application to the API. + ### Creating an Application Click on the "Applications" button at the top of the screen to manage your Applications. @@ -50,30 +99,16 @@ When finished, click on the "Save" button. ### Configuring Production Keys - -Once the Application is created, create the Production Keys (Client ID and Client Secret) by clicking on "Production Keys" +Once the Application is created, create the Production Keys (`Client ID` and `Client Secret`) by clicking on "Production Keys" ![FirecREST production keys](../images/firecrest/f7t-keys.png) -There are two ways to configuring production keys: - -=== "use existing OAuth keys" - - This approach can be used if you have already generated keys for FirecREST. - - * click on the "Provide Existing OAuth Keys" button - * enter the Consumer Key (Client ID) and Consumer Secret (Client Secret) - * click the "Provide" button to confirm - ![FirecREST existing keys](../images/firecrest/f7t-existing-keys.png) +Use this if this is your first FirecREST application, or if you wish to create new keys. -=== "generate new OAuth keys" +* click on the "Generate Keys" button at the bottom of the page - Use this if this is your first FirecREST application, or if you wish to create new keys. - - * click on the "Generate Keys" button at the bottom of the page - - ![FirecREST existing keys](../images/firecrest/f7t-generate-keys.png) +![FirecREST existing keys](../images/firecrest/f7t-generate-keys.png) Once the keys are generated, you will see the pair "Consumer Key" and "Consumer Secret". @@ -107,75 +142,241 @@ To use your Application to access FirecREST, follow the [API documentation](http One way to get started is by using [pyFirecREST](https://pyfirecrest.readthedocs.io/), a Python package with a collection of wrappers for the different functionalities of FirecREST. This package simplifies the usage of FirecREST by making multiple requests in the background for more complex workflows as well as by refreshing the access token before it expires. +#### Try version 2 -!!! example "Try FirecREST using pyFirecREST" +!!! example "Try FirecREST using pyFirecREST v2" ```python - import firecrest as fc + import json + import firecrest as f7t + + client_id = "" + client_secret = "" + token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token" + + # Setup the client for the specific account + # For instance, for the Alps HPC Platform system Daint: + client = f7t.v2.Firecrest( + firecrest_url="https://api.cscs.ch/hpc/firecrest/v2", + authorization=f7t.ClientCredentialsAuth(client_id, client_secret, token_uri) + ) + + # Status of the systems, filesystems and schedulers: + print(json.dumps(client.systems(), indent=2)) + + # Output: information about systems and health status of the infrastructure + # [ + # { + # "name": "daint", + # "ssh": { # --> SSH settings + # "host": "daint.alps.cscs.ch", + # "port": 22, + # "maxClients": 100, + # "timeout": { + # "connection": 5, + # "login": 5, + # "commandExecution": 5, + # "idleTimeout": 60, + # "keepAlive": 5 + # } + # }, + # "scheduler": { # --> Scheduler settings + # "type": "slurm", + # "version": "24.05.4", + # "apiUrl": null, + # "apiVersion": null, + # "timeout": 10 + # }, + # "servicesHealth": [ # --> Health status of services + # { + # "serviceType": "scheduler", + # "lastChecked": "2025-03-18T23:34:51.167545Z", + # "latency": 0.4725925922393799, + # "healthy": true, + # "message": null, + # "nodes": { + # "available": 21, + # "total": 858 + # } + # }, + # { + # "serviceType": "ssh", + # "lastChecked": "2025-03-18T23:34:52.054056Z", + # "latency": 1.358715295791626, + # "healthy": true, + # "message": null + # }, + # { + # "serviceType": "filesystem", + # "lastChecked": "2025-03-18T23:34:51.969350Z", + # "latency": 1.2738196849822998, + # "healthy": true, + # "message": null, + # "path": "/capstor/scratch/cscs" + # }, + # (...) + # "fileSystems": [ # --> Filesystem settings + # { + # "path": "/capstor/scratch/cscs", + # "dataType": "scratch", + # "defaultWorkDir": true + # }, + # { + # "path": "/users", + # "dataType": "users", + # "defaultWorkDir": false + # }, + # { + # "path": "/capstor/store/cscs", + # "dataType": "store", + # "defaultWorkDir": false + # } + # ] + # } + # ] + + # List content of directories + print(json.dumps(client.list_files("daint", "/capstor/scratch/cscs/"), + indent=2)) + + # [ + # { + # "name": "directory", + # "type": "d", + # "linkTarget": null, + # "user": "", + # "group": "", + # "permissions": "rwxr-x---+", + # "lastModified": "2024-09-02T12:34:45", + # "size": "4096" + # }, + # { + # "name": "file.txt", + # "type": "-", + # "linkTarget": null, + # "user": "", + # "group": "", + # "permissions": "rw-r-----+", + # "lastModified": "2024-09-02T08:26:04", + # "size": "131225" + # } + # ] + ``` - client_id = - client_secret = + +#### Try version 1 + +!!! example "Try FirecREST using pyFirecREST v1" + ```python + import json + import firecrest as f7t + + client_id = "" + client_secret = "" token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token" # Setup the client for the specific account # For instance, for the Alps HPC Platform system Daint: - client = fc.Firecrest( + client = f7t.v1.Firecrest( firecrest_url="https://api.cscs.ch/hpc/firecrest/v1", - authorization=fc.ClientCredentialsAuth(client_id, client_secret, token_uri) + authorization=f7t.ClientCredentialsAuth(client_id, client_secret, token_uri) ) - print(client.all_systems()) + print(json.dumps(client.all_systems(), indent=2)) # Output: (one dictionary per system) # [{ - # 'system': 'daint' - # 'status': 'available', # 'description': 'System ready', - # }] + # 'status': 'available', + # 'system': 'daint' + # }] - print(client.list_files('daint', '/capstor/scratch/cscs/')) + print(json.dumps(client.list_files('daint', '/capstor/scratch/cscs/'), + indent=2)) # Example output: (one dictionary per file) - # [{ - # 'name': 'file.txt', - # 'user': 'username' - # 'last_modified': '2024-04-28T12:03:33', - # 'permissions': 'rw-r--r--', - # 'size': '2021', - # 'type': '-', - # 'group': 'project', - # 'link_target': '', - # }, - # { - # 'name': 'test-dir', - # 'user': 'username' + # [ + # { + # 'name': 'directory', + # 'user': '' # 'last_modified': '2024-04-20T11:22:41', # 'permissions': 'rwxr-xr-x', # 'size': '4096', # 'type': 'd', - # 'group': 'project', + # 'group': '', # 'link_target': '', - # }] + # } + # { + # 'name': 'file.txt', + # 'user': '' + # 'last_modified': '2024-09-02T08:26:04', + # 'permissions': 'rw-r--r--', + # 'size': '131225', + # 'type': '-', + # 'group': '', + # 'link_target': '', + # } + # ] ``` + The tutorial is written for a generic instance of FirecREST but if you have a valid user at CSCS you can test it directly with your resource allocation on the exposed systems. -### CSCS Developer Portal +### Data Transfer with FirecREST -A client application that makes requests to FirecREST will not be using directly the credentials of the user for the authentication, but an access token instead. The access token is a signed JSON Web Token (JWT) which contains expiry information. Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. You can manage your client application on the CSCS Developer Portal. +In addition to the [external transfer methods at CSCS][ref-data-xfer-external], FirecREST provides automated data transfer within the API. -Every client will have a client ID (Consumer Key) and a secret (Consumer Secret) that will be used to get a short-lived access token with an HTTP request. +A staging area is used for external transfers and downloading/uploading a file from/to a CSCS filesystem. -!!! example "curl call to fetch the access token" - ``` - curl -s -X POST https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token \ - --data "grant_type=client_credentials" \ - --data "client_id=" \ - --data "client_secret=" +!!!Note + pyFirecREST (both v1 and v2) hides this complexity to the user. We strongly recommend to use this library for these tasks. + +#### Version 2 + +!!! example "Upload a large file using FirecREST-v2" + ```python + + import firecrest as f7t + + (...) + + system = "daint" + source_path = "/path/to/local/file" + target_dir = "/capstor/scratch/cscs/" + target_file = "file" + account = "" + + + upload_task = client.upload(system, + local_file=source_path, + directory=target_dir, + filename=target_file, + account=account, + blocking=True) ``` -### Downloading Large Files +!!! example "Download a large file using FirecREST-v2" + ```python -A staging area is used for external transfers and downloading/uploading a file from/to a CSCS filesystem. + import firecrest as f7t + + (...) + + system = "daint" + source_path = "/capstor/scratch/cscs//file" + target_path = "/path/to/local/file" + account = "" + + + download_task = client.download(system, + source_path=source_path, + target_path=target_path, + account=account, + blocking=True) + + + ``` + +#### Version 1 Please follow the steps below to download a file: @@ -195,7 +396,7 @@ If you wish to transfer data bigger than the limit mentioned above, you can chec The limit on the time and size of files that can be download/uploaded via FirecREST might change if needed. -!!! example "checking the current values in the parameters endpoint" +!!! example "Checking the current values in the parameters endpoint" ```python >>> print(json.dumps(client.parameters(), indent = 2)) { @@ -233,9 +434,14 @@ This option ensures that the job submitted uses the same environment as your log ## Further Information -* [Full list of FirecREST calls](https://firecrest.readthedocs.io/en/latest/reference.html) -* [Documentation of FirecREST](https://firecrest.readthedocs.io/) -* [Documentation of pyFirecREST](https://pyfirecrest.readthedocs.io/) -* [Demo FirecREST environment in Git](https://github.com/eth-cscs/firecrest/tree/master/deploy/demo) -* [www.oauth.com](https://www.oauth.com) +* [FirecREST UI for HPC Platform](https://firecrest-ui-hpc.cscs.ch) +* [FirecREST OpenAPI Specification Version 1](https://firecrest.readthedocs.io/en/latest/reference.html) +* [FirecREST OpenAPI Specification Version 2](https://eth-cscs.github.io/firecrest-v2/openapi) +* [FirecREST Docs Version 1](https://firecrest.readthedocs.io/) +* [FirecREST Docs Version 2](https://eth-cscs.github.io/firecrest-v2) +* [Documentation of pyFirecREST (v1/v2)](https://pyfirecrest.readthedocs.io/) +* [FirecREST repository Version 1](https://github.com/eth-cscs/firecrest) +* [FirecREST repository Version 2](https://github.com/eth-cscs/firecrest-v2) +* [What are JSON Web Tokens](https://jwt.io/introduction) * [Python Requests](https://requests.readthedocs.io/en/master/user/quickstart) +* [Python Async API Calls](https://docs.aiohttp.org/en/stable/) From e054d011f6c2ca0c12b4b0a3c649a517c434d4e8 Mon Sep 17 00:00:00 2001 From: jdorsch Date: Thu, 20 Mar 2025 09:51:40 +0100 Subject: [PATCH 2/7] changed intro sentence --- docs/services/firecrest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/services/firecrest.md b/docs/services/firecrest.md index 8a64001a..f010156f 100644 --- a/docs/services/firecrest.md +++ b/docs/services/firecrest.md @@ -15,7 +15,7 @@ Users can make HTTP requests to perform the following operations: ## FirecREST versions -Beginning 2025, CSCS has introduced a new version of the API: [FirecREST version 2](https://eth-cscs.github.io/firecrest-v2). +Starting early 2025, CSCS has introduced a new version of the API: [FirecREST version 2](https://eth-cscs.github.io/firecrest-v2). Version 2 is faster, easier to use, and more efficient in resource management than its predecesor, therefore, if you are new to FirecREST start directly using **version 2** (if available for your platform). From 69bc5fe08eb874ed3561602a53b79f14ba8e1642 Mon Sep 17 00:00:00 2001 From: jdorsch Date: Thu, 20 Mar 2025 11:37:21 +0100 Subject: [PATCH 3/7] added codeowners for firecrest --- .github/CODEOWNERS | 1 + 1 file changed, 1 insertion(+) diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index 10d13999..f84c550c 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -1,3 +1,4 @@ * @bcumming @msimberg @RMeli docs/software/sciapps/cp2k @abussy @RMeli docs/software/communication @msimberg +docs/services/firecrest @jpdorsch @ekouts From 60b42ec9d340aec5596d53052b53c615739956c1 Mon Sep 17 00:00:00 2001 From: Juan Pablo Dorsch Date: Thu, 20 Mar 2025 11:54:31 +0100 Subject: [PATCH 4/7] Update docs/services/firecrest.md Co-authored-by: Ben Cumming --- docs/services/firecrest.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/services/firecrest.md b/docs/services/firecrest.md index f010156f..42c6efbc 100644 --- a/docs/services/firecrest.md +++ b/docs/services/firecrest.md @@ -55,7 +55,9 @@ FirecREST is available for all three major [Alps platforms][ref-alps-platforms], ### Clients and Access tokens -A **client application** that makes requests to FirecREST will not be using directly the credentials of the user for the authentication, but an **access token** instead. The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and it's valid for a short time (5 minutes). Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. +For authenticating requests to FirecREST, **client applications** use an **access token** instead of directly using the user's credentials. +The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and is only valid for a short time (5 minutes). +Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. Every client will have a client ID (Consumer Key) and a secret (Consumer Secret) that will be used to get a short-lived access token with an HTTP request. From f7595d347d4dee88c4f9cfe307dc87d39f7d3edd Mon Sep 17 00:00:00 2001 From: jdorsch Date: Fri, 21 Mar 2025 14:40:08 +0100 Subject: [PATCH 5/7] added requested changes --- docs/services/firecrest.md | 533 +++++++++++++++++++------------------ 1 file changed, 270 insertions(+), 263 deletions(-) diff --git a/docs/services/firecrest.md b/docs/services/firecrest.md index f010156f..670b8d97 100644 --- a/docs/services/firecrest.md +++ b/docs/services/firecrest.md @@ -3,7 +3,7 @@ FirecREST is a RESTful API for programatically accessing High-Performance Computing resources, developed at CSCS. -Users can make use of FirecREST to automate access to HPC, enabling [CI/CD pipelines](https://github.com/eth-cscs/firecrest/tree/master/examples/CI-pipeline), [Worflow Managers](https://github.com/eth-cscs/firecrest/tree/master/examples/airflow-operators), and other tools agains HPC resources. +Users can make use of FirecREST to automate access to HPC, enabling [CI/CD pipelines](https://github.com/eth-cscs/firecrest/tree/master/examples/CI-pipeline), [workflow managers](https://github.com/eth-cscs/firecrest/tree/master/examples/airflow-operators), and other tools against HPC resources. Additionally, scientific platform developers can integrate FirecREST into [web-enabled portals](https://firecrest-ui-hpc.cscs.ch) and [applications](https://github.com/eth-cscs/firecrest/tree/master/examples/UI-client-credentials), allowing them to securely access authenticated and authorized CSCS services such as job submission and data transfer on HPC systems. @@ -21,20 +21,20 @@ Version 2 is faster, easier to use, and more efficient in resource management th If you're using **version 1**, we recommend you to port your applications to use the new version. -### Version 2 +=== "Version 2" -!!!Warning - Documentation for version 2 is still work in progress + !!!Warning + Documentation for version 2 is still work in progress -For a full feature set, have a look at the latest [FirecREST version 2 API specification](https://eth-cscs.github.io/firecrest-v2/openapi) deployed at CSCS. + For a full feature set, have a look at the latest [FirecREST version 2 API specification](https://eth-cscs.github.io/firecrest-v2/openapi) deployed at CSCS. -Please refer to the [FirecREST-v2 documentation](https://eth-cscs.github.io/firecrest-v2/use) for detailed documentation. + Please refer to the [FirecREST-v2 documentation](https://eth-cscs.github.io/firecrest-v2/use) for detailed documentation. -### Version 1 +=== "Version 1" -For a full feature set, have a look at the latest [FirecREST version 1 API specification](https://firecrest-docs.v1.svc.cscs.ch/) deployed at CSCS. + For a full feature set, have a look at the latest [FirecREST version 1 API specification](https://firecrest-docs.v1.svc.cscs.ch/) deployed at CSCS. -Please refer to the [FirecREST-v1 documentation](https://firecrest.readthedocs.io/en/latest/) for detailed documentation. + Please refer to the [FirecREST-v1 documentation](https://firecrest.readthedocs.io/en/latest/) for detailed documentation. ## FirecREST Deployment on Alps @@ -53,13 +53,13 @@ FirecREST is available for all three major [Alps platforms][ref-alps-platforms], ## Accessing FirecREST -### Clients and Access tokens +### Clients and access tokens A **client application** that makes requests to FirecREST will not be using directly the credentials of the user for the authentication, but an **access token** instead. The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and it's valid for a short time (5 minutes). Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. -Every client will have a client ID (Consumer Key) and a secret (Consumer Secret) that will be used to get a short-lived access token with an HTTP request. +Every client has a client ID (Consumer Key) and a secret (Consumer Secret) that are used to get a short-lived access token with an HTTP request. -!!! example "curl call to fetch the access token" +??? example "`curl` call to fetch the access token" ``` curl -s -X POST https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token \ --data "grant_type=client_credentials" \ @@ -72,7 +72,7 @@ You can manage your client application on the [CSCS Developer Portal][ref-devpor [](){#ref-devportal} ### CSCS Developer Portal -The [Developer Portal](https://developer.cscs.ch) facilitates CSCS Users to manage subscriptions to an API at CSCS (such as FirecREST v1/v2). +The [Developer Portal](https://developer.cscs.ch) facilitates CSCS users to manage subscriptions to an API at CSCS (such as FirecREST v1/v2). Start by browsing to [developer.cscs.ch](https://developer.cscs.ch), then sign in by clicking the "SIGN-IN" button on the top right hand corner of the page. @@ -142,186 +142,184 @@ To use your Application to access FirecREST, follow the [API documentation](http One way to get started is by using [pyFirecREST](https://pyfirecrest.readthedocs.io/), a Python package with a collection of wrappers for the different functionalities of FirecREST. This package simplifies the usage of FirecREST by making multiple requests in the background for more complex workflows as well as by refreshing the access token before it expires. -#### Try version 2 - -!!! example "Try FirecREST using pyFirecREST v2" - ```python - import json - import firecrest as f7t - - client_id = "" - client_secret = "" - token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token" - - # Setup the client for the specific account - # For instance, for the Alps HPC Platform system Daint: - - client = f7t.v2.Firecrest( - firecrest_url="https://api.cscs.ch/hpc/firecrest/v2", - authorization=f7t.ClientCredentialsAuth(client_id, client_secret, token_uri) - ) - - # Status of the systems, filesystems and schedulers: - print(json.dumps(client.systems(), indent=2)) - - # Output: information about systems and health status of the infrastructure - # [ - # { - # "name": "daint", - # "ssh": { # --> SSH settings - # "host": "daint.alps.cscs.ch", - # "port": 22, - # "maxClients": 100, - # "timeout": { - # "connection": 5, - # "login": 5, - # "commandExecution": 5, - # "idleTimeout": 60, - # "keepAlive": 5 - # } - # }, - # "scheduler": { # --> Scheduler settings - # "type": "slurm", - # "version": "24.05.4", - # "apiUrl": null, - # "apiVersion": null, - # "timeout": 10 - # }, - # "servicesHealth": [ # --> Health status of services - # { - # "serviceType": "scheduler", - # "lastChecked": "2025-03-18T23:34:51.167545Z", - # "latency": 0.4725925922393799, - # "healthy": true, - # "message": null, - # "nodes": { - # "available": 21, - # "total": 858 - # } - # }, - # { - # "serviceType": "ssh", - # "lastChecked": "2025-03-18T23:34:52.054056Z", - # "latency": 1.358715295791626, - # "healthy": true, - # "message": null - # }, - # { - # "serviceType": "filesystem", - # "lastChecked": "2025-03-18T23:34:51.969350Z", - # "latency": 1.2738196849822998, - # "healthy": true, - # "message": null, - # "path": "/capstor/scratch/cscs" - # }, - # (...) - # "fileSystems": [ # --> Filesystem settings - # { - # "path": "/capstor/scratch/cscs", - # "dataType": "scratch", - # "defaultWorkDir": true - # }, - # { - # "path": "/users", - # "dataType": "users", - # "defaultWorkDir": false - # }, - # { - # "path": "/capstor/store/cscs", - # "dataType": "store", - # "defaultWorkDir": false - # } - # ] - # } - # ] - - # List content of directories - print(json.dumps(client.list_files("daint", "/capstor/scratch/cscs/"), - indent=2)) - - # [ - # { - # "name": "directory", - # "type": "d", - # "linkTarget": null, - # "user": "", - # "group": "", - # "permissions": "rwxr-x---+", - # "lastModified": "2024-09-02T12:34:45", - # "size": "4096" - # }, - # { - # "name": "file.txt", - # "type": "-", - # "linkTarget": null, - # "user": "", - # "group": "", - # "permissions": "rw-r-----+", - # "lastModified": "2024-09-02T08:26:04", - # "size": "131225" - # } - # ] - ``` - - -#### Try version 1 - -!!! example "Try FirecREST using pyFirecREST v1" - ```python - import json - import firecrest as f7t - - client_id = "" - client_secret = "" - token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token" - - # Setup the client for the specific account - # For instance, for the Alps HPC Platform system Daint: - - client = f7t.v1.Firecrest( - firecrest_url="https://api.cscs.ch/hpc/firecrest/v1", - authorization=f7t.ClientCredentialsAuth(client_id, client_secret, token_uri) - ) - - print(json.dumps(client.all_systems(), indent=2)) - # Output: (one dictionary per system) - # [{ - # 'description': 'System ready', - # 'status': 'available', - # 'system': 'daint' - # }] - - print(json.dumps(client.list_files('daint', '/capstor/scratch/cscs/'), - indent=2)) - # Example output: (one dictionary per file) - # [ - # { - # 'name': 'directory', - # 'user': '' - # 'last_modified': '2024-04-20T11:22:41', - # 'permissions': 'rwxr-xr-x', - # 'size': '4096', - # 'type': 'd', - # 'group': '', - # 'link_target': '', - # } - # { - # 'name': 'file.txt', - # 'user': '' - # 'last_modified': '2024-09-02T08:26:04', - # 'permissions': 'rw-r--r--', - # 'size': '131225', - # 'type': '-', - # 'group': '', - # 'link_target': '', - # } - # ] - ``` - +=== "Version 2" + + ??? example "Try FirecREST using pyFirecREST v2" + ```python + import json + import firecrest as f7t + + client_id = "" + client_secret = "" + token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token" + + # Setup the client for the specific account + # For instance, for the Alps HPC Platform system Daint: + + client = f7t.v2.Firecrest( + firecrest_url="https://api.cscs.ch/hpc/firecrest/v2", + authorization=f7t.ClientCredentialsAuth(client_id, client_secret, token_uri) + ) + + # Status of the systems, filesystems and schedulers: + print(json.dumps(client.systems(), indent=2)) + + # Output: information about systems and health status of the infrastructure + # [ + # { + # "name": "daint", + # "ssh": { # --> SSH settings + # "host": "daint.alps.cscs.ch", + # "port": 22, + # "maxClients": 100, + # "timeout": { + # "connection": 5, + # "login": 5, + # "commandExecution": 5, + # "idleTimeout": 60, + # "keepAlive": 5 + # } + # }, + # "scheduler": { # --> Scheduler settings + # "type": "slurm", + # "version": "24.05.4", + # "apiUrl": null, + # "apiVersion": null, + # "timeout": 10 + # }, + # "servicesHealth": [ # --> Health status of services + # { + # "serviceType": "scheduler", + # "lastChecked": "2025-03-18T23:34:51.167545Z", + # "latency": 0.4725925922393799, + # "healthy": true, + # "message": null, + # "nodes": { + # "available": 21, + # "total": 858 + # } + # }, + # { + # "serviceType": "ssh", + # "lastChecked": "2025-03-18T23:34:52.054056Z", + # "latency": 1.358715295791626, + # "healthy": true, + # "message": null + # }, + # { + # "serviceType": "filesystem", + # "lastChecked": "2025-03-18T23:34:51.969350Z", + # "latency": 1.2738196849822998, + # "healthy": true, + # "message": null, + # "path": "/capstor/scratch/cscs" + # }, + # (...) + # "fileSystems": [ # --> Filesystem settings + # { + # "path": "/capstor/scratch/cscs", + # "dataType": "scratch", + # "defaultWorkDir": true + # }, + # { + # "path": "/users", + # "dataType": "users", + # "defaultWorkDir": false + # }, + # { + # "path": "/capstor/store/cscs", + # "dataType": "store", + # "defaultWorkDir": false + # } + # ] + # } + # ] + + # List content of directories + print(json.dumps(client.list_files("daint", "/capstor/scratch/cscs/"), + indent=2)) + + # [ + # { + # "name": "directory", + # "type": "d", + # "linkTarget": null, + # "user": "", + # "group": "", + # "permissions": "rwxr-x---+", + # "lastModified": "2024-09-02T12:34:45", + # "size": "4096" + # }, + # { + # "name": "file.txt", + # "type": "-", + # "linkTarget": null, + # "user": "", + # "group": "", + # "permissions": "rw-r-----+", + # "lastModified": "2024-09-02T08:26:04", + # "size": "131225" + # } + # ] + ``` + +=== "Version 1" + + ??? example "Try FirecREST using pyFirecREST v1" + ```python + import json + import firecrest as f7t + + client_id = "" + client_secret = "" + token_uri = "https://auth.cscs.ch/auth/realms/firecrest-clients/protocol/openid-connect/token" + + # Setup the client for the specific account + # For instance, for the Alps HPC Platform system Daint: + + client = f7t.v1.Firecrest( + firecrest_url="https://api.cscs.ch/hpc/firecrest/v1", + authorization=f7t.ClientCredentialsAuth(client_id, client_secret, token_uri) + ) + + print(json.dumps(client.all_systems(), indent=2)) + # Output: (one dictionary per system) + # [{ + # 'description': 'System ready', + # 'status': 'available', + # 'system': 'daint' + # }] + + print(json.dumps(client.list_files('daint', '/capstor/scratch/cscs/'), + indent=2)) + # Example output: (one dictionary per file) + # [ + # { + # 'name': 'directory', + # 'user': '' + # 'last_modified': '2024-04-20T11:22:41', + # 'permissions': 'rwxr-xr-x', + # 'size': '4096', + # 'type': 'd', + # 'group': '', + # 'link_target': '', + # } + # { + # 'name': 'file.txt', + # 'user': '' + # 'last_modified': '2024-09-02T08:26:04', + # 'permissions': 'rw-r--r--', + # 'size': '131225', + # 'type': '-', + # 'group': '', + # 'link_target': '', + # } + # ] + ``` The tutorial is written for a generic instance of FirecREST but if you have a valid user at CSCS you can test it directly with your resource allocation on the exposed systems. -### Data Transfer with FirecREST +### Data transfer with FirecREST In addition to the [external transfer methods at CSCS][ref-data-xfer-external], FirecREST provides automated data transfer within the API. @@ -330,107 +328,116 @@ A staging area is used for external transfers and downloading/uploading a file f !!!Note pyFirecREST (both v1 and v2) hides this complexity to the user. We strongly recommend to use this library for these tasks. -#### Version 2 - -!!! example "Upload a large file using FirecREST-v2" - ```python +=== "Version 2" - import firecrest as f7t + #### Upload + !!! example "Upload a large file using FirecREST-v2" + ```python - (...) + import firecrest as f7t - system = "daint" - source_path = "/path/to/local/file" - target_dir = "/capstor/scratch/cscs/" - target_file = "file" - account = "" + (...) + system = "daint" + source_path = "/path/to/local/file" + target_dir = "/capstor/scratch/cscs/" + target_file = "file" + account = "" - upload_task = client.upload(system, - local_file=source_path, - directory=target_dir, - filename=target_file, - account=account, - blocking=True) - ``` -!!! example "Download a large file using FirecREST-v2" - ```python + upload_task = client.upload(system, + local_file=source_path, + directory=target_dir, + filename=target_file, + account=account, + blocking=True) + ``` + #### Download - import firecrest as f7t + !!! example "Download a large file using FirecREST-v2" + ```python - (...) + import firecrest as f7t - system = "daint" - source_path = "/capstor/scratch/cscs//file" - target_path = "/path/to/local/file" - account = "" + (...) + system = "daint" + source_path = "/capstor/scratch/cscs//file" + target_path = "/path/to/local/file" + account = "" - download_task = client.download(system, - source_path=source_path, - target_path=target_path, - account=account, - blocking=True) - - ``` + download_task = client.download(system, + source_path=source_path, + target_path=target_path, + account=account, + blocking=True) -#### Version 1 + + ``` -Please follow the steps below to download a file: +=== "Version 1" -1. Request FirecREST to move the file to the staging area: a download link will be provided -2. The file will remain in the staging area for 7 days or until the link gets invalidated with a request to the [`/storage/xfer-external/invalidate`](https://firecrest.readthedocs.io/en/latest/reference.html#post--storage-xfer-external-invalidate) endpoint or through the pyfirecrest method -3. The staging area is common for all users, therefore users should invalidate the link as soon as the download has been completed + Please follow the steps below to download a file: -You can see the full process in this [tutorial](https://firecrest.readthedocs.io/en/latest/tutorial.html#upload-with-non-blocking-call-something-bigger). + 1. Request FirecREST to move the file to the staging area: a download link will be provided + 2. The file will remain in the staging area for 7 days or until the link gets invalidated with a request to the [`/storage/xfer-external/invalidate`](https://firecrest.readthedocs.io/en/latest/reference.html#post--storage-xfer-external-invalidate) endpoint or through the pyfirecrest method + 3. The staging area is common for all users, therefore users should invalidate the link as soon as the download has been completed -We may be forced to delete older files sooner than 7 days whenever large files are moved to the staging area and the link is not invalidated after the download, to avoid issues for other users: we will contact the user in this case. + You can see the full process in this [tutorial](https://firecrest.readthedocs.io/en/latest/tutorial.html#upload-with-non-blocking-call-something-bigger). -When uploading files through the staging area, you don't need to invalidate the link. FirecREST will do it automatically as soon as it transfers the file to the filesystem of CSCS. + We may be forced to delete older files sooner than 7 days whenever large files are moved to the staging area and the link is not invalidated after the download, to avoid issues for other users: we will contact the user in this case. -There is also a constraint on the size of a single file to transfer externally to our systems via FirecREST: 5 GB. + When uploading files through the staging area, you don't need to invalidate the link. FirecREST will do it automatically as soon as it transfers the file to the filesystem of CSCS. -If you wish to transfer data bigger than the limit mentioned above, you can check the [compress](https://firecrest.readthedocs.io/en/latest/reference.html#post--storage-xfer-internal-compress) and [extract](https://firecrest.readthedocs.io/en/latest/reference.html#post--storage-xfer-internal-extract) endpoints or follow the following [example on how to split large files](https://github.com/eth-cscs/firecrest/blob/master/examples/download-large-files/README.md) and download/upload them using FirecREST. + There is also a constraint on the size of a single file to transfer externally to our systems via FirecREST: 5 GB. -The limit on the time and size of files that can be download/uploaded via FirecREST might change if needed. + If you wish to transfer data bigger than the limit mentioned above, you can check the [compress](https://firecrest.readthedocs.io/en/latest/reference.html#post--storage-xfer-internal-compress) and [extract](https://firecrest.readthedocs.io/en/latest/reference.html#post--storage-xfer-internal-extract) endpoints or follow the following [example on how to split large files](https://github.com/eth-cscs/firecrest/blob/master/examples/download-large-files/README.md) and download/upload them using FirecREST. -!!! example "Checking the current values in the parameters endpoint" - ```python - >>> print(json.dumps(client.parameters(), indent = 2)) - { - (...) + The limit on the time and size of files that can be download/uploaded via FirecREST might change if needed. - "storage": [ - { - "description": "Type of object storage, like `swift`, `s3v2` or `s3v4`.", - "name": "OBJECT_STORAGE", - "unit": "", - "value": "s3v4" - }, - { - "description": "Expiration time for temp URLs.", - "name": "STORAGE_TEMPURL_EXP_TIME", - "unit": "seconds", - "value": "604800" ## <-------- 7 days - }, + !!! example "Checking the current values in the parameters endpoint" + ```python + >>> print(json.dumps(client.parameters(), indent = 2)) { - "description": "Maximum file size for temp URLs.", - "name": "STORAGE_MAX_FILE_SIZE", - "unit": "MB", - "value": "5120" ## <--------- 5 GB + (...) + + "storage": [ + { + "description": "Type of object storage, like `swift`, `s3v2` or `s3v4`.", + "name": "OBJECT_STORAGE", + "unit": "", + "value": "s3v4" + }, + { + "description": "Expiration time for temp URLs.", + "name": "STORAGE_TEMPURL_EXP_TIME", + "unit": "seconds", + "value": "604800" ## <-------- 7 days + }, + { + "description": "Maximum file size for temp URLs.", + "name": "STORAGE_MAX_FILE_SIZE", + "unit": "MB", + "value": "5120" ## <--------- 5 GB + } + (...) } - (...) - } - ``` -### Job Submission to the Workload Manager through FirecREST + ``` +!!! Note on Job Submission through FirecREST + + FirecREST provides an abstraction for job submission using in the backend the SLURM scheduler of the vCluster. -FirecREST provides an abstraction for job submission using in the backend the SLURM scheduler of the vCluster (in the case of CSCS). + When submitting a job via the different [endpoints](https://firecrest.readthedocs.io/en/latest/reference.html#compute), you should pass the `-l` option to the `/bin/bash` command on the batch file. -When submitting a job via the different [endpoints](https://firecrest.readthedocs.io/en/latest/reference.html#compute), you should pass the `-l` option to the `/bin/bash` command on the batch file. + ```bash + #!/bin/bash -l + + #SBATCH --nodes=1 + ... + ``` -This option ensures that the job submitted uses the same environment as your login shell to access the system-wide profile (`/etc/profile`) or to your profile (in files like `~/.bash_profile`, `~/.bash_login`, or `~/.profile`). + This option ensures that the job submitted uses the same environment as your login shell to access the system-wide profile (`/etc/profile`) or to your profile (in files like `~/.bash_profile`, `~/.bash_login`, or `~/.profile`). ## Further Information From c483100018dfdc148e0dd4c73f95208588753910 Mon Sep 17 00:00:00 2001 From: jdorsch Date: Fri, 21 Mar 2025 14:42:41 +0100 Subject: [PATCH 6/7] added change on clients and access token --- docs/services/firecrest.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/docs/services/firecrest.md b/docs/services/firecrest.md index 670b8d97..d2d35024 100644 --- a/docs/services/firecrest.md +++ b/docs/services/firecrest.md @@ -55,7 +55,9 @@ FirecREST is available for all three major [Alps platforms][ref-alps-platforms], ### Clients and access tokens -A **client application** that makes requests to FirecREST will not be using directly the credentials of the user for the authentication, but an **access token** instead. The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and it's valid for a short time (5 minutes). Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. +For authenticating requests to FirecREST, **client applications** use an **access token** instead of directly using the user's credentials. +The access token is a signed JSON Web Token ([JWT](https://jwt.io/introduction)) which contains user information and is only valid for a short time (5 minutes). +Behind the API, all commands launched by the client will use the account of the user that registered the client, inheriting their access rights. Every client has a client ID (Consumer Key) and a secret (Consumer Secret) that are used to get a short-lived access token with an HTTP request. From f0179f92d9c6a28ecbbf1a1d5add4c03b9bd0a3a Mon Sep 17 00:00:00 2001 From: jdorsch Date: Fri, 21 Mar 2025 15:14:29 +0100 Subject: [PATCH 7/7] changed to note about job submission --- docs/services/firecrest.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/services/firecrest.md b/docs/services/firecrest.md index d2d35024..b4504a86 100644 --- a/docs/services/firecrest.md +++ b/docs/services/firecrest.md @@ -426,7 +426,7 @@ A staging area is used for external transfers and downloading/uploading a file f (...) } ``` -!!! Note on Job Submission through FirecREST +!!! Note "Job submission through FirecREST" FirecREST provides an abstraction for job submission using in the backend the SLURM scheduler of the vCluster.