|
8 | 8 | This library covers the UNICORE REST API, making common tasks like |
9 | 9 | file access, job submission and management, workflow submission and |
10 | 10 | management more convenient, and integrating UNICORE features better |
11 | | -with typical Python usage. |
| 11 | +with typical Python usage. Both blocking and non-blocking (asyncio) |
| 12 | +styles of communication with the server are supported. |
12 | 13 |
|
13 | 14 | The full, up-to-date documentation of the REST API can be found |
14 | 15 | [here](https://unicore-docs.readthedocs.io/en/latest/user-docs/rest-api) |
@@ -90,7 +91,7 @@ print(json.dumps(work_dir.properties, indent = 2)) |
90 | 91 |
|
91 | 92 | stdout = work_dir.stat("/stdout") |
92 | 93 | print(json.dumps(stdout.properties, indent = 2)) |
93 | | -content = stdout.raw().read() |
| 94 | +content = stdout.read() |
94 | 95 | print(content) |
95 | 96 | ``` |
96 | 97 |
|
@@ -152,120 +153,51 @@ You can use this feature in two ways |
152 | 153 |
|
153 | 154 | [More...](https://pyunicore.readthedocs.io/en/latest/port_forwarding.html) |
154 | 155 |
|
155 | | -## Dask cluster implementation (experimental) |
156 | | - |
157 | | -PyUNICORE provides an implementation of a Dask Cluster, allowing to |
158 | | -run the Dask client on your local host (or in a Jupyter notebook in |
159 | | -the Cloud), and have the Dask scheduler and workers running remotely |
160 | | -on the HPC site. |
161 | | - |
162 | | -[More...](https://pyunicore.readthedocs.io/en/latest/dask.html) |
| 156 | +## Asyncio support |
163 | 157 |
|
| 158 | +PyUNICORE has a fully async implementation of the basic UNICORE APIs in |
| 159 | +the package `pyunicore.aio.client`. As an example, running a job |
| 160 | +would look like this: |
164 | 161 |
|
165 | | -### Convert a CWL job to UNICORE |
166 | | - |
167 | | -PyUNICORE provides a tool to convert a CWL CommanLineTool and input into a |
168 | | -UNICORE job file. Given the following YAML files that describe a |
169 | | -CommandLineTool wrapper for the echo command and an input file: |
170 | | - |
171 | | -```yaml |
172 | | -# echo.cwl |
173 | | - |
174 | | -cwlVersion: v1.2 |
175 | | - |
176 | | -class: CommandLineTool |
177 | | -baseCommand: echo |
| 162 | +```Python |
| 163 | +import pyunicore.client as uc_client |
| 164 | +import pyunicore.credentials as uc_credentials |
178 | 165 |
|
179 | | -inputs: |
180 | | - message: |
181 | | - type: string |
182 | | - inputBinding: |
183 | | - position: 1 |
| 166 | +base_url = "https://localhost:8080/DEMO-SITE/rest/core" |
| 167 | +credential = uc_credentials.UsernamePassword("demouser", "test123") |
184 | 168 |
|
185 | | -outputs: [] |
| 169 | +async with uc_client.Client(credential, base_url) as client: |
| 170 | + my_job = {'Executable': 'date'} |
| 171 | + job = await client.new_job(job_description=my_job, inputs=[]) |
| 172 | + await job.poll() # wait for job to finish |
| 173 | + work_dir = await job.working_dir |
| 174 | + stdout = await work_dir.stat("/stdout") |
| 175 | + content = await stdout.read() |
| 176 | + print(content) |
186 | 177 | ``` |
187 | 178 |
|
188 | | -```yaml |
189 | | -# hello_world.yml |
| 179 | +More code examples can be found in the "integration-tests" |
| 180 | +folder in the source code repository. |
190 | 181 |
|
191 | | -message: "Hello World" |
192 | | -``` |
| 182 | +## Dask cluster implementation (experimental) |
193 | 183 |
|
194 | | -A UNICORE job file can be generated using the following command: |
| 184 | +PyUNICORE provides an implementation of a Dask Cluster, allowing to |
| 185 | +run the Dask client on your local host (or in a Jupyter notebook in |
| 186 | +the Cloud), and have the Dask scheduler and workers running remotely |
| 187 | +on the HPC site. |
| 188 | + |
| 189 | +[More...](https://pyunicore.readthedocs.io/en/latest/dask.html) |
195 | 190 |
|
196 | | -```bash |
197 | | -unicore-cwl-runner echo.cwl hello_world.yml > hello_world.u |
198 | | -``` |
199 | 191 |
|
200 | 192 | ## Helpers |
201 | 193 |
|
202 | 194 | The `pyunicore.helpers` module provides helper code for: |
203 | 195 |
|
204 | | -* Connecting to |
205 | | - * a Registry (`pyunicore.helpers.connect_to_registry`). |
206 | | - * a site via a Registry URL (`pyunicore.helpers.connect_to_site_from_registry`). |
207 | | - * a site via its core URL (`pyunicore.helpers.connect_to_site`). |
208 | 196 | * Defining descriptions as a dataclass and easily converting to a `dict` as required by `pyunicore.client.Client.new_job` via a `to_dict()` method: |
209 | 197 | * `pyunicore.helpers.jobs.Description` for `pyunicore.client.Client.new_job()` |
210 | 198 | * `pyunicore.helpers.workflows.Description` for `pyunicore.client.WorkflowService.new_workflow()` |
211 | 199 | * Defining a workflow description |
212 | 200 |
|
213 | | -### Connecting to a Registry |
214 | | - |
215 | | -```Python |
216 | | -import json |
217 | | -import pyunicore.credentials as uc_credentials |
218 | | -import pyunicore.helpers as helpers |
219 | | - |
220 | | -registry_url = "https://localhost:8080/REGISTRY/rest/registries/default_registry" |
221 | | - |
222 | | -credentials = uc_credentials.UsernamePassword("demouser", "test123") |
223 | | - |
224 | | -client = helpers.connection.connect_to_registry( |
225 | | - registry_url=registry_url, |
226 | | - credentials=credentials, |
227 | | -) |
228 | | -print(json.dumps(client.properties, indent=2)) |
229 | | -``` |
230 | | - |
231 | | -### Connecting to a site via a Registry |
232 | | - |
233 | | -```Python |
234 | | -import json |
235 | | -import pyunicore.credentials as uc_credentials |
236 | | -import pyunicore.helpers as helpers |
237 | | - |
238 | | -registry_url = "https://localhost:8080/REGISTRY/rest/registries/default_registry" |
239 | | -site = "DEMO-SITE" |
240 | | - |
241 | | -credentials = uc_credentials.UsernamePassword("demouser", "test123") |
242 | | - |
243 | | -client = helpers.connection.connect_to_site_from_registry( |
244 | | - registry_url=registry_url, |
245 | | - site_name=site, |
246 | | - credentials=credentials, |
247 | | -) |
248 | | -print(json.dumps(client.properties, indent=2)) |
249 | | -``` |
250 | | - |
251 | | -### Connecting to a site directly |
252 | | - |
253 | | -```Python |
254 | | -import json |
255 | | -import pyunicore.credentials as uc_credentials |
256 | | -import pyunicore.helpers as helpers |
257 | | - |
258 | | -site_url = "https://localhost:8080/DEMO-SITE/rest/core" |
259 | | - |
260 | | -credentials = uc_credentials.UsernamePassword("demouser", "test123") |
261 | | - |
262 | | -client = helpers.connection.connect_to_site( |
263 | | - site_api_url=site_url , |
264 | | - credentials=credentials, |
265 | | -) |
266 | | -print(json.dumps(client.properties, indent=2)) |
267 | | -``` |
268 | | - |
269 | 201 | ### Defining a job or workflow |
270 | 202 |
|
271 | 203 | ```Python |
|
0 commit comments