-
Notifications
You must be signed in to change notification settings - Fork 1.1k
Pulsar-based ARC Job Runner #20598
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: dev
Are you sure you want to change the base?
Pulsar-based ARC Job Runner #20598
Conversation
`FastAPIJobFiles` is the new, FastAPI version of `JobFilesAPIController`. The endpoints that have been migrated should exhibit exactly the same behavior as the old ones from `FastAPIJobFiles`. Something to keep in mind is that while FastAPI has some extra built-in features that the legacy WSGI system did not have, such as answering HEAD requests, those do not work because of the way legacy WSGI endpoints are injected into the FastAPI app (using `app.mount("/", wsgi_handler)`), meaning that for example, HEAD requests are passed to the `wsgi_handler` sub-application.
Endpoints dedicated to TUS uploads work in tandem with the WSGI middleware `TusMiddleware` from the `tuswsgi` package. As explained above, WSGI middlewares and endpoints are injected into the FastAPI app after FastAPI routes as a single sub-application `wsgi_handler` using `app.mount("/", wsgi_handler)`, meaning that requests are passed to the `wsgi_handler` sub-application (and thus to `TusMiddleware`) only if there was no FastAPI endpoint defined to handle them. Therefore, they cannot be migrated to FastAPI unless `TusMiddleware` is also migrated to ASGI.
Work around a bug in FastAPI (fastapi/fastapi#13175) that assigns the same operation id to both request methods GET and HEAD of the endpoint `/api/jobs/{job_id}/files` when using the `@router.api_route()` decorator with `methods=["GET", "HEAD"]` as keyword argument.
…T requests to `/api/jobs/{job_id}/files`
Pulsar formats the `path` and `job_key` parameters as query parameters when submitting POST requests to `/api/jobs/{job_id}/files`. However, many Galaxy tests format them as form parameters. The only way to keep the endpoint working as it should (as it worked before the migration to FastAPI) is to accept both query and form parameters.
… requests to `/api/jobs/{job_id}/files`
FastAPI will not use the parameter aliases of form parameters in the OpenAPI docs, but the name of their Python variables. Therefore, the API docs show `path_form` and `job_key_form`. Rename them so that the API docs show the correct parameter names.
…ndpoint)
This commit introduces the following differences in behavior:
- GET and HEAD requests to `/api/jobs/{job_id}/files` return HTTP status code 400 when the given path does not refer to a file or the input datasets for the job have been purged and 404 when the given path does not exist.
- PROPFIND requests to `/api/jobs/{job_id}/files` are answered with HTTP status code 501 (read motivation for this change below).
- POST requests to `/api/jobs/{job_id}/files` are answered with HTTP status code 400 when no file is provided.
The reason behind the code explicitly answering `PROPFIND` requests with status code 501 is an unfortunate interaction between the ARC remote job runner that is under development, the behavior of legacy API endpoints and how they are integrated within the FastAPI app.
The ARC remote job runner (which will be implemented as `lib.galaxy.jobs.runners.pulsar.PulsarARCJobRunner`) expects this endpoint to return HTTP codes other than 404 when `PROPFIND` requests are issued. They are not part of the HTTP spec, but they are used in the WebDAV protocol. The correct answer to such requests is likely 501 (not implemented).
FastAPI returns HTTP 405 (method not allowed) for `PROPFIND`, which maybe is not fully correct but tolerable because it is one less quirk to maintain.
However, because of the way legacy WSGI endpoints are injected into the FastAPI app (using `app.mount("/", wsgi_handler)`), the built-in support for returning HTTP 405 for `PROPFIND` breaks, because such requests are passed to the `wsgi_handler` sub-application. This means that the endpoint still needs to include some code to handle this behavior.
When ALL routes have been migrated to ASGI (no WSGI handler sub-application needed anymore), some lines of code can be removed, they are labeled using comments.
…iles` This path already supports GET, HEAD and POST requests; add support for PUT requests. There is a significant difference in behavior between POST and PUT requests: - POST requests take `path` and `job_key` both as query parameters or as body parameters belonging to a multipart request. PUT requests take them only as query parameters (just like GET and HEAD). - POST requests submit a file as one of the fields of the multipart request, whereas the submitted file is the whole body of the request for PUT requests. - POST requests can append to the `tool_stdout` and `tool_stderr`, PUT requests can only create new files or overwrite whole files. - POST requests support resumable uploads but PUT requests do not. - POST requests take the form parameters `__file_path` (path of a file uploaded via the nginx upload module) and `__file` but PUT requests do not.
| "content": {"application/json": None, "application/octet-stream": {"example": None}}, | ||
| }, | ||
| 400: { | ||
| "description": "Path does not refer to a file, or input dataset(s) for job have been purged.", |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
| "content": { | ||
| "application/json": { | ||
| "example": { | ||
| "detail": ( |
Check failure
Code scanning / CodeQL
Polynomial regular expression used on uncontrolled data High
regular expression
user-provided value
| raise exceptions.RequestParameterInvalidException("Path does not refer to a file.") | ||
|
|
||
| return GalaxyFileResponse(path) | ||
|
|
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
| @router.put( | ||
| "/api/jobs/{job_id}/files", | ||
| summary="Populate an output file.", | ||
| responses={ |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
This path depends on a
user-provided value
| @router.put( | ||
| "/api/jobs/{job_id}/files", | ||
| summary="Populate an output file.", | ||
| responses={ |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
| summary="Populate an output file.", | ||
| responses={ | ||
| 201: {"description": "A new file has been created."}, | ||
| 204: {"description": "An existing file has been replaced."}, |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
user-provided value
Define a new Pulsar-based job runner to integrate Galaxy with the Advanced Resource Connector (ARC) middleware.
Add method to inject kwargs into the `build_client_manager()` function.
Rewrite the method `PulsarARCJobRunner.queue_job()` so that it: - Obtains the ARC endpoint URL either from the user's preferences or the destination parameters. - Requests an OIDC access token for the user running the job. - Decides which OIDC provider to get the token from if multiple are available, based on the user's preferences and the destination parameters. To let users configure their own settings, admins have to set the destination parameter "arc_user_preferences_key". Galaxy will then read the options "arc_url" and "arc_oidc_provider" under that key from the user extra preferences. Both are optional; if the user does not configure a value, the destination default will be used. If no destination default exists and the user account is associated with exactly one OIDC provider, then Galaxy will use that provider.
These tests verify that the ARC endpoint URL and OIDC provider selection logic works correctly when queuing jobs. They do not test actual job execution, as that is covered by Pulsar's own tests.
|
Note to self: adapt to changes from #20862 (for example how the client manager kwargs are populated). |
Implement a new Pulsar-based job runner to integrate Galaxy with the Advanced Resource Connector (ARC) middleware.
Requires:
JobFilesAPIControllerto FastAPI (excluding TUS uploads) #20235FastAPIJobFileswith the HTTP spec (files endpoint) #20351/api/jobs/{job_id}/filestoFastAPIJobFiles#20353How to test the changes?
(Select all options that apply)
License