Skip to content

Commit f2f6f6b

Browse files
d-j-hattond-j-hatton
authored
Send LIF to TIFF conversion to a cluster (#287)
Adds an endpoint plugin for workflows with the intention of extending in the future. Requires LIF to TIFF conversion to be packaged into a command line application
1 parent f0aa6c8 commit f2f6f6b

File tree

5 files changed

+115
-3
lines changed

5 files changed

+115
-3
lines changed

‎logfile‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
2024-05-09 09:10:03.740 BST [4144] LOG: starting PostgreSQL 15.1 on aarch64-apple-darwin20.0.0, compiled by clang version 14.0.6, 64-bit
2+
2024-05-09 09:10:03.742 BST [4144] LOG: listening on IPv4 address "127.0.0.1", port 5432
3+
2024-05-09 09:10:03.742 BST [4144] LOG: listening on IPv6 address "::1", port 5432
4+
2024-05-09 09:10:03.752 BST [4144] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
5+
2024-05-09 09:10:03.771 BST [4147] LOG: database system was shut down at 2024-04-18 00:09:47 BST
6+
2024-05-09 09:10:03.798 BST [4144] LOG: database system is ready to accept connections
7+
2024-05-09 09:10:11.460 BST [4184] FATAL: password authentication failed for user "kif41228"
8+
2024-05-09 09:10:11.460 BST [4184] DETAIL: Password does not match for user "kif41228".
9+
Connection matched pg_hba.conf line 87: "host all all 127.0.0.1/32 md5"
10+
2024-05-09 09:12:06.698 BST [4442] ERROR: null value in column "visit" of relation "session" violates not-null constraint
11+
2024-05-09 09:12:06.698 BST [4442] DETAIL: Failing row contains (2, Client connection, null, null).
12+
2024-05-09 09:12:06.698 BST [4442] STATEMENT: INSERT INTO session (name) VALUES ('Client connection') RETURNING session.id
13+
2024-05-09 09:12:07.026 BST [4447] ERROR: column rsyncinstance.client_id does not exist at character 110
14+
2024-05-09 09:12:07.026 BST [4447] STATEMENT: SELECT rsyncinstance.source AS rsyncinstance_source, rsyncinstance.destination AS rsyncinstance_destination, rsyncinstance.client_id AS rsyncinstance_client_id, rsyncinstance.files_transferred AS rsyncinstance_files_transferred, rsyncinstance.files_counted AS rsyncinstance_files_counted, rsyncinstance.transferring AS rsyncinstance_transferring
15+
FROM rsyncinstance
16+
WHERE 0 = rsyncinstance.client_id
17+
2024-05-09 09:15:03.771 BST [4145] LOG: checkpoint starting: time
18+
2024-05-09 09:16:21.945 BST [4145] LOG: checkpoint complete: wrote 199 buffers (1.2%); 0 WAL file(s) added, 0 removed, 0 recycled; write=78.122 s, sync=0.006 s, total=78.174 s; sync files=69, longest=0.001 s, average=0.001 s; distance=1479 kB, estimate=1479 kB
19+
2024-05-09 09:20:03.953 BST [4145] LOG: checkpoint starting: time
20+
2024-05-09 09:20:05.094 BST [4145] LOG: checkpoint complete: wrote 12 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.125 s, sync=0.004 s, total=1.142 s; sync files=9, longest=0.002 s, average=0.001 s; distance=37 kB, estimate=1335 kB
21+
2024-05-09 09:25:03.096 BST [4145] LOG: checkpoint starting: time
22+
2024-05-09 09:25:03.636 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.509 s, sync=0.020 s, total=0.540 s; sync files=6, longest=0.018 s, average=0.004 s; distance=4 kB, estimate=1202 kB
23+
2024-05-09 09:30:03.638 BST [4145] LOG: checkpoint starting: time
24+
2024-05-09 09:30:04.161 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.511 s, sync=0.003 s, total=0.524 s; sync files=6, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=1082 kB
25+
2024-05-09 09:35:03.164 BST [4145] LOG: checkpoint starting: time
26+
2024-05-09 09:35:03.696 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.512 s, sync=0.003 s, total=0.532 s; sync files=6, longest=0.002 s, average=0.001 s; distance=4 kB, estimate=974 kB
27+
2024-05-09 09:46:13.109 BST [4145] LOG: checkpoint starting: time
28+
2024-05-09 09:46:13.984 BST [4145] LOG: checkpoint complete: wrote 9 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.849 s, sync=0.003 s, total=0.876 s; sync files=9, longest=0.002 s, average=0.001 s; distance=17 kB, estimate=878 kB
29+
2024-05-09 09:51:13.982 BST [4145] LOG: checkpoint starting: time
30+
2024-05-09 09:51:14.737 BST [4145] LOG: checkpoint complete: wrote 8 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.724 s, sync=0.003 s, total=0.755 s; sync files=9, longest=0.001 s, average=0.001 s; distance=20 kB, estimate=793 kB
31+
2024-05-09 10:16:13.735 BST [4145] LOG: checkpoint starting: time
32+
2024-05-09 10:16:14.286 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.520 s, sync=0.003 s, total=0.552 s; sync files=6, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=713 kB
33+
2024-05-09 10:26:13.313 BST [4145] LOG: checkpoint starting: time
34+
2024-05-09 10:26:13.876 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.522 s, sync=0.003 s, total=0.564 s; sync files=6, longest=0.002 s, average=0.001 s; distance=2 kB, estimate=642 kB
35+
2024-05-09 10:41:13.885 BST [4145] LOG: checkpoint starting: time
36+
2024-05-09 10:41:14.429 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.520 s, sync=0.002 s, total=0.544 s; sync files=6, longest=0.001 s, average=0.001 s; distance=2 kB, estimate=578 kB
37+
2024-05-09 10:56:13.437 BST [4145] LOG: checkpoint starting: time
38+
2024-05-09 10:56:13.978 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.521 s, sync=0.002 s, total=0.542 s; sync files=6, longest=0.001 s, average=0.001 s; distance=4 kB, estimate=521 kB
39+
2024-05-09 11:01:13.974 BST [4145] LOG: checkpoint starting: time
40+
2024-05-09 11:01:14.501 BST [4145] LOG: checkpoint complete: wrote 6 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.508 s, sync=0.003 s, total=0.528 s; sync files=6, longest=0.001 s, average=0.001 s; distance=3 kB, estimate=469 kB
41+
2024-05-09 18:10:32.856 BST [4144] LOG: received smart shutdown request
42+
2024-05-09 18:10:32.861 BST [4144] LOG: background worker "logical replication launcher" (PID 4150) exited with exit code 1
43+
2024-05-09 18:10:32.861 BST [4145] LOG: shutting down
44+
2024-05-09 18:10:32.862 BST [4145] LOG: checkpoint starting: shutdown immediate
45+
2024-05-09 18:10:32.870 BST [4145] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.002 s, sync=0.001 s, total=0.010 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=422 kB
46+
2024-05-09 18:10:32.881 BST [4144] LOG: database system is shut down
47+
2024-05-18 20:06:25.662 BST [18530] LOG: starting PostgreSQL 15.1 on aarch64-apple-darwin20.0.0, compiled by clang version 14.0.6, 64-bit
48+
2024-05-18 20:06:25.665 BST [18530] LOG: listening on IPv4 address "127.0.0.1", port 5432
49+
2024-05-18 20:06:25.665 BST [18530] LOG: listening on IPv6 address "::1", port 5432
50+
2024-05-18 20:06:25.667 BST [18530] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432"
51+
2024-05-18 20:06:25.682 BST [18533] LOG: database system was shut down at 2024-05-16 18:11:47 BST
52+
2024-05-18 20:06:25.701 BST [18530] LOG: database system is ready to accept connections
53+
2024-05-18 20:11:25.681 BST [18531] LOG: checkpoint starting: time
54+
2024-05-18 20:11:25.730 BST [18531] LOG: checkpoint complete: wrote 2 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.021 s, sync=0.007 s, total=0.050 s; sync files=3, longest=0.007 s, average=0.003 s; distance=0 kB, estimate=0 kB
55+
2024-05-18 22:01:25.746 BST [18531] LOG: checkpoint starting: time
56+
2024-05-18 22:01:48.742 BST [18531] LOG: checkpoint complete: wrote 226 buffers (1.4%); 0 WAL file(s) added, 0 removed, 0 recycled; write=22.935 s, sync=0.007 s, total=22.996 s; sync files=70, longest=0.001 s, average=0.001 s; distance=1666 kB, estimate=1666 kB
57+
2024-05-18 22:14:16.494 BST [23372] ERROR: duplicate key value violates unique constraint "user_pkey"
58+
2024-05-18 22:14:16.494 BST [23372] DETAIL: Key (username)=(murfey) already exists.
59+
2024-05-18 22:14:16.494 BST [23372] STATEMENT: INSERT INTO "user" (username, hashed_password) VALUES ('murfey', '$2b$12$xkM0RPF2WO6XRFMBp/dKD.0C/8VKIdbnuknly.YyEEwemcgzrx3M2')
60+
2024-05-18 22:16:21.359 BST [23489] ERROR: relation "murfeyuser" does not exist at character 13
61+
2024-05-18 22:16:21.359 BST [23489] STATEMENT: INSERT INTO murfeyuser (username, hashed_password) VALUES ('murfey', '$2b$12$MAw4sOnG4/x18Hg4gFnFXeL1G4Mzh1OPXoH79HlUyL130L2opp2De')
62+
2024-05-18 22:16:25.748 BST [18531] LOG: checkpoint starting: time
63+
2024-05-18 22:16:26.205 BST [18531] LOG: checkpoint complete: wrote 5 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.438 s, sync=0.002 s, total=0.458 s; sync files=4, longest=0.001 s, average=0.001 s; distance=8 kB, estimate=1500 kB
64+
2024-05-18 22:21:25.207 BST [18531] LOG: checkpoint starting: time
65+
2024-05-18 22:21:46.496 BST [18531] LOG: checkpoint complete: wrote 209 buffers (1.3%); 0 WAL file(s) added, 0 removed, 0 recycled; write=21.236 s, sync=0.007 s, total=21.289 s; sync files=71, longest=0.002 s, average=0.001 s; distance=1599 kB, estimate=1599 kB
66+
2024-05-18 22:36:25.518 BST [18531] LOG: checkpoint starting: time
67+
2024-05-18 22:36:25.865 BST [18531] LOG: checkpoint complete: wrote 3 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.316 s, sync=0.002 s, total=0.348 s; sync files=2, longest=0.001 s, average=0.001 s; distance=12 kB, estimate=1440 kB
68+
2024-05-19 00:44:29.430 BST [18531] LOG: checkpoint starting: time
69+
2024-05-19 00:44:30.930 BST [18531] LOG: checkpoint complete: wrote 15 buffers (0.1%); 0 WAL file(s) added, 0 removed, 0 recycled; write=1.469 s, sync=0.003 s, total=1.501 s; sync files=9, longest=0.002 s, average=0.001 s; distance=41 kB, estimate=1300 kB
70+
2024-05-19 00:49:30.684 BST [18530] LOG: received smart shutdown request
71+
2024-05-19 00:49:30.685 BST [18530] LOG: background worker "logical replication launcher" (PID 18536) exited with exit code 1
72+
2024-05-19 00:49:30.686 BST [18531] LOG: shutting down
73+
2024-05-19 00:49:30.686 BST [18531] LOG: checkpoint starting: shutdown immediate
74+
2024-05-19 00:49:30.691 BST [18531] LOG: checkpoint complete: wrote 0 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0 recycled; write=0.001 s, sync=0.001 s, total=0.006 s; sync files=0, longest=0.000 s, average=0.000 s; distance=0 kB, estimate=1170 kB
75+
2024-05-19 00:49:30.703 BST [18530] LOG: database system is shut down

‎pyproject.toml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -86,6 +86,8 @@ murfey = "murfey.client:run"
8686
"murfey.spa_inject" = "murfey.cli.inject_spa_processing:run"
8787
"murfey.spa_ispyb_entries" = "murfey.cli.spa_ispyb_messages:run"
8888
"murfey.transfer" = "murfey.cli.transfer:run"
89+
[project.entry-points."murfey.workflows"]
90+
"lif_to_tiff" = "murfey.workflows.lif_to_tiff:zocalo_cluster_request"
8991

9092
[tool.setuptools]
9193
package-dir = {"" = "src"}

‎src/murfey/server/clem/api.py‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,10 @@
11
from __future__ import annotations
22

3+
import importlib.metadata
4+
35
from fastapi import APIRouter
46

7+
from murfey.server import _transport_object
58
from murfey.util.lif import convert_lif_to_tiff
69
from murfey.util.models import LifFileInfo
710

@@ -15,7 +18,15 @@ def lif_to_tiff(
1518
session_id: int, # Used by the decorator
1619
lif_info: LifFileInfo,
1720
):
18-
convert_lif_to_tiff(
19-
file=lif_info.name,
20-
root_folder="images",
21+
murfey_workflows = importlib.metadata.entry_points().select(
22+
group="murfey.workflows", name="lif_to_tiff"
2123
)
24+
if murfey_workflows:
25+
murfey_workflows[0].load()(
26+
file=lif_info.name, root_folder="images", messenger=_transport_object
27+
)
28+
else:
29+
convert_lif_to_tiff(
30+
file=lif_info.name,
31+
root_folder="images",
32+
)

‎src/murfey/workflows/__init__.py‎

Whitespace-only changes.

‎src/murfey/workflows/lif_to_tiff.py‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
from pathlib import Path
2+
3+
try:
4+
from murfey.server.ispyb import TransportManager # Session
5+
except AttributeError:
6+
pass # If ISPyB credentials environment variable not set ignore
7+
8+
9+
def zocalo_cluster_request(
10+
file: Path, root_folder: str, messenger: TransportManager | None = None
11+
):
12+
if messenger:
13+
messenger.send(
14+
"processing_recipe",
15+
{
16+
"recipes": ["lif-to-tiff"],
17+
"parameters": {
18+
"working_dir": str(file.parent.parent),
19+
"lif_path": str(file),
20+
"root_dir": root_folder,
21+
},
22+
},
23+
new_connection=True,
24+
)

0 commit comments

Comments
 (0)