Skip to content

Commit 26de8eb

Browse files
committed
Add information on developing policy package algorithms
1 parent 942728c commit 26de8eb

File tree

1 file changed

+71
-3
lines changed

1 file changed

+71
-3
lines changed

docs/operator_policy_packages/policy_package_algorithms.md

Lines changed: 71 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -3,13 +3,81 @@ id: policy-package-algorithms
33
title: Developing Policy Package algorithms
44
---
55

6-
### Adding a new algorithm class
6+
A policy package can be used to define custom algorithms
7+
based on core Rucio algorithms, in order to support custom logic.
8+
9+
The current core algorithms are:
10+
- [`NonDeterministicPFNAlgorithms`](https://github.com/rucio/rucio/blob/0d44febcfd5d0a773a24d60668177324c534bd18/lib/rucio/common/utils.py#L384): Construct PFNs for non-deterministic RSEs
11+
- [`AutoApprove`](https://github.com/rucio/rucio/blob/0d44febcfd5d0a773a24d60668177324c534bd18/lib/rucio/core/rule.py#L96): Handle automatic approval for replication rules
12+
- [`ScopeExtractionAlgorithms`](https://github.com/rucio/rucio/blob/0d44febcfd5d0a773a24d60668177324c534bd18/lib/rucio/common/utils.py#L546): Extract scope from DID
13+
- [`RSEDeterministicScopeTranslation`](https://github.com/rucio/rucio/blob/0d44febcfd5d0a773a24d60668177324c534bd18/lib/rucio/rse/translation.py#L31): Translate a PFN dictionary into a scope and name
14+
- [`RSEDeterministicTranslation`](https://github.com/rucio/rucio/blob/0d44febcfd5d0a773a24d60668177324c534bd18/lib/rucio/rse/translation.py#L102): Translate an LFN to a path
15+
- [`FTS3TapeMetadataPlugin`](https://github.com/rucio/rucio/blob/0d44febcfd5d0a773a24d60668177324c534bd18/lib/rucio/transfertool/fts3_plugins.py#L29): Add `archive_metadata` to FTS transfers to `TAPE`.
16+
17+
Most of these algorithms implement some default behaviour.
18+
It is recommended to check this default behaviour to see
19+
if it is suitable for your needs; if it needs to be changed,
20+
please see below for how to create custom algorithms.
21+
22+
For `FTS3TapeMetadataPlugin` in particular,
23+
please check the (FTS3 Transfertool Plugin documentation)[operator_transfers/configure_fts3_plugins.md].
24+
25+
## Developing a custom algorithm
26+
To develop a custom algorithm:
27+
28+
1. Create a new class for your algorithm, subclassing the relevant core algorithm
29+
2. Register the custom algorithm with a name that is unique in relation to all other algorithms of that type
30+
(i.e. you cannot have two `RSEDeterministicScopeTranslation`-based algorithms with the same name,
31+
but you can have a `RSEDeterministicScopeTranslation`-based algorithm and a `ScopeExtractionAlgorithms`-based algorithm
32+
with the same name. In general, using your VO name should suffice.)
33+
3. Trigger registration of the algorithm by calling the relevant class method at the bottom of your file
34+
35+
As an example, this is the custom `RSEDeterministicScopeTranslation` algorithm used in ATLAS:
36+
37+
```python
38+
class ATLASScopeExtractionAlgorithm(rucio.common.utils.ScopeExtractionAlgorithms):
39+
def __init__(self) -> None:
40+
"""
41+
Initialises scope extraction algorithm object
42+
"""
43+
super().__init__()
44+
45+
@classmethod
46+
def _module_init_(cls) -> None:
47+
"""
48+
Registers the included scope extraction algorithms
49+
"""
50+
cls.register('atlas', cls.extract_scope_atlas)
51+
52+
@staticmethod
53+
def extract_scope_atlas(did: str, scopes: Optional['Sequence[str]']) -> 'Sequence[str]':
54+
# Try to extract the scope from the DSN
55+
if did.find(':') > -1:
56+
if len(did.split(':')) > 2:
57+
raise RucioException('Too many colons. Cannot extract scope and name')
58+
scope, name = did.split(':')[0], did.split(':')[1]
59+
if name.endswith('/'):
60+
name = name[:-1]
61+
return scope, name
62+
else:
63+
scope = did.split('.')[0]
64+
if did.startswith('user') or did.startswith('group'):
65+
scope = ".".join(did.split('.')[0:2])
66+
if did.endswith('/'):
67+
did = did[:-1]
68+
return scope, did
69+
70+
71+
ATLASScopeExtractionAlgorithm._module_init_()
72+
```
73+
74+
## Registering a custom algorithm in your policy package
775

876
The system for registering algorithms within policy packages is
977
intended to be extensible so that new algorithm classes can be added
1078
relatively easily. The basic workflow is as follows:
1179

12-
- The `get_algorithms` function within the policy package (see above)
80+
- The `get_algorithms` function within the policy package
1381
should return a dictionary of functions of the new class, indexed
1482
by name
1583
- The core Rucio code should maintain a dictionary of functions of the
@@ -19,7 +87,7 @@ relatively easily. The basic workflow is as follows:
1987
be used will be selected by a value in the config file, as for the
2088
current `lfn2pfn` and `non_deterministic_pfn` algorithm types.
2189

22-
### lfn2pfn vs. non_deterministic_pfn algorithms
90+
## lfn2pfn vs. non_deterministic_pfn algorithms
2391

2492
`lfn2pfn` algorithms and `non_deterministic_pfn` algorithms are
2593
conceptually similar, but there are important differences between

0 commit comments

Comments
 (0)