|
| 1 | +--- |
| 2 | +id: policy-package-algorithms |
| 3 | +title: Developing Policy Package algorithms |
| 4 | +--- |
| 5 | + |
| 6 | +### Adding a new algorithm class |
| 7 | + |
| 8 | +The system for registering algorithms within policy packages is |
| 9 | +intended to be extensible so that new algorithm classes can be added |
| 10 | +relatively easily. The basic workflow is as follows: |
| 11 | + |
| 12 | +- The `get_algorithms` function within the policy package (see above) |
| 13 | + should return a dictionary of functions of the new class, indexed |
| 14 | + by name |
| 15 | +- The core Rucio code should maintain a dictionary of functions of the |
| 16 | + new class, ready to be called when required. The details of this |
| 17 | + will differ depending on what the new class actually does and how it |
| 18 | + integrates with the Rucio code, but typically the algorithm name to |
| 19 | + be used will be selected by a value in the config file, as for the |
| 20 | + current `lfn2pfn` and `non_deterministic_pfn` algorithm types. |
| 21 | + |
| 22 | +### lfn2pfn vs. non_deterministic_pfn algorithms |
| 23 | + |
| 24 | +`lfn2pfn` algorithms and `non_deterministic_pfn` algorithms are |
| 25 | +conceptually similar, but there are important differences between |
| 26 | +them. Both produce a physical filename for use on an RSE, however |
| 27 | +`lfn2pfn` algorithms can only be used on deterministic RSEs - for |
| 28 | +example, disk systems where the appropriate physical filename can be |
| 29 | +derived from the file's scope and name alone (as well as |
| 30 | +protocol-specific information for the RSE in question). |
| 31 | +`non_deterministic_pfn` algorithms are used on non-deterministic |
| 32 | +RSEs (most often tape systems), and may use additional information |
| 33 | +about the file (such as its metadata, any datasets that it is a part |
| 34 | +of, etc.) to construct the physical filename. Because files cannot |
| 35 | +be uploaded directly to non-deterministic storage, |
| 36 | +`non_deterministic_pfn` algorithms are only ever called for |
| 37 | +replications, but `lfn2pfn` algorithms can also be called for |
| 38 | +initial uploads. |
| 39 | + |
| 40 | +The `lfn2pfn` algorithm to be used is determined by the |
| 41 | +`lfn2pfn_algorithm` attribute of the relevant RSE. If this is not set, |
| 42 | +the `lfn2pfn_algorithm_default` value from the `[policy]` section of |
| 43 | +the config file is used instead. The `non_deterministic_pfn` algorithm |
| 44 | +to be used is determined by the `naming_convention` attribute of the |
| 45 | +relevant RSE. |
0 commit comments