Skip to content

Commit e65e37b

Browse files
scarlehoffscarlehoff
authored
Merge pull request #2393 from NNPDF/fiatlux-enhanced
Make the photon a downloadable resource
2 parents 6dd3978 + d33571e commit e65e37b

File tree

14 files changed

+685
-195
lines changed

14 files changed

+685
-195
lines changed

‎doc/sphinx/source/tutorials/run-qed-fit.rst‎

Lines changed: 240 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -3,32 +3,255 @@
33
==========================
44
How to run a QED fit
55
==========================
6+
This tutorial describes how to run a QED fit using the LuxQED approach, as
7+
described in `arXiv:1607.04266 <https://arxiv.org/abs/1607.04266>`_ and
8+
`arXiv:1708.01256 <https://arxiv.org/abs/1708.01256>`_.
69

7-
It is possible to perform a QED fit adding the key ``fiatlux`` to the runcard. In this way
8-
a photon PDF will be generated using the FiatLux public library that implements the LuxQED
9-
(see :cite:p:`Manohar:2016nzj` and :cite:p:`Manohar:2017eqh`) approach.
10-
The parameters to be added are the following:
10+
Setting up the runcard
11+
----------------------
12+
13+
It is possible to perform a QED fit by adding a ``fiatlux`` block to the
14+
runcard. The following code snippet shows all the available options to set up
15+
a QED fit:
1116

1217
.. code-block:: yaml
1318
1419
fiatlux:
15-
luxset: NNPDF40_nnlo_as_01180
20+
luxset: NNPDF40_nnlo_as_01180_qcd
1621
additional_errors: true
1722
luxseed: 1234567890
23+
compute_in_setupfit: false
24+
eps_base: 1e-5
25+
26+
The parameters contained in the ``fiatlux`` block are:
27+
28+
* ``luxset``
29+
The name of the PDF set used to generate the photon PDF with `FiatLux
30+
<https://github.com/scarrazza/fiatlux/>`_. The code will use as many
31+
photon replicas as the number of replicas contained in the ``luxset``. If
32+
the user tries to generate a replica with ID higher than the maximum ID of
33+
the ``luxset``, the code will start reusing photon replica from the first
34+
replica. Being the LuxQED approach an iterated procedure, and that some
35+
replicas do not pass the ``postfit`` selection criteria, the user should
36+
make sure that the number of replicas in the ``luxset`` is high enough
37+
such that in the final iteration there will be a number of replicas higher
38+
than the final replicas desired.
39+
40+
* ``additional_errors``
41+
Boolean flag to switch on and off the additional errors of the LuxQED approach.
42+
43+
.. note::
44+
45+
The ``additional_errors`` flag should be switched on only in the very last
46+
iteration of the procedure.
47+
48+
* ``luxseed``
49+
The seed used to generate the additional errors of the LuxQED as in ``additional_errors``.
50+
51+
* ``compute_in_setupfit``
52+
Boolean flag to trigger the computation of the photon PDF set during
53+
``vp-setupfit``. By default, it is set to `false`. This is not required if
54+
the photon PDF set is already available locally or on the NNPDF server.
55+
See the :ref:`section below <running-with-photon-sets>` for more details.
56+
57+
* ``eps_base``
58+
(optional) The base precision of the FiatLux computation, which controls
59+
the precision on final integration of double integral. By default, it is
60+
set to ``1e-5``. This parameter is used only if the photon PDF set is
61+
generated on the fly, either during ``vp-setupfit`` or ``n3fit``. See
62+
:ref:`Generating new Photon sets <generating-photon-sets>` for more
63+
details.
64+
65+
The code uses information contained in both the ``fiatlux`` block and the
66+
``theoryid`` specified in the runcard to identify the photon PDF set, which will
67+
be assigned the name ``photon_theoryID_<theoryid>_fit_<luxset>``. This name will
68+
be used to either look for existing photon PDF sets or to store newly generated
69+
ones, as explained below.
70+
71+
.. _running-with-photon-sets:
1872

19-
``luxset`` is the PDF set used to generate the photon PDF with `FiatLux <https://github.com/scarrazza/fiatlux/>`.
20-
The code will generate as many photon replicas as the number of replicas contained in the ``luxset``. Therefore, if the user
21-
tries to generate a replica with ID higher than the maximum ID of the ``luxset``, the code will
22-
raise an error. Moreover, being the LuxQED approach an iterated prcedure, and given that some replicas
23-
do not pass the ``postfit`` selection criteria, the user should make sure that the number of replicas in
24-
the ``luxset`` is high enough such that in the final iteration there will be a number of replicas
25-
higher than the final replicas desired.
26-
``additional_errors`` is a parameter that switches on and off the additional errors of the LuxQED approach,
27-
while ``luxseed`` is the seed used to generate such errors.
28-
This errors should be switched on only in the very last iteration of the procedure.
73+
Running with Photon PDF sets
74+
-----------------------------
2975

30-
Whenever the photon PDF is generated, it will remain constant during the fit and will be considered in the momentum sum rule.
76+
The generation of a photon PDF set can add significant overhead to the fitting
77+
procedure. Moreover, the same photon PDF set might be used in different fits. To
78+
minimize the overhead due to the generation of photon PDF sets and avoid
79+
redundant computations, the code looks for precomputed photon resources either
80+
locally or on the NNPDF server. If a desired photon PDF set does not exist in
81+
either of the two locations, it will be computed on the fly and stored locally.
82+
If the user is satisfied with the new local photon PDF set, they can upload it to
83+
the NNPDF server following the instructions in :ref:`this section <generating-photon-sets>`.
84+
The following sections describe how to use existing photon PDF sets or generate
85+
new ones.
86+
87+
Using available Photon PDF sets
88+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
89+
Before running a QED fit, it is strongly advised to prepare the fit using
90+
``vp-setupfit``, as explained in :ref:`this tutorial <run-n3fit-fit>`. This will
91+
ensure that all the required resources are available, including the photon PDF.
92+
The desired photon PDF is specified by the ``luxset`` parameter in the
93+
``fiatlux`` block and the ``theoryid``, as explained above. The code will first
94+
look for the photon PDF set in the local directory specified in the
95+
:ref:`profile file <nnprofile>`. If the set is not found locally, it will try to
96+
download it from the NNPDF server. If the photon PDF set is not found on the
97+
server either, and the user has not requested to compute it during
98+
``vp-setupfit`` through the flag ``compute_in_setupfit``, the photon replica
99+
will be computed as needed for the replica fitted during the execution of
100+
``n3fit`` (see :ref:`here <photon_n3fit>` for more details).
101+
102+
The following is an example of running ``vp-setupfit`` using the
103+
``fiatlux`` block shown above and setting ``compute_in_setupfit: false``:
104+
105+
.. code-block:: bash
106+
107+
$ vp-setupfit qed_example_runcard.yml
108+
[INFO]: Could not find a resource (photonQED): Could not find Photon QED set /user/envs/nnpdf/share/NNPDF/photons_qed/photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed in theory: 40000100. Attempting to download it.
109+
[INFO]: Downloading https://data.nnpdf.science/photons/photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed.tar.
110+
[==================================================] (100%)
111+
[INFO]: Extracting archive to /opt/homebrew/Caskroom/miniconda/base/envs/nnpdf/share/NNPDF/photons_qed
112+
[INFO]: Photon QED set found for 40000100 with luxset NNPDF40_nnlo_as_01180_qed.
113+
[WARNING]: Using q2min from runcard
114+
[WARNING]: Using w2min from runcard
115+
Using Keras backend
116+
[INFO]: All requirements processed and checked successfully. Executing actions.
117+
...
118+
119+
In this case, the desired photon PDF set was already stored and precomputed on
120+
the server. This is downloaded and extracted in the local
121+
``photons_qed_path`` specified in the :ref:`profile file <nnprofile>`.
122+
123+
The ``vp-list`` utility tool can be used to list all the available photon PDF
124+
sets locally and on the NNPDF server. To list the available photon PDF sets,
125+
just run `vp-list photons` to see which photon PDF sets are available locally
126+
and which ones can be downloaded from the server. The user can manually download
127+
a photon PDF set using the ``vp-get`` tool as explained :ref:`here <download>`.
128+
For example:
129+
130+
.. code-block:: bash
131+
132+
$ vp-get photonQED 40000100 NNPDF40_nnlo_as_01180_qed
133+
[INFO]: Could not find a resource (photonQED): Could not find Photon QED set /user/envs/nnpdf/share/NNPDF/photons_qed/photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed in theory: 40000100. Attempting to download it.
134+
[INFO]: Downloading https://data.nnpdf.science/photons/photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed.tar.
135+
[==================================================] (100%)
136+
[INFO]: Extracting archive to /user/envs/nnpdf/share/NNPDF/photons_qed
137+
PosixPath('/user/envs/nnpdf/share/NNPDF/photons_qed/photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qed')
138+
139+
As in the case of ``vp-setupfit``, this will download and extract the photon PDF
140+
set in the local ``photons_qed_path`` specified in the :ref:`profile file <nnprofile>`.
141+
Once the photon PDF set is available locally, the user can proceed to run the
142+
fit with ``n3fit`` as usual.
31143

32144
.. warning::
33145

34-
At the moment it is not possible to run QED fits in parallel, as the FiatLux library cannot run in parallel.
146+
If ``vp-setupfit`` is not run before ``n3fit``, and the photon PDF set is not
147+
available locally, the code will **not** attempt to download it from the server,
148+
but will directly proceed to compute it on the fly. See the :ref:`next section <generating-photon-sets>` for
149+
more details.
150+
151+
.. _generating-photon-sets:
152+
153+
Generating new Photon PDF sets
154+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
155+
If the desired photon PDF set is not available locally nor on the NNPDF server, the
156+
code will generate the required photon PDF set replicas on the fly using `FiatLux <https://github.com/scarrazza/fiatlux/>`_.
157+
This can be done either during the ``vp-setupfit`` step, which precomputes all photon
158+
replicas before starting the fit, or during the fitting step with ``n3fit``, which
159+
generates the photon replica as needed for each replica being fitted.
160+
161+
.. note::
162+
163+
Generating photon PDF sets on the fly can add significant overhead to the
164+
fitting procedure. It is therefore strongly advised to precompute the photon
165+
PDF set using ``vp-setupfit`` before running the fit with ``n3fit``.
166+
167+
In either case, the generated photon PDF set will be stored locally in the
168+
``photons_qed_path`` specified in the :ref:`profile file <nnprofile>`, so that
169+
it can be reused in future fits. The folder containing the photon replicas will
170+
be named as ``photon_theoryID_<theoryid>_fit_<luxset>`` where ``<theoryid>`` and
171+
``<luxset>`` are the values specified in the runcard. The folder contains a ``npz``
172+
file for each photon replica, named as ``replica_<replica_id>.npz``. Each replica
173+
file contains the photon PDF grid computed with FiatLux at :math:`Q_{\rm init} = 100` GeV,
174+
prior to the evolution through EKO.
175+
176+
.. important::
177+
178+
Automatic upload to the NNPDF server through ``vp-upload`` is **not**
179+
supported at the moment. The user should manually create a ``tar`` archive
180+
file containing the photon replicas
181+
182+
.. code-block:: bash
183+
184+
$ cd my_photon_folder
185+
$ tar -czf my_photon.tgz *.npz
186+
187+
Once the archive file is created, the user can upload it to the server. Refer
188+
to the :ref:`profile file <nnprofile>` to find the remote URL where photon PDF
189+
sets are stored.
190+
191+
192+
Using ``vp-setupfit`` (preferred)
193+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
194+
195+
In order to trigger the computation of the photon PDF set during ``vp-setupfit``,
196+
the user needs to add the flag ``compute_in_setupfit: true`` in the ``fiatlux`` block
197+
discussed above. The following is an example of running ``vp-setupfit`` with
198+
this flag enabled:
199+
200+
.. code-block:: bash
201+
202+
$ vp-setupfit qed_example_runcard.yml
203+
[INFO]: Could not find a resource (photonQED): Could not find Photon QED set /user/envs/nnpdf/share/NNPDF/photons_qed/photon_theoryID_40000100_fit_NNPDF40_nnlo_as_01180_qcd in theory: 40000100. Attempting to download it.
204+
[ERROR]: Resource not in the remote repository: Photon QED set for TheoryID 40000100 and luxset NNPDF40_nnlo_as_01180_qcd is not available in the remote server.
205+
[WARNING]: Photon QED set for theory 40000100 with luxset NNPDF40_nnlo_as_01180_qcd not found. It will be computed in vp-setupfit.
206+
[INFO]: Forcing photon computation with FiatLux during setupfit.
207+
[WARNING]: Using q2min from runcard
208+
[WARNING]: Using w2min from runcard
209+
Using Keras backend
210+
[INFO]: All requirements processed and checked successfully. Executing actions.
211+
...
212+
213+
In addition, the user can also specify the optional parameter ``eps_base`` to
214+
the ``fiatlux`` block to set the base precision of the FiatLux computation,
215+
which controls the precision on final integration of double integral. By
216+
default, it is set to ``1e-5``. If the ``compute_in_setupfit`` flag is set to
217+
``true`` despite the photon PDF being available locally, the code will
218+
recompute and overwrite the existing photon PDF set
219+
220+
.. tip::
221+
222+
During ``vp-setupfit``, the code tries to distribute the computation of the
223+
photon replicas among the available CPU cores as specified in the
224+
``maxcores`` key of the runcard. This can significantly speed up the
225+
computation of the photon PDF set. Make sure to set ``maxcores`` to a value
226+
compatible with the available hardware. For example, using ``maxcores: 64``
227+
will try to compute up to 64 photon replicas in parallel using 64 GB of RAM.
228+
229+
Once the preparation step is completed, and the photon PDF set is generated and stored
230+
locally, the user can proceed to run the fit with ``n3fit`` as usual.
231+
232+
.. _photon_n3fit:
233+
Using ``n3fit`` (discouraged)
234+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
235+
If the user prefers to compute the photon PDF set during the fitting step with
236+
``n3fit``, no additional flag is needed in the runcard and ``vp-setupfit`` is
237+
not required beforehand (unless other resources are needed such as the
238+
:ref:`theory covariance matrix <vptheorycov-index>`). The code will check for
239+
the presence of the photon PDF set locally before starting the fit for each
240+
replica. If it is not found, it will proceed to compute the photon replica as
241+
needed for each replica being fitted. The following is an example of running
242+
``n3fit`` where the photon PDF set is computed during the fitting step:
243+
244+
.. code-block:: bash
245+
246+
$ n3fit qed_example_runcard.yml 1
247+
[INFO]: Creating replica output folder in /user/runcards/qed_example_runcard/nnfit/replica_1
248+
[WARNING]: Using q2min from runcard
249+
[WARNING]: Using w2min from runcard Using Keras backend
250+
[WARNING]: No Photon QED set found for Theory 40000100 with luxset NNPDF40_nnlo_as_01180_qed. It will be computed using FiatLux. This may impact performance. It is recommended to precompute the photon set before running the fit. Refer to https://docs.nnpdf.science/tutorials/run-qed-fit.html for more details on precomputing photon PDF sets.
251+
[INFO]: All requirements processed and checked successfully. Executing actions.
252+
253+
.. warning::
254+
255+
Computing the photon PDF set during `n3fit` with multiple replicas or using GPUs
256+
has not been tested and may lead to unexpected behaviour. It is strongly advised to
257+
precompute the photon PDF set using ``vp-setupfit`` before running the fit with ``n3fit``.

‎doc/sphinx/source/vp/download.rst‎

Lines changed: 35 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -6,16 +6,16 @@ Downloading resources
66
``validphys`` is designed so that, by default, resources stored in known remote
77
locations are downloaded automatically and seamlessly used where necessary.
88
Available resources include PDF sets, completed fits, theories, and results of
9-
past ``validphys`` runs that have been :ref:`uploaded to the server <upload>`.
10-
The ``vp-get`` tool, :ref:`described below <vp-get>`,
9+
past ``validphys`` runs that have been :ref:`uploaded to the server <upload>`.
10+
The ``vp-get`` tool, :ref:`described below <vp-get>`,
1111
can be used to download the same items manually.
1212

1313
Automatic operation
1414
-------------------
1515

1616
By default when some resource such as a PDF is required by ``validphys`` (or
1717
derived tools such as ``vp-setupfit``), the code will first look for it in some
18-
local directory specified in the [profile file](nnprofile). If it is not found
18+
local directory specified in the :ref:`profile file <nnprofile>`. If it is not found
1919
there, it will try to download it from some remote repository (also specified in
2020
the profile).
2121

@@ -70,6 +70,21 @@ Theories
7070
Theories (specified by the ``theoryid`` key) are downloaded and
7171
uncompressed.
7272

73+
Photon PDF sets
74+
Photon PDF sets can be downloaded if they have been previously generated
75+
using `FiatLux <https://github.com/scarrazza/fiatlux/>`_. A photon PDF set
76+
is uniquely identified by the input PDF set used to generate the photon replicas
77+
and the theory ID. See :ref:`this tutorial <run-qed-fit>` for more details on
78+
how to set up a QED fit.
79+
80+
EKOs
81+
Evolution kernels (EKOs) can be downloaded if they have been previously
82+
generated using EKO.
83+
84+
Hyperparameter scans
85+
Results of the hyperparameter scans generated in the hyperoptimization
86+
procedure.
87+
7388
``validphys`` output files
7489
Files produced by ``validphys`` can be used as input to subsequent validphys
7590
analyses (for example χ² tables are used for αs fits). The user needs to
@@ -80,7 +95,7 @@ Theories
8095

8196
.. _vp-get:
8297

83-
The `vp-get` tool
98+
The ``vp-get`` tool
8499
-----------------
85100

86101
The ``vp-get`` tool can be used to download resources manually, in the same way
@@ -99,8 +114,11 @@ They correspond to the resources described :ref:`above <what-can-be-downloaded>`
99114
100115
$ vp-get --list
101116
Available resource types:
117+
- eko
102118
- fit
119+
- hyperscan
103120
- pdf
121+
- photonQED
104122
- theoryID
105123
- vp_output_file
106124
@@ -118,6 +136,15 @@ information on it and bail out:
118136
$ vp-get fit NNPDF31_nlo_as_0118_1000
119137
FitSpec(name='NNPDF31_nlo_as_0118_1000', path=PosixPath('/home/zah/anaconda3/envs/nnpdf-dev/share/NNPDF/results/NNPDF31_nlo_as_0118_1000'))
120138
139+
.. note::
140+
141+
In order to download photon PDF sets, the user needs to specify both the input PDF
142+
set and the theory ID. For example:
143+
144+
.. code:: bash
145+
146+
$ vp-get photonQED 40000000 NNPDF40_nnlo_as_01180
147+
121148
Downloading resources in code (``validphys.loader``)
122149
----------------------------------------------------
123150

@@ -164,18 +191,18 @@ Conversely the ``Loader`` class will only search locally.
164191
----> 1 l.check_dataset('NMC', theoryid=151)
165192
166193
~/nngit/nnpdf/validphys2/src/validphys/loader.py in check_dataset(self, name, rules, sysnum, theoryid, cfac, frac, cuts, use_fitcommondata, fit, weight)
167-
416
194+
416
168195
417 if not isinstance(theoryid, TheoryIDSpec):
169196
--> 418 theoryid = self.check_theoryID(theoryid)
170-
419
197+
419
171198
420 theoryno, _ = theoryid
172199
173200
~/nngit/nnpdf/validphys2/src/validphys/loader.py in check_theoryID(self, theoryID)
174201
288 if not theopath.exists():
175202
289 raise TheoryNotFound(("Could not find theory %s. "
176203
--> 290 "Folder '%s' not found") % (theoryID, theopath) )
177204
291 return TheoryIDSpec(theoryID, theopath)
178-
292
205+
292
179206
180207
TheoryNotFound: Could not find theory 151. Folder '/home/zah/anaconda3/share/NNPDF/data/theory_151' not found
181208
@@ -184,10 +211,9 @@ Output files uploaded to the ``validphys`` can be retrieved specifying their pat
184211
(starting from the report ID). They will be either downloaded (when using
185212
``FallbackLoader``) or retrieved from the cache:
186213

187-
.. code:: python
214+
.. code:: python
188215
189216
from validphys.loader import FallbackLoader as Loader
190217
l = Loader()
191218
l.check_vp_output_file('qTpvLZLwS924oAsmpMzhFw==/figures/f_ns0_fitunderlyinglaw_plot_closure_pdf_histograms_0.pdf')
192219
PosixPath('/home/zah/anaconda3/share/NNPDF/vp-cache/qTpvLZLwS924oAsmpMzhFw==/figures/f_ns0_fitunderlyinglaw_plot_closure_pdf_histograms_0.pdf')
193-

0 commit comments

Comments
 (0)