diff --git a/.buildconfig/ci-linux.yml b/.buildconfig/ci-linux.yml index 65e815b54..64fa1a55d 100644 --- a/.buildconfig/ci-linux.yml +++ b/.buildconfig/ci-linux.yml @@ -10,27 +10,27 @@ channels: - nodefaults dependencies: - email-validator==2.2.0 - - h5py==3.12.1 - - hypothesis==6.119.4 + - h5py==3.13.0 + - hypothesis==6.131.20 - ipykernel==6.29.5 - - ipympl==0.9.4 - - ipywidgets==8.1.5 + - ipympl==0.9.7 + - ipywidgets==8.1.7 - lazy-loader==0.4 - - mantid==6.11.0 - - matplotlib==3.7.3 - - mpltoolbox==24.05.1 - - plopp==24.10.0 + - mantid==6.12.0 + - matplotlib==3.9.4 + - mpltoolbox==25.4.0 + - plopp==25.05.0 - pooch==1.8.2 - - pydantic==2.10.5 - - pytest==8.3.3 - - pytest-asyncio==0.24.0 + - pydantic==2.11.4 + - pytest==8.3.5 + - pytest-asyncio==0.26.0 - python-dateutil==2.9.0 - python-graphviz==0.20.3 - pythreejs==2.4.2 - - scipp==24.11.1 - - scippnexus==24.11.0 - - scipy==1.15.1 - - tox==4.23.2 + - scipp==25.05.0 + - scippnexus==25.04.0 + - scipy==1.15.2 + - tox==4.26.0 # docs - myst-parser==4.0.0 @@ -43,10 +43,7 @@ dependencies: - sphinx-copybutton==0.5.2 - sphinx-design==0.6.1 - sphinxcontrib-bibtex==2.6.3 - - tof==25.1.2 + - tof==25.05.0 # docs and tests - - sciline==24.10.0 - - # Temporary pin, see https://github.com/mantidproject/mantid/issues/39393 - - muparser==2.3.4 + - sciline==25.05.1 diff --git a/docs/tutorials/3_understanding-event-data.ipynb b/docs/tutorials/3_understanding-event-data.ipynb deleted file mode 100644 index 08fc3e5bc..000000000 --- a/docs/tutorials/3_understanding-event-data.ipynb +++ /dev/null @@ -1,1115 +0,0 @@ -{ - "cells": [ - { - "cell_type": "markdown", - "id": "0", - "metadata": { - "tags": [] - }, - "source": [ - "# Understanding Event Data" - ] - }, - { - "cell_type": "markdown", - "id": "1", - "metadata": {}, - "source": [ - "## Introduction\n", - "\n", - "Neutron-scattering data may be recorded in \"event mode\":\n", - "For each detected neutron a (pulse) timestamp and a time-of-flight is stored.\n", - "This notebook will develop an understanding of how do work with this type of data.\n", - "\n", - "Our objective is *not* to demonstrate or develop a full reduction workflow.\n", - "Instead we *develop understanding of data structures and opportunities* that event data provides.\n", - "\n", - "This tutorial contains exercises, but solutions are included directly.\n", - "We encourage you to download this notebook and run through it step by step before looking at the solutions.\n", - "Event data is a particularly challenging concept so make sure to understand every aspect before moving on.\n", - "We recommend to use a recent version of *JupyterLab*:\n", - "The solutions are included as hidden cells and shown only on demand.\n", - "\n", - "We use data containing event data from the POWGEN powder diffractometer at SNS.\n", - "Note that the data has been modified for the purpose of this tutorial and is not entirely in its original state.\n", - "We begin by loading the file and plot the raw data:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "2", - "metadata": {}, - "outputs": [], - "source": [ - "import scipp as sc\n", - "import scippneutron as scn\n", - "import scippneutron.data\n", - "import plopp as pp\n", - "\n", - "dg = scn.data.tutorial_event_data()\n", - "dg" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "3", - "metadata": {}, - "outputs": [], - "source": [ - "events = dg['events']\n", - "events.hist(spectrum=500, tof=400).plot()" - ] - }, - { - "cell_type": "markdown", - "id": "4", - "metadata": {}, - "source": [ - "We can see some diffraction lines, but they are oddly blurry.\n", - "There is also an artifact from the prompt-pulse visible at $4000~\\mu s$.\n", - "This tutorial illustrates how event data gives us the power to understand and deal with the underlying issues.\n", - "Before we start the investigation we cover some basics of working with event data.\n", - "\n", - "## Inspecting event data\n", - "\n", - "As usual, to begin exploring a loaded file, we can inspect the HTML representation of a scipp object shown by Jupyter when typing a variable at the end of a cell (this can also be done using `sc.to_html(da)`, anywhere in a cell):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "5", - "metadata": {}, - "outputs": [], - "source": [ - "events" - ] - }, - { - "cell_type": "markdown", - "id": "6", - "metadata": {}, - "source": [ - "We can tell that this is binned (event) data from the `dtype` of the data (usually `DataArrayView`) as well as the inline preview, denoting that this is binned data with lists of given lengths.\n", - "The meaning of these can best be understood using a graphical depiction of `da`, created using `sc.show`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "7", - "metadata": {}, - "outputs": [], - "source": [ - "sc.show(events)" - ] - }, - { - "cell_type": "markdown", - "id": "8", - "metadata": {}, - "source": [ - "Each value (yellow cube with dots) is a small table containing event parameters such as pulse time, time-of-flight, and weights (usually 1 for raw data).\n", - "\n", - "**Definitions**:\n", - "\n", - "1. In scipp we refer to each of these cubes (containing a table of events) as a *bin*.\n", - " We can think of this as a bin (or bucket) containing a number of records.\n", - "2. An array of bins (such as the array a yellow cubes with dots, shown above) is referred to as *binned variable*.\n", - " For example, `da.data` is a binned variable.\n", - "3. A data array with data given by a binned variable is referred to as *binned data*.\n", - " Binned data is a precursor to dense or histogrammed data.\n", - "\n", - "As we will see below binned data lets us do things that cannot or cannot properly be done with dense data, such as filtering or resampling.\n", - "\n", - "Each bin \"contains\" a small table, essentially a 1-D data array.\n", - "For efficiency and consistency scipp does not actually store an individual data array for every bin.\n", - "Instead each bin is a view to a section (slice) of a long table containing all the events from all bins combined.\n", - "This explains the `dtype=DataArrayView` seen in the HTML representation above.\n", - "For many practical purposes such a view of a data arrays behaves just like any other data array.\n", - "\n", - "The values of the bins can be accessed using the `values` property.\n", - "For dense data this might give us a `float` value, for binned data we obtain a table.\n", - "Here we access the 500th event list (counting from zero):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "9", - "metadata": {}, - "outputs": [], - "source": [ - "events.values[500]" - ] - }, - { - "cell_type": "markdown", - "id": "10", - "metadata": { - "tags": [] - }, - "source": [ - "### Exercise\n", - "\n", - "Use `sc.to_html()`, `sc.show()`, and `sc.table()` to explore and understand `da` as well as individual values of `da` such as `da.values[500]`." - ] - }, - { - "cell_type": "markdown", - "id": "11", - "metadata": {}, - "source": [ - "## From binned data to dense data\n", - "\n", - "While we often want to perform many operations on our data in event mode, a basic but important step is transformation of event data into dense data, since typically only the latter is suitable for data analysis software or plotting purposes.\n", - "There are two options we can use for this transformation, described in the following.\n", - "\n", - "### Option 1: Summing bins\n", - "\n", - "If the existing binning is sufficient for our purpose we may simply sum over the rows of the tables making up the bin values:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "12", - "metadata": {}, - "outputs": [], - "source": [ - "da_bin_sum = events.bins.sum()" - ] - }, - { - "cell_type": "markdown", - "id": "13", - "metadata": {}, - "source": [ - "Here we used the special `bins` property of our data array to apply an operation to each of the bins in `da`.\n", - "Once we have summed the bin values there are no more bins, and the `bins` property is `None`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "14", - "metadata": {}, - "outputs": [], - "source": [ - "print(da_bin_sum.bins)" - ] - }, - { - "cell_type": "markdown", - "id": "15", - "metadata": {}, - "source": [ - "We can visualize the result, which dense (histogram) data.\n", - "Make sure to compare the representations with those obtained above for binned data (`da`):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "16", - "metadata": {}, - "outputs": [], - "source": [ - "sc.to_html(da_bin_sum)\n", - "sc.show(da_bin_sum)" - ] - }, - { - "cell_type": "markdown", - "id": "17", - "metadata": {}, - "source": [ - "We can use `da_bins_sum` to, e.g., plot the total counts per spectrum by summing over the `tof` dimension:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "18", - "metadata": {}, - "outputs": [], - "source": [ - "da_bin_sum.sum('tof').plot(marker='.')" - ] - }, - { - "cell_type": "markdown", - "id": "19", - "metadata": {}, - "source": [ - "Note:\n", - "In this case there is just a single time-of-flight bin so we could have used `da_bin_sum['tof', 0]` instead of `da_bin_sum.sum('tof')`." - ] - }, - { - "cell_type": "markdown", - "id": "20", - "metadata": { - "tags": [] - }, - "source": [ - "### Option 2: Histogramming\n", - "\n", - "For performance and memory reasons binned data often contains the minimum number of bins that is \"necessary\" for a given purpose.\n", - "In this case `da` only contains a single time-of-flight bin (essentially just as information what the lower and upper bounds are in which we can expect events), which is not practical for downstream applications such as data analysis or plotting.\n", - "\n", - "Instead of simply summing over all events in a bin we may thus *histogram* data.\n", - "Note that scipp makes the distinction between binning data (preserving all events individually) and histogramming data (summing all events that fall inside a bin).\n", - "\n", - "For simplicity we consider only a single spectrum:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "21", - "metadata": {}, - "outputs": [], - "source": [ - "spec = events['spectrum', 8050]\n", - "sc.show(spec)" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "22", - "metadata": {}, - "outputs": [], - "source": [ - "sc.table(spec.values[0]['event', :5])" - ] - }, - { - "cell_type": "markdown", - "id": "23", - "metadata": {}, - "source": [ - "Note the chained slicing above:\n", - "We access the zeroth event list and select the first 5 slices along the `event` dimension (which is the only dimension, since the event list is a 1-D table).\n", - "\n", - "We use one of the [scipp functions for creating a new variable](https://scipp.github.io/reference/creation-functions.html) to define the desired bin edge of our histogram.\n", - "In this case we use `sc.linspace` (another useful option is `sc.geomspace`):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "24", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tof_edges = sc.linspace(dim='tof', start=18.0, stop=17000, num=101, unit='us')\n", - "spec.hist(tof=tof_edges).plot()" - ] - }, - { - "cell_type": "markdown", - "id": "25", - "metadata": { - "tags": [] - }, - "source": [ - "#### Exercise\n", - "\n", - "Change `tof_edges` to control what is plotted:\n", - "\n", - "- Change the number of bins, e.g., to a finer resolution.\n", - "- Change the start and stop of the edges to plot only a smaller time-of-flight region.\n", - "\n", - "#### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "26", - "metadata": { - "tags": [ - "solution" - ] - }, - "outputs": [], - "source": [ - "tof_edges = sc.linspace(dim='tof', start=2000.0, stop=15000, num=201, unit='us')\n", - "spec.hist(tof=tof_edges).plot()" - ] - }, - { - "cell_type": "markdown", - "id": "27", - "metadata": { - "tags": [] - }, - "source": [ - "## Masking event data — Binning by existing parameters\n", - "\n", - "While quickly converting binned (event) data into dense (histogrammed) data has its applications, we may typically want to work with binned data as long as possible.\n", - "We have learned in [Working with masks](2_working-with-masks.ipynb) how to mask dense, histogrammed, data.\n", - "How can we mask a time-of-flight region, e.g., to mask a prompt-pulse, in *event mode*?\n", - "\n", - "Let us sum all spectra and define a dummy data array (named `prompt`) to illustrate the objective:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "28", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "spec = events['spectrum', 8050].copy()\n", - "# Start and stop are fictitious and this prompt pulse is not actually present in the raw data from SNS\n", - "prompt_start = 4000.0 * sc.Unit('us')\n", - "prompt_stop = 5000.0 * sc.Unit('us')\n", - "prompt_tof_edges = sc.sort(\n", - " sc.concat([spec.coords['tof'], prompt_start, prompt_stop], 'tof'), 'tof'\n", - ")\n", - "prompt = sc.DataArray(\n", - " data=sc.array(dims=['tof'], values=[0, 11000, 0], unit='counts'),\n", - " coords={'tof': prompt_tof_edges},\n", - ")\n", - "spec_hist = events.bins.concat('spectrum').hist(tof=1701)\n", - "sc.plot({'spec': spec_hist, 'prompt': prompt})" - ] - }, - { - "cell_type": "markdown", - "id": "29", - "metadata": {}, - "source": [ - "### Masking events\n", - "\n", - "We now want to mask out the prompt-pulse, i.e., the peak with exponential falloff inside the region where `prompt` in the figure above is nonzero.\n", - "\n", - "We can do so by checking (for every event) whether the time-of-flight is within the region covered by the prompt-pulse.\n", - "As above, we first consider only a single spectrum.\n", - "The result can be stored as a new mask:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "30", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "spec1 = events['spectrum', 8050].copy() # copy since we do some modifications below\n", - "event_tof = spec.bins.coords['tof']\n", - "mask = (prompt_start <= event_tof) & (event_tof < prompt_stop)\n", - "spec1.bins.masks['prompt_pulse'] = mask\n", - "sc.plot(\n", - " {\n", - " 'original': events['spectrum', 8050].hist(tof=100),\n", - " 'prompt_mask': spec1.hist(tof=100),\n", - " },\n", - " errorbars=False,\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "31", - "metadata": {}, - "source": [ - "Here we have used the `bins` property once more.\n", - "Take note of the following:\n", - "\n", - "- We can access coords \"inside\" the bins using the `coords` dict provided by the `bins` property.\n", - " This provides access to \"columns\" of the event tables held by the bins such as `spec.bins.coords['tof']`.\n", - "- We can do arithmetic (or other) computation with these \"columns\", in this case comparing with scalar (non-binned) variables.\n", - "- New \"columns\" can be added, in this case we add a new mask column via `spec.bins.masks`.\n", - "\n", - "**Definitions**:\n", - "\n", - "For a data array `da` we refer to\n", - "- coordinates such as `da.coords['tof']` as *bin coordinate* and\n", - "- coordinates such as `da.bins.coords['tof']` as *event coordinate*.\n", - "\n", - "The table representation (`sc.table`) and `sc.show` illustrate this process of masking:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "32", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "sc.table(spec1.values[0]['event', :5])\n", - "sc.show(spec1)" - ] - }, - { - "cell_type": "markdown", - "id": "33", - "metadata": { - "tags": [] - }, - "source": [ - "We have added a new column to the event table, defining *for every event* whether it is masked or not.\n", - "\n", - "The generally recommended solution is different though, since masking individual events has unnecessary overhead and forces masks to be applied when converting to dense data.\n", - "A better approach is described in the next section.\n", - "\n", - "#### Exercise\n", - "\n", - "To get familiar with the `bins` property, try the following:\n", - "\n", - "- Compute the neutron velocities for all events in `spec1`.\n", - " Note: The total flight path length can be computed using `scn.Ltotal(spec1, scatter=True)`.\n", - "- Add the neutron velocity as a new event coordinate.\n", - "- Use, e.g., `sc.show` to verify that the coordinate has been added as expected.\n", - "- Use `del` to remove the event coordinate and verify that the coordinate was indeed removed.\n", - "\n", - "#### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "34", - "metadata": { - "tags": [ - "solution" - ] - }, - "outputs": [], - "source": [ - "spec1.bins.coords['v'] = scn.Ltotal(spec1, scatter=True) / spec1.bins.coords['tof']\n", - "sc.show(spec1)\n", - "sc.to_html(spec1.values[0])\n", - "del spec1.bins.coords['v']\n", - "sc.to_html(spec1.values[0])" - ] - }, - { - "cell_type": "markdown", - "id": "35", - "metadata": {}, - "source": [ - "### Masking bins\n", - "\n", - "Rather than masking individual events, let us simply \"sort\" the events depending on whether they fall below, inside, or above the region of the prompt-pulse.\n", - "We do not actually need to fully sort the events but rather use a *binning* procedure, using `sc.bin`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "36", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "spec2 = events['spectrum', 8050].copy() # copy since we do some modifications below\n", - "spec2 = sc.bin(spec2, tof=prompt_tof_edges)\n", - "prompt_mask = sc.array(dims=spec2.dims, values=[False, True, False])\n", - "spec2.masks['prompt_pulse'] = prompt_mask\n", - "sc.show(spec2)" - ] - }, - { - "cell_type": "markdown", - "id": "37", - "metadata": {}, - "source": [ - "Compare this to the graphical representation for `spec1` above and to the figure of the prompt pulse.\n", - "The start and stop of the prompt pulse are used to cut the total time-of-flight interval into three sections (bins).\n", - "The center bin is masked:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "38", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "spec2.masks['prompt_pulse']" - ] - }, - { - "cell_type": "markdown", - "id": "39", - "metadata": { - "tags": [] - }, - "source": [ - "#### Bonus question\n", - "\n", - "Why did we not use a fine binning, e.g., with 1000 time-of-flight bins and mask a range of bins, similar to how it would be done for histogrammed (non-event) data?\n", - "\n", - "#### Solution" - ] - }, - { - "cell_type": "markdown", - "id": "40", - "metadata": { - "tags": [ - "solution" - ] - }, - "source": [ - "- This would add a lot of over overhead from handling many bins.\n", - " If our instrument had 1.000.000 pixels we would have 1.000.000.000 bins, which comes with significant memory overhead but first and foremost compute overhead." - ] - }, - { - "cell_type": "markdown", - "id": "41", - "metadata": { - "tags": [] - }, - "source": [ - "## Binning by new parameters\n", - "\n", - "After having understood how to mask a prompt-pulse we continue by considering the proton-charge log:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "42", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "proton_charge = dg['proton_charge']\n", - "proton_charge.plot(marker='.')" - ] - }, - { - "cell_type": "markdown", - "id": "43", - "metadata": {}, - "source": [ - "To mask a time-of-flight range, we have used `sc.bin` to adapt the binning along the *existing* `tof` dimension.\n", - "`sc.bin` can also be used to introduce binning along *new* dimension.\n", - "We define our desired pulse-time edges:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "44", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tmin = proton_charge.coords['time'].min()\n", - "tmax = proton_charge.coords['time'].max()\n", - "pulse_time = sc.arange(\n", - " dim='pulse_time',\n", - " start=tmin.value,\n", - " stop=tmax.value,\n", - " step=(tmax.value - tmin.value) / 10,\n", - ")\n", - "pulse_time" - ] - }, - { - "cell_type": "markdown", - "id": "45", - "metadata": {}, - "source": [ - "As above we work with a single spectrum for now and then use `sc.bin`.\n", - "The result has two dimensions, `tof` and `pulse_time`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "46", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "spec = events['spectrum', 8050]\n", - "binned_spec = spec.bin(pulse_time=pulse_time)\n", - "binned_spec" - ] - }, - { - "cell_type": "markdown", - "id": "47", - "metadata": {}, - "source": [ - "We can plot the binned spectrum, resulting in a 2-D plot:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "48", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "binned_spec.hist(tof=20, pulse_time=100).plot()" - ] - }, - { - "cell_type": "markdown", - "id": "49", - "metadata": {}, - "source": [ - "We may also ignore the `tof` dimension if we are simply interested in the time-evolution of the counts in this spectrum.\n", - "We can do so by concatenating all bins along the `tof` dimension as follows:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "50", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "binned_spec.bins.concat('tof').hist(pulse_time=100).plot(errorbars=False)" - ] - }, - { - "cell_type": "markdown", - "id": "51", - "metadata": { - "tags": [] - }, - "source": [ - "### Exercise\n", - "\n", - "Using the same approach as for masking a time-of-flight bin in the previous section, mask the time period starting at about 16:30 where the proton charge is very low.\n", - "\n", - "- Define appropriate edges for pulse time (use as few bins as possible, not the 10 pulse-time bins from the binning example above).\n", - "- Use `sc.bin` to apply the new binning.\n", - " Make sure to combine this with the time-of-flight binning to mask the prompt pulse.\n", - "- Set an appropriate bin mask.\n", - "- Plot the result to confirm that the mask is set and defined as expected.\n", - "\n", - "Note:\n", - "In practice masking bad pulses would usually be done on a pulse-by-pulse basis.\n", - "This requires a slightly more complex approach and is beyond the scope of this introduction.\n", - "\n", - "Hint:\n", - "Pulse time is stored as `datetime64`.\n", - "A simple way to create these is using an offset from a know start time such as `tmin`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "52", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tmin + sc.to_unit(sc.scalar(7, unit='min'), 'ns')" - ] - }, - { - "cell_type": "markdown", - "id": "53", - "metadata": { - "tags": [] - }, - "source": [ - "### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "54", - "metadata": { - "tags": [ - "solution" - ] - }, - "outputs": [], - "source": [ - "pulse_time_edges = tmin + sc.to_unit(\n", - " sc.array(dims=['pulse_time'], values=[0, 43, 55, 92], unit='min'), 'ns'\n", - ")\n", - "# Alternative solution to creating edges:\n", - "# t1 = tmin + sc.to_unit(43 * sc.Unit('min'), 'ns')\n", - "# t2 = tmin + sc.to_unit(55 * sc.Unit('min'), 'ns')\n", - "# pulse_time_edges = sc.array(dims=['pulse_time'], unit='ns', values=[tmin.value, t1.value, t2.value, tmax.value])\n", - "\n", - "pulse_time_mask = sc.array(dims=['pulse_time'], values=[False, True, False])\n", - "binned_spec = spec.bin(tof=prompt_tof_edges, pulse_time=pulse_time_edges)\n", - "binned_spec.masks['prompt_pulse'] = prompt_mask\n", - "binned_spec.masks['bad_beam'] = pulse_time_mask\n", - "binned_spec.hist(tof=20, pulse_time=100).plot()" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "55", - "metadata": { - "tags": [ - "solution" - ] - }, - "outputs": [], - "source": [ - "sc.show(binned_spec)" - ] - }, - { - "cell_type": "markdown", - "id": "56", - "metadata": { - "tags": [] - }, - "source": [ - "## Higher dimensions and cuts\n", - "\n", - "For purposes of plotting, fitting, or data analysis in general we will typically need to convert binned data to dense data.\n", - "We discussed the basic options for this in [From binned data to dense data](#From-binned-data-to-dense-data).\n", - "In particular when dealing with higher-dimensional data these options may not be sufficient.\n", - "For example we may want to:\n", - "\n", - "- Create a 1-D or 2-D cut through a 3-D volume.\n", - "- Create a 2-D cut but integrate over an interval in the remaining dimension.\n", - "- Create multi-dimensional cuts that are not aligned with existing binning.\n", - "\n", - "All of the above can be achieved using tools we have already used, but not all of them are covered in this tutorial.\n", - "\n", - "### Exercise\n", - "\n", - "Adapt the above code used for binning and masking the *single spectrum* (`spec`) along `pulse_time` and `tof` to the *full data array* (`da`).\n", - "\n", - "Hint: This is trivial.\n", - "\n", - "### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "57", - "metadata": { - "tags": [ - "solution" - ] - }, - "outputs": [], - "source": [ - "%matplotlib widget\n", - "binned_da = events.bin(tof=prompt_tof_edges, pulse_time=pulse_time_edges)\n", - "binned_da.masks['prompt_pulse'] = prompt_mask\n", - "binned_da.masks['bad_beam'] = pulse_time_mask\n", - "pp.slicer(\n", - " binned_da.transpose(['pulse_time', 'spectrum', 'tof']).hist(spectrum=500, tof=400)\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "58", - "metadata": {}, - "source": [ - "### Removing binned dimensions\n", - "\n", - "Let us now convert our data to $d$-spacing (interplanar lattice spacing).\n", - "This works just like for dense data:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "59", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "import scippneutron as scn\n", - "\n", - "dspacing_graph = {\n", - " **scn.conversion.graph.beamline.beamline(scatter=True),\n", - " **scn.conversion.graph.tof.elastic_dspacing('tof'),\n", - "}\n", - "da_dspacing = binned_da.transform_coords('dspacing', graph=dspacing_graph)\n", - "# `dspacing` is now a multi-dimensional coordinate, which makes plotting inconvenient, so we adapt the binning\n", - "dspacing = sc.linspace(dim='dspacing', unit='Angstrom', start=0.0, stop=3.0, num=4)\n", - "da_dspacing = sc.bin(da_dspacing, dspacing=dspacing, dim=())\n", - "da_dspacing" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "60", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "pp.slicer(\n", - " da_dspacing.transpose(['pulse_time', 'spectrum', 'dspacing']).hist(\n", - " spectrum=500, dspacing=400\n", - " )\n", - ")" - ] - }, - { - "cell_type": "markdown", - "id": "61", - "metadata": {}, - "source": [ - "After conversion to $d$-spacing we may want to combine data from all spectra.\n", - "For dense data we would have used `da_dspacing.sum('spectrum')`.\n", - "For binned data this is not possible (since the events list in every spectrum have different lengths).\n", - "Instead we need to *concatenate* the lists from bins across spectra:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "62", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "da_dspacing_total = da_dspacing.bins.concat('spectrum')\n", - "da_dspacing_total.hist(dspacing=400, pulse_time=500).plot()" - ] - }, - { - "cell_type": "markdown", - "id": "63", - "metadata": {}, - "source": [ - "If we zoom in we can now understand the reason for the blurry diffraction lines observed at the very start of this tutorial:\n", - "The lines are not horizontal, i.e., $d$-spacing appears to depend on the pulse time.\n", - "Note that the effect depicted here was added artificially for the purpose of this tutorial and is likely much larger than what could be observed in practice from changes in sample environment parameters such as (pressure or temperature).\n", - "\n", - "Our data has three pulse-time bins (setup earlier for masking an area with low proton charge).\n", - "We can thus use slicing to compare the diffraction pattern at different times (used as a stand-in for a changing sample-environment parameter):" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "64", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "tmp = da_dspacing_total\n", - "lines = {}\n", - "lines['total'] = tmp.bins.concat('pulse_time')\n", - "for i in 0, 2:\n", - " lines[f'interval{i}'] = tmp['pulse_time', i]\n", - "sc.plot({k: line.hist(dspacing=1000) for k, line in lines.items()}, norm='log')" - ] - }, - { - "cell_type": "markdown", - "id": "65", - "metadata": {}, - "source": [ - "How can we extract thinner `pulse_time` slices?\n", - "We can use `sc.bin` with finer pulse-time binning, such that individual slices are thinner.\n", - "Instead of manually setting up a `dict` of slices we can use `sc.collapse`:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "66", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "pulse_time = sc.arange(\n", - " dim='pulse_time',\n", - " start=tmin.value,\n", - " stop=tmax.value,\n", - " step=(tmax.value - tmin.value) / 10,\n", - ")\n", - "split = da_dspacing_total.bin(pulse_time=pulse_time)\n", - "sc.plot(sc.collapse(split.hist(dspacing=1000), keep='dspacing'))" - ] - }, - { - "cell_type": "markdown", - "id": "67", - "metadata": {}, - "source": [ - "### Making a 1-D cut\n", - "\n", - "Instead of summing over all spectra we may want to group spectra based on a $2\\theta$ interval they fall into.\n", - "$2\\theta$ was computed earlier as a side effect of the conversion from time-of-flight to $d$-spacing:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "68", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "da_dspacing.coords['two_theta']" - ] - }, - { - "cell_type": "markdown", - "id": "69", - "metadata": {}, - "source": [ - "We can then define the boundaries we want to use for our \"cut\".\n", - "Here we use just a single bin in each of the three dimensions:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "70", - "metadata": { - "tags": [] - }, - "outputs": [], - "source": [ - "two_theta_cut = sc.linspace(dim='two_theta', unit='rad', start=0.4, stop=1.0, num=2)\n", - "# Do not use many bins, fewer is better for performance\n", - "dspacing_cut = sc.linspace(dim='dspacing', unit='Angstrom', start=0.0, stop=2.0, num=2)\n", - "pulse_time_cut = tmin + sc.to_unit(\n", - " sc.array(dims=['pulse_time'], unit='s', values=[0, 10 * 60]), 'ns'\n", - ")\n", - "\n", - "cut = da_dspacing.bin(\n", - " two_theta=two_theta_cut, dspacing=dspacing_cut, pulse_time=pulse_time_cut\n", - ")\n", - "cut" - ] - }, - { - "cell_type": "markdown", - "id": "71", - "metadata": {}, - "source": [ - "We can then use slicing (to remove unwanted dimensions) and `sc.histogram` to get the desired binning:" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "72", - "metadata": {}, - "outputs": [], - "source": [ - "cut = cut['pulse_time', 0] # squeeze pulse time (dim of length 1)\n", - "cut = cut['two_theta', 0] # squeeze two_theta (dim of length 1)\n", - "cut = sc.hist(cut, dspacing=1000)\n", - "cut.plot()" - ] - }, - { - "cell_type": "markdown", - "id": "73", - "metadata": {}, - "source": [ - "#### Exercise\n", - "\n", - "- Adjust the start and stop values in the cut edges above to adjust the \"thickness\" of the cut.\n", - "- Adjust the edges used for histogramming." - ] - }, - { - "cell_type": "markdown", - "id": "74", - "metadata": {}, - "source": [ - "### Making a 2-D cut\n", - "\n", - "#### Exercise\n", - "\n", - "- Adapt the code of the 1-D cut to create 100 `two_theta` bins.\n", - "- Make a 2-D plot (with `dspacing` and `two_theta` on the axes).\n", - "\n", - "#### Solution" - ] - }, - { - "cell_type": "code", - "execution_count": null, - "id": "75", - "metadata": { - "tags": [ - "solution" - ] - }, - "outputs": [], - "source": [ - "two_theta_cut = sc.linspace(dim='two_theta', unit='rad', start=0.4, stop=1.0, num=101)\n", - "dspacing_cut = sc.linspace(dim='dspacing', unit='Angstrom', start=0.0, stop=2.0, num=2)\n", - "pulse_time_cut = tmin + sc.array(dims=['pulse_time'], unit='s', values=[0, 10 * 60]).to(\n", - " unit='ns'\n", - ")\n", - "cut = da_dspacing.bin(\n", - " two_theta=two_theta_cut, dspacing=dspacing_cut, pulse_time=pulse_time_cut\n", - ")\n", - "cut = cut['pulse_time', 0] # squeeze pulse time (dim of length 1)\n", - "dspacing_edges = sc.linspace(\n", - " dim='dspacing', unit='Angstrom', start=0.0, stop=2.0, num=1000\n", - ")\n", - "cut = cut.hist(dspacing=dspacing_edges)\n", - "cut.plot()" - ] - } - ], - "metadata": { - "kernelspec": { - "display_name": "Python 3 (ipykernel)", - "language": "python", - "name": "python3" - }, - "language_info": { - "codemirror_mode": { - "name": "ipython", - "version": 3 - }, - "file_extension": ".py", - "mimetype": "text/x-python", - "name": "python", - "nbconvert_exporter": "python", - "pygments_lexer": "ipython3", - "version": "3.10.14" - } - }, - "nbformat": 4, - "nbformat_minor": 5 -} diff --git a/docs/tutorials/index.md b/docs/tutorials/index.md index aa7b4cb92..49c3b08a7 100644 --- a/docs/tutorials/index.md +++ b/docs/tutorials/index.md @@ -7,6 +7,5 @@ maxdepth: 1 1_exploring-data 2_working-with-masks -3_understanding-event-data powder-diffraction ```