-
Notifications
You must be signed in to change notification settings - Fork 5
Home
This page is a Work-in-Progress Proposal for how to build a ''Next Generation XAFS Data Library'' The main idea is to allow XAFS Data Library that
- facilitate the exchange XAFS spectra, particularly on model compounds, such as found in Model Compound Libraries.
- store and manage multiple spectra, but still easily export to plain ASCII files.
- allow users to have public and private libraries of spectra.
- make it easy to share sets of XAFS spectra.
To date, there have been several Web-based Databases. While these all organize XAFS spectra on model compounds, they all have shortcomings such as incomplete data, difficulty of adding new data, and incompatible formats.
The aim here is to create a way to store multiple XAFS spectra in a manner that can be used within dedicated applications and embedded into existing data processing software with minimal effort. This will also alleviate many of the problems associated with the current databases.
Storing and Exchanging XAFS Data is a common need for everyone using XAFS. In particular, retrieving data on ''Standards'' or ''Model Compounds'' is a continuing need for both XANES and EXAFS analysis. Additionally, data taken on samples at different facilities and beamlines need to be compared and analyzed together. At this point, there is no commonly accepted data format for XAFS data. There have been a few attempts to standardize the traditional "ASCII Column File". To be sure, ASCII Column Files have some strong appeal. Most data collection software saves such files, and most processing and analysis software use some variation of this format. In addition, such files may be easily read by humans and used in a wide variety of third party applications. Still, most ASCII Column Files need some intimate knowledge of the data layout to use the data. The lack of a standard ASCII format is a serious problem. The work here is parallel to the efforts to come up with a standard. and will be able to convert data into such standard file formats.
For the effort here, in which storing multiple spectra is a key requirement, ASCII Column Data Files offer no standard way to hold multiple spectra. In order to facilitate the sharing data between beamlines and the exchange (and recognition) of high-quality data on standards, it is useful to have Data Libraries or Repositories which a scientist can use at will and exchange with others. A key point is that not only should data be well-formatted and vetted for quality, but should also be easily be viewed as a part of Suite of Spectra.
A Data Library is often envisioned as a single centralized Library (see http://xafs.org/Databases) which is meant to house ''Standard Data'' with some assurance of quality and the expectation that the data is to be shared with the entire community. In contrast, many researchers keep their own set of data closely guarded and do not want to share their data. Another model for a Data Library is a set of spectra being shared between collaborators, but then possibly allowed to have a wider distribution (say, after some paper has been published). We are considering all of these as legitimate use cases, and are more interested here in creating a format that can be used for all these cases.
With the motivation from the previous section, the goals for the format and library are:
- store spectra for exchange, especially for model compounds. Raw data, direct from the beamline will probably need to be converted to this format.
- store information about the sample, measurement conditions, etc.
- store multiple spectra, either on the same sample or multiple samples, and possibly taken at many facilities.
- provide programming libraries and simple standalone applications that can read, write, and manage such data libraries. Programming libraries would have to support multiple languages.
There are a few reasonable ways to solve this problem. What follows below is a methods which makes heavy use of ''relational databases'' and SQL. The principle argument here is that relational databases offer a well-understood, proven way to store data with extensible meta-data. The use of SQL also makes the programming libraries simpler, as they can rely on tested SQL syntax to access the underlying data store.
As the XAS Data Library is being developed, code and examples will be available at https://github.com/XraySpectroscopy/XASDataLibrary
I propose using SQLite, a widely used, Free relational database engine as the primary store for the XAFS Data Library. A key feature of SQLite is that it needs no external server or configuration -- the database is contained in a single disk file. SQLite databases can accessed with a variety of tools, for example DB Browser for SQLite and SQLite Manager addon for Firefox.
SQL-based relational databases may not be the most obvious choice for storing scientific data composing of arrays of related data. One obvious limitation is that relational databases don't store array data very well. Thus storing array data in a portable way within the confines of an SQL database needs special attention. The approach adopted here is to JSON, which can encapsulate an array, or other complex data structure into a string.
JSON -- Javascript Object Notation -- provides a standard, easy-to-use method for encapsulating complex data structures into strings that can be parsed and used by a large number of programming languages as the original data. In this respect, the requirements for the XAS Data Library -- numerical arrays of data -- are fairly modest. Storing array data in strings is, of course, what ASCII Column Files have done for years, only not with the benefit of a standard programming interface to read them. As an example, an array of data [8000, 8001.0 , 8002.0] would be encoded in JSON as
'[8000, 8001.0, 8002.0]'
This is considerably easier and lighter weight than using XML to encode array data.
In addition to encoding numerical arrays, JSON can also encode an associative array (also known as a Hash Table, Dictionary, Record, or Key/Value List. This can be a very useful construct for storing attribute information. It might be tempting to use such Associative Arrays for many pieces of data inside the database, this would prevent those data from being used in SQL SELECT and other statements: such data would not be available for making relations. But, as Associative Arrays can so useful and extensible, several of the tables in the database include a attributes column that is always stored as text. This data will be expected to hold a JSON-encoded Associative Array that may be useful to complement the corresponding notes column. This data cannot be used directly in searching the database, but may be useful to particular applications.
While robust, powerful and compliant with SQL standards, SQLite does not always provide as rich set of Data Types as some SQL relational databases. In particular for the design here, SQLite does not support Boolean values or Enum fields. Integer Values are used in place of Boolean Values. Enum values (which may have been used to encode Elements, Collection Modes, etc) are implemented as indexes into foreign tables, and JOINs must be used to relate the data in the tables.
The principle kind of data held in a XAFS Data Library is XAFS Spectra. Additionally, several other kinds of data can be useful to include, such as for sample preparation, measurement conditions, and so on. Of course, a key feature of a multi-spectral database is to be able to combine several spectra into a Suite of Spectra, and also to identify the people adding data to a library. Thus the XAFS Data Library contains the following main tables:
| Table Name | Description |
|---|---|
| spectra | main XAS spectra, pointers to other tables |
| sample | Samples |
| crystal_structure | Crystal structures |
| person | People |
| citation | Literature or Other Citations |
| format | Data Formats |
| suite | Spectra Suites |
| facility | Facilities |
| beamline | Beamlines |
| monochromator | Monochromators |
| mode | Modes of Data Collection |
| ligand | Ligands |
| element | names of Elements |
| edge | names of x-ray Edges |
| energy_units | units for energies stored for a spectra |
| info | General Information, version etc |
While some of these tables (spectra, sample) are fairly complex, many of the tables are really quite simple, holding a few pieces of information.
In addition there are a few Join Tables to tie together information and allow Many-to-One and Many-to-Many' relations. These tables include
| Table Name | Description |
|---|---|
| spectra_mode | mode(s) used for a particular spectra |
| spectra_ligand | ligand(s) present in a particular spectra |
| spectra_suite | spectra contained in a suite |
| spectra_rating | People's comments and scores for Spectra |
| suite_rating | People's comments and scores for Suites |
A key feature of this layout is that a Suite is very light-weight, simply comprising lists of spectra. Multiple suites can contain an individual spectra, and a particular spectra can be contained in multiple suites without repeated data.
The tables are described in more detail below. While many are straightforward, a few tables may need further explanation.
This is the principle table for the entire database, and needs extensive discussion. Several of the thorniest issues have to be dealt with in this table, making this likely to be the place where most attention and discussion should probably be focused.
--
create table spectra (
id integer primary key
name text not null,
notes text,
attributes text,
file_link text,
data_energy text,
data_i0 text default '[1.0]',
data_itrans text default '[1.0]',
data_iemit text default '[1.0]',
data_irefer text default '[1.0]',
data_dtime_corr text default '[1.0]',
calc_mu_trans text default '-log(itrans/i0)',
calc_mu_emit text default '(iemit*dtime_corr/i0)',
calc_mu_refer text default '-log(irefer/itrans)',
notes_i0 text,
notes_itrans text,
notes_iemit text,
notes_irefer text,
temperature text,
submission_date datetime,
collection_date datetime,
reference_used integer,
energy_units_id -- > energy_units table
monochromator_id -- > monochromator table
person_id -- > person table
edge_id -- > edge table
element_z -- > element table
sample_id -- > sample table
beamline_id -- > beamline table
format_id -- > format table
citation_id -- > citatione table
reference_id -- > sample table (for sample used as reference
);We'll discuss the table entries more by grouping several of them together. First, Each entry in the spectra table contains links to many other tables.
| spectra Column Name | Description |
|---|---|
| energy_units_id | index of energy_units table |
| person_id | index of person table for person donating spectra |
| edge_id | index of edge table for X-ray Edge |
| element_z | index of element table for absorbing element |
| sample_id | index of sample table, describing the sample |
| reference_id | index of sample table, describing the reference sample |
| beamline_id | index of the beamline where data was collected |
| monochromator_id | index of the monochromator table for mono used |
| format_id | index of the format table for data format used |
| citation_id | index of the citation table for literature citation |
Next, the table contains ancillary information (you may ask why some of these are explicit while others are allowed to be put in the attributes field).
| spectra Column Name | Description |
|---|---|
| notes | any notes on data |
| attributes | JSON-encoded hash table of extra attributes |
| temperature | Sample temperature during measurement |
| submission_date | date of submission |
| reference_used | Boolean (0=False, 1=True) of whether a Reference was used |
| file_link | link to external file |
Here, reference_used* means whether data was also measured in the reference channel for additional energy calibration . If 1 (True), the reference sample must be given. The *file_link entry would be the file and path name for an external file. This must be relative to the directory containing database file itself, and cannot be an absolute path. It may be possible to include URLs, ....
Finally, we have the information for internally stored data arrays themselves
| spectra Column Name | Description | Default |
|---|---|---|
| data_energy | JSON data for energy | -- |
| data_i0 | JSON data for I_0 (Monitor) | 1.0 |
| data_itrans | JSON data for I_transmission (I_1) | 1.0 |
| data_iemit | JSON data for I_emisssion (fluorescence, electron yield) | 1.0 |
| data_irefer | JSON data for I_trans for reference channel | 1.0 |
| data_dtime_corr | JSON data for Multiplicative Deadtime Correction for I_emit | 1.0 |
| calc_mu_trans | calculation for mu_transmission | -log(dat_itrans/dat_i0) |
| calc_mu_emit | calculation for mu_emission | dat_iemit * dat_dtime_corr / dat_i0 |
| calc_mu_refer | calculation for mu_reference | -log(dat_irefer/dat_itrans) |
| calc_energy_ev | calculation to convert energy to eV | None |
| notes_energy | notes on energy | |
| notes_i0 | notes on dat_i0 | |
| notes_itrans | notes on dat_itrans | |
| notes_iemit | notes on dat_iemit | |
| notes_irefer | notes on dat_irefer |
The '''data_*****''' entries will be JSON encoded strings of the array data. The calculations will be covered in more detail below. Note that the '''''spectra_mode''''' table below will be used to determine in which modes the data is recorded.
As alluded to above, the data*****_ will be stored as JSON-encoded strings.
The calculations of mu in the various modes (transmission, fluorescence) are generally well defined, but it is possible to override them by explicitly documenting the expression used to calculate mu. Note that this expression should not be expected to be fully and correctly evaluated by the database -- it is meant for human reading.