Target programming languages

[bruceravel]

A minimal set of languages for which XDI support should be provided to assist in its wide-spread adoption includes: C, C++, Fortran, Java, Python, Perl, LabView, IDL. But all language are welcome.

[MariusVaarkamp]

Have the tools that are needed already been defined? I can help with the Fortran.

[newville]

Hi Marius,

Great to hear from you, and thanks for the offer. I had a Fortran version started but it's quite out-of-date with the current (still moving??) spec.

I think the C version is approximately ready for more elaborate testing at least for reading XDI files (writing is not yet implemented!), but I've been swamped with beamline work. Anyway, I think the aim is to make the C "the standard", but the Fortran version should be approximately equivalent (except that it won't return a single data structure)....

[MariusVaarkamp]

Hi Matt,

Let me know once you think the spec is sufficiently stable to start working on Fortran code for reading XDI files and what functionality is needed. I'll see what I can do.

Regards,
Marius
On Apr 16, 2012, at 1:16 AM, Matt Newville wrote:

Hi Marius,

Great to hear from you, and thanks for the offer. I had a Fortran version started but it's quite out-of-date with the current (still moving??) spec.

I think the C version is approximately ready for more elaborate testing at least for reading XDI files (writing is not yet implemented!), but I've been swamped with beamline work. Anyway, I think the aim is to make the C "the standard", but the Fortran version should be approximately equivalent (except that it won't return a single data structure)....

---

Reply to this email directly or view it on GitHub:
#16 (comment)

[newville]

Hi Marius,

I think it's ready, but I'll move this conversation to the mailing list....

[Yohko]

I just wanted to add this to IgorPro,
but what are the specs to read "nonxafs_1d.xdi" and "nonxafs_2d.xdi".
According to the specification these are no proper xdi files as some "must haves" are missing and I didn't find any information how to deal with "Outer.value" and "Outer.name".

[newville]

The changes to support these two files (from about a year ago) are fairly minor relaxations of XDI to support non-XAFS data, including 2D data as from a "mesh scan".

The current C code supports (or should!) these files, but it's reasonable to consider these non-compliant and ignore them as important parts of the format.

For an IgorPro support for XDI files, I would recommend linking with the C interface, using the dynamic library, as opposed to writing a reader in native IgorPro. That way you wouldn't have to worry much about which files were compliant.

[newville]

Just a short follow up:

the "nonxafs_1d.xdi" simply contains data with the independent data in the first column asserted to be something other than energy, say a scan of some other motor. It turns out that it actually is energy, but the extension allows there to be no column named "energy", as if you were reading some other form of data. I think this should be allowed -- the parsing and extraction of data from the file is still done with the same rules, but there isn't the implicit expectation of the data having to be XAFS data. This would allow beamlines to use this format more widely, and for applications consuming these files (say, to plot data) to support more types of data.

The 2-d data in "nonxafs_2d.xdi" is meant to show how one could store 2- (or even multi-) dimensional data, by inserting
#Outer.name XXX
to name the outer direction axis (and possibly axes) and then
#Outer.value XXX
is inserted in the data arrays to specify a change in that outer axis (or axes) -- the value is assumed to remain constant until otherwise noted, as if this were the "outer axis" of a 2-D scan. On reading the data, the values for the "outer data" are kept in the array "outer_array", and the indices at which these values change kept in "outer_breakpoints".

Though 2-d scans will commonly be a rectangular grid, the idea here is to support rows of non-uniform length. In fact the "nonxafs_2d.xdi" file does not have uniform "inner rows" -- sometimes there are 5 and sometimes there are 4 elements between steps of the outer axis.

The changes in the code to allow this extension is pretty small and non-intrusive to the XAFS portion of the code and spec. As with the non-XAFS 1d scans, whether these features need to be well-supported or should be removed is a separate question. It's slightly outside the main motivation for the format (to standardize a plaintext format for XAFS data), but it seems to me like it isn't a problem to have them.

If others think either of these changes is too far "out of scope" and should be removed, I'd be willing to do that.

[bruceravel]

@newville: I don't find it to be out of scope. XDI certainly could play a role for other measurements -- X-ray Emission Spectroscopy being one obvious example.

2D scans are, in some sense, at the edge of "scope". In our JSR paper, we were careful to say that large, multi-dimensional data sets are more appropriately encapsulated in something other than a plain text data file. That said, one could, for instance, imagine wanting to encapsulating a sequence (time, temperature, whatever) in a single plain text data file. In that case, one needs a way of specifying inner and outer axes.

So, I am fine with the concept of supporting 2D data.

That said, it doesn't seem like good practice to push an implementation of something that is clearly an extension of the work discussed and performed thus far to the HEAD of the master branch without having a discussion among the participants.

@Yohko: I hope that you will make a pull request once your the Igor interface is ready. We would like as many languages supported here as there are willing participants to write the code.

[newville]

Yes, I agree completely with all of this. I should not have put these changes in the master branch.

Supporting non-XAFS 1D scans seems like a fine "extension" or "relaxation". I'm +1 on this part.

For 2D scans, there are a few ways to sensibly put 2D data into a column file. The scheme I added is one option... possibly not even the best. It's also more clearly outside the scope. I'm willing to say that this should be either removed or left in but deprecated. I'm open to suggestions....

It's also reasonable to think about a 2D XDI-variant (properly versioned, such as XDI2D/1.0 or so) that does support this scheme-- but only if properly tagged.

[bruceravel]

Since this is an issue about language implementations, I'm going to go ahead and open an issue on 2D scans.