XISF header length not defined in reference document

[han-k59]

I have created an XISF reader for the ASTAP program written in Object Pascal. I noted that there is no pre-defined definition how long the header is or maybe I missed it. In practice for uncompressed files it is $1000 equals 4096 bytes. After the XML part the file is filled with unused zero's De attached data block starts then at $1000 or probably later if the header is larger. There is no unused space if the file contains a compressed datablock.

For efficient file reading and to avoid file "seek" is would be convenient to define an initial length of the header to read the first information. So the file reader knows how many bytes to read initially.

I haven't studied the original code. Maybe it reads all data in one array at once, but the current start position for data at $1000 indicates it is not.

[jconejero]

Hello,

In first place, thank you so much for supporting XISF in your application. The header length is defined at the beginning of each monolithic XISF file. See section 9.1.1 Monolithic XISF Unit in the specification, and more specifically section 9.2 Monolithic XISF File, as well as Figure 8, where you'll find a graphical description of the monolithic XISF file structure. The header length field stores the length in bytes of the XML header as an unsigned 32-bit integer encoded in little-endian byte order, and is located after the 8-byte file signature at the beginning of each .xisf file.

As for unused space blocks, they are optional in the XISF specification (again, see section 9.2 and Figure 8), so your reader must not rely on their existence. For example, the current XISF implementation in PixInsight generates unused blocks by default for uncompressed files (although the user may disable them), but does not generate them for compressed files, where the assumption is that saving storage space is the main priority, instead of optimizing file accesses.

Please let me know if I can help you further with your implementation.

[han-k59]

Hello Juan,

Thanks for the explanation. The XISF reader implementation is now improved and works for all sample formats and uncompressed XISF files.

In my reader the underlying TFilestream reads the file in blocks of 4096 bytes, Above this Tfilestream there is a TReader doing the following steps:

1. Read signature (length 16 bytes)

2. Read header (length as defined in signature)

3. Read unused space (length as calculated)

4. Read image line 1 (length from header info)

5. Read image line 2

6. . . . .

7. Read last line of image

In this concept there is no need for 3) "unused space". Theoretical is causes some delay.

The only thing I'm missing at Github is the XISF documentation in one file e.q. pdf rather then .pidoc. But it is available at PixInsight webpage as html.

Kind regards,
Han

[jconejero]

Hi Han,

Glad to know you are making progress. However, from your description it seems that you are simply skipping the XISF header, but not actually using it. If this is the case (please correct me if I am wrong), your implementation cannot be a valid XISF reader.

Actually, you should load and parse the header's XML source code because it describes how the images are (image geometries, color spaces, properties, etc) and where they are stored in the .xisf file. Otherwise you are making risky assumptions based on a particular implementation working with very specific parameters. This may work with a particular set of .xisf files, but is not going to work as a general XISF reader.

My recommendation is that you use our official PCL implementation to support XISF in your application. I assume that Object Pascal is able to interface with C++. If this is not possible, you can write a shared object in C++ (a .dll, .so or .dylib component, respectively for Windows, Linux and macOS) which you can invoke from your application. The PCL implementation provides everything you need to work with XISF in a completely portable way and is very easy to use.

[han-k59]

Hello Juan,

No worry, the XISF header is read for the data dimensions, color space and so on. The FITS keyword information is retrieved. I just tried to describe the reader steps and that the unused space in the file seems unnecessary for my concept.

The only things not supported now seems decompression, thumbnails and an External XISF Data Block. Maybe in future.

Object Pascal can interface with C++ but my program is created for Windows, Linux, OSX and Armhf and I have preference to have one master source code for all rather then supplying it with an external ''component''.

My source code is available at SourceForge.