Assisting

[larsnaesbye]

Hi, I might want to contribute to the development of rinex, as my day to day work is in need of better tools than tecq, Anubis, gfzrnx etc. I have experience with RINEX, and though fairly new to Rust, I'm already fascinated by the tooling and community.

Are there some smaller tasks you'd like to be worked on first, like contributing test data, hardening or fixing the epochs?

[gwbres]

Hello @larsnaesbye, welcome onboard!

At the moment it is fair to say that the parser is the most stable part of the project.
Navigation, Observation, Crinex (compressed Obs) and Meteo records are very stable (up to V3). I just marked Antex and Clocks as stable in the last few days. You can refer to the entire /test_resources database, which is used for automated test purposes (for information, there are about 40 tests run automatically at the moment, mostly about the parser).

The rinex directory is a rust library, which everyone can include and take advantage of, the official documentation can be found here. It is shipped with a couple of examples, you can browse their codes in rinex/examples directory, I should get back to enhance them and provide more examples (clocks, antex..).
For example, you can run the observation example with:

cargo run --example observation

I am most focused on the parser (file reader) at the moment as opposed to

• the command line tools: which you are probably more interested in
• and the file writer (file production) which you might be interested in

I am currently concluding support for IonosphereMaps and Troposphere data, then I will get back to the command line tool.

Regarding the command line tools, rinex-cli is an application, written in rust. That means it builds easily with "cargo" but it is supposed to be run in a shell. It uses the main library (but hidden to the user), so the user can easily take advantage of it. The command line application is somewhat capable of analyzing OBS/NAV/MET. It is not capable of plotting at the moment, I will try to provide that in a near future. There is a lot that can still be achieved with this tool. If you want to give it a try, you can do something like this:

cd rinex-cli
cargo run -- --help 
# print available obscodes in this RINEX
cargo run -- -fp ../test_resources/OBS/V3/DUTH0630.22O --obscodes --pretty
# extract data for Sv "G01" but only for epochs with an `ok` flag attached to them:
cargo run -- -fp ../test_resources/OBS/V3/DUTH0630.22O -e --sv G01 --pretty

As for the teqc-like operations, which you are interested in, there are three things we can say:

• the action takes place at the Rinex structure level, see split_record_at_epoch() and neighboring methods,
  like merge().
• they are not too complex, but very verbose.. because the core differs for every record type (hidden to user).
  I barely spent any time on these methods at the moment. I would say, the best thing to do is to focus on the merge() ops, and test it properly in rinex/tests/merge.rs. I am also looking forward to providing a down_sample() method.
• once we stabilize such methods, it can easily be exposed in the command line tool to create a probably similar (and possibly more advanced) teqc operation

Overall, and I will conclude with this:

• I started using RINEX this year and I still have doubts / questions about things.
  I also struggle to find any test data, I don't have a single V4 for instance.
  If we can discuss things in the future, just that would be already valuable help. My job involves timing and related measurements (hence the ClockData support), I am looking forward to providing proper Time System conversions and related calculations
• the record manipulation is the most complex thing in this project. First of all because the record type depends on the type of file, so they are all different. Then, because it involves iterator. Iteration is very efficient and powerful, it provides generic filter and sorting methods (see the example codes), but it is also very verbose and hard to grasp in my opinion.
• for example, OBS/NAV/MET are all iterated on an epoch basis, a merge operation will obviously involve epoch iterations
• So far I have used 3 structures to declare Data Arrays:
  • Vec : standard Vector indexed by an integer, they are not sorted! order is unpredictable
  • Hash: Vector but indexed by an object (which must implement == comparison). I used Space vehicule indexing a lot, which sorts data efficiently
  • BTreeMap : same as Hashmap but data is ordered! (must implement Ord). I use BTreeMaps to have the record chronologically sorted on an epoch basis

Now that is already a lot of information. Let me know if you want to know more.

[larsnaesbye]

Hello @gwbres,

Thank you for the very descriptive welcome! There is absolutely everything I would like to know in there for now.

You are right, the file production and command-line tools are what I ultimately would like to use for daily operations. I oversee a system that produces RINEX files from raw receiver data and distributes them freely for public consumption.

In short, the tasks are:

• Convert raw binary from receiver to RINEX - obviously not in scope for this project
• Pack and unpack gzip
• Compress and decompress Hatanaka
• Rewrite RINEX headers
• Splice 1-hour files into 24-hour files
• Decimate 1-sec files into 30-sec files
• Gap analysis
• QC on the 30-sec files

For this we use a combination of gfzrnx, bnc, Anubis and CRX2RNX/RNX2CRX (the one by Hatanaka). If these tasks could in the future be done using the same framework, it would make everything easier and more transparent.

I think I can pretty easily get some RINEX v4 files, since the RAW to RINEX software supports it. Let me get back to you on that.

After that I might look into the rinex-cli, if that's OK with you.

[gwbres]

Hello @larsnaesbye ,

I oversee a system that produces RINEX files from raw receiver data and distributes them freely for public consumption

I initiated a "ublox-rinex" application, it is also contained in this folder, and it serves that role as long as you intend to use "ublox" receivers. It basically combines the "ublox" crate (very powerful! I use it at work on a daily basis), and the "rinex" crate (which already knows the most common RINEXes), that could be of interest for you. I barely spent any time on this application. We need to conclude the Rinex::to_file() method to make this application happen. Which is not too painful and should not take long. After that, we can combine the best of both worlds. If you intend to use different receivers, obviously that will not help.

Convert raw binary from receiver to RINEX - obviously not in scope for this project

The rinex-ublox app does that from a ublox receiver. The official ublox crate lacks some Ublox frames, therefore I have the application point to my fork (timing, and other ubx frames)

Pack and unpack gzip

I just added the hability to parse .gz files directly. I am improving the BufferedReader (most efficient way to browse the file) in the sense it is already capable to uncompress .gzip in a hidden fashion and soon Hatanaka too (makes things much easier).
If we start tackling the to_file() side of things, we would need the mirror operations, BufferedWritter for efficient file production and smooth operations.

Compress and decompress Hatanaka

Supported natively, will be managed by the BufferedReader directly in my next commits.
Note I have not completed the Hatanaka compression (mirror operation) in the Hatanaka core structures! needs to be finalized to have an efficient to_file()

Rewrite RINEX headers

Rinex::to_file() is partial, it needs to be improved, support a buffered stream efficient approach, support all known attributes and record types..

Splice 1-hour files into 24-hour files

I have not initiate the Rinex::splice operation, because I do not know what splicing means

Decimate 1-sec files into 30-sec files

Decimate/down sampling is under development but is closer to stable than all other topics.
See Rinex::resample(std::time::Duration) to rework a sampling interval. Or Rinex::Decimate(N) to reduce the epoch rate by N

Gap analysis

See Rinex::dead_times() for gap analysis, also Rinex::anomalies() for epoch flags analysis (flags only truly relevant in OBS files)

[larsnaesbye]

I initiated a "ublox-rinex" application, it is also contained in this folder, and it serves that role as long as you intend to use "ublox" receivers. It basically combines the "ublox" crate (very powerful! I use it at work on a daily basis), and the "rinex" crate (which already knows the most common RINEXes), that could be of interest for you. I barely spent any time on this application. We need to conclude the Rinex::to_file() method to make this application happen. Which is not too painful and should not take long. After that, we can combine the best of both worlds. If you intend to use different receivers, obviously that will not help.

Yeah, we use and have used other types of receiver (Leica, Trimble, Septentrio...), not u-blox.

I have not initiate the Rinex::splice operation, because I do not know what splicing means

I think splice and merge is the same thing - combining two files from the same receiver to get one continuous. A bit like they describe on the tecq mailing list here: https://postal.unavco.org/pipermail/teqc/2016/002144.html

I have submitted a pull request with a RINEX 4 file (Hatanaka 3 compressed). I hope it can be of use for testing :-)