Improve Oscillators on RPi #6
Description
Spawned from YT comment by request.
Forgive me if this rapidly descends into rambling...
A suggestion I had was using better oscillators on the NTP host to reduce tracking errors. The commodity crystals that feed the various clock trees, can induce PLL drift with their respective peripherals. By using more expensive oscillators, and subsequently disciplining them to a GPS derived long-period-stable source (the cesium standards in the GPS satellites).
The original oscillators are "fine" if what you're doing is not precise timing (IMHO). The frequency response can change due to voltage and temperature fluctuations. A first lay improvement can be had by purchasing an oscillator that has tighter tolerances. This can be improved by obtaining an oscillator that is compensation for temperature changes. This can be further improved by using both a macro temperature compensation, and by comparing its output to a "known good" timing source and introducing corrections to the clock.
Crystals can be replaced with an active oscillator fairly easily - a crystal circuit is effectively a tuned delay loop. An output (XO) drives a signal into one end of a crystal and it propagates through that physical component eventually emerging at the other side and being fed into the input side (XI) of the crystal circuit of the IC. An oscillator does not require a start pulse to delay, but the output can still be fed into the XI port of an oscillator circuit. Thus dropping in an oscillator can be fairly pain-free as long as the voltages are similar.
By removing the factory crystal and injecting the output of any clock source at XI (function generator, oscillator, etc...) one can begin to improve on the factory crystal characteristics.
A few years back, I had a timing company Connor-Winfield build me some custom oscillators for a car audio project. This resulted in the three time-domains getting TB604 clocks, one at 54MHz for the Broadcom core, 25MHz for the Ethernet NIC, and 24.567MHz for the audio domain (not included on the CM4 module I was working with, but used by the AKM protocol converter and DACs). It was overkill. What I did was I used a Ublox ZED-F9P GPS receiver to generate a 10MHz clock which ran a few PIC processors as well as providing GPS location data for the vehicle. The power domain of the GPS receiver and oscillators was separate from the RPi, meaning this was allowed to run even when the RPi was powered down (clock buffers were switched off with the RPi). One of the things I learned here was that GPS 1PPS is not reliable if it loses lock, and so if the clocks drift, from the GPS cesium standard while they are out of view, the drift can be pretty awful.
The PIC micros had a free-running asynchronous counter fed from the three separate time domain oscillators - the code would latch the count for these clocks every second (1PPS), and add it to a 86400 second window-box average LUT. Over time this correction would be feathered in using a "normal distribution" (Monte-Carlo) over the subsequent 24 hours. Some things you have to look out for are:
- GPS SAW-tooth timing error causing short term jitter (OMG, leap seconds are hard to deal with short term if the circuit is naive)
- Correction jumps too large unlocking your PLLs (this is why you feather it in)
- Position error. Since GPS uses the phase difference between cesium time standards to derive both a local clock "truth" and a distance from the ephemeral position of the satellite, through the ionosphere - having to solve for both location and time means you get error in both.
This taught me that a bit of a buffer needed to happen between the GPS receiver and the system clock domain to cover for some of the issue with GPS timing sources, and for higher precision and accuracy I needed a fixed and very accurate location.
For a home build Master Clock I'm in the process of (not RPi based, rather a Lenovo mini-micro 1Liter PC), I'm starting with the 10MHz and 1PPS source. This is the "Known Good" timing source question. GPS receivers work on a different timing domain than 10.00MHz (and interestingly if you search google for "Ubox ZED-F9T Internal Clock Rate" my support forum question from 2020 is the third hit, with no better info provided), even the signals GPS transmits/receives are 10.23million chips/second - the numbers provided by clive in the support thread and the chip rate are base-2-like clock values. What this means is that if you are trying to get a clock value like 10MHz (a decimal value) from say a 48MHz clock it won't divide evenly and will result in some quantization error. Interestingly, the ZED-F9T does feature a quantization register for the 1PPS output that is claimed to be good down to picoseconds... reasonable, one can derive a "GPS solution" and observe its skew from the host-platform's clock. Knowing the edge of the IO clock domain that the 1PPS signal will be output, a relative quantization error could be calculated to a fairly high degree of accuracy for the different of when it would have preferred to output the 1PPS versus when it was actually output due to the limitations of the IO clock domain.
So at that point a half decade ago, frustrated, I realized I'd have to find a better local timing domain. You show off your CSAC... I'm a bit jealous for time nerd reasons ;-). I had talked to Microsemi a while back (before Microchip bought them up) and got close to getting my hands on a raw unmounted CSAC SA.45 module (now retired and replaced with the SA65). But I started comparing other devices and off the shelf options out there, and found Stanford Research Systems' PRS10 online (used) for only a few hundred bucks (as of this writing, I see one for about the same I paid on e-place). The CSAC and PRS10 basically work the same way, and cost about the same. If you don't need it to be rugged (I don't it's going in a shock mounted rack and will be fixed in place), it's basically a toss-up which one to get. I bought one and got used to what it could do, it was well documented, and support was pretty forthcoming even as an aftermarket owner. But with aftermarket you get "what's available", not necessarily "what you want". I ended up spec'ing and purchasing a new PRS10 this past summer - brand new, no age on the rubidium lamp, got the extremely-low-aging option, and the 2-year factory warranty.
So this takes care of the buffer between the GPS and the local clocks.
To address the location issue, the Ublox ZED-F9T/P receivers allow for a fixed position mode, where a location can be input, thereby removing one of the variables of the GPS timing solution. Sparkfun did a blog article about the RTK uses of the ZED-F9 family, which talked about the setup of an RTK base-station (the fixed side of <=1cm GPS mobile applications) - a useful side effect of having one of these receivers - you may wish to look into this for future projects. For that they gave instructions on how to get a good corrected location by recording RAW GPS data and sending it off to a correction service for tuning your location - I'd recommend 168 hours of RAW sampling, they only recommend 24. 168hours is a whole week, so any variation in radio interference, and heat etc... should be fairly well averaged out.
A solid, can't move/shake/vibrate GPS antenna mount is really critical. When USGS and others place a GNSS reference antenna for ground movement measurement (earthquake early warning, volcano eruption detection - but also allows for a good GPS receiver to discipline a local oscillator for wide area time-aligned sensory uses.), they usually put it on a pole, then a 3 or 4 legged tripod, set in concrete about 10feet deep so that nothing but the ground movement can be the cause of the antenna position. I recommend at least looking at high wind satellite dish mounts for the GPS antenna - this is like the standard J-pipe 18" DSS dish mount, but with two extra legs for added stability. Attach this to a shear wall on the building to reduce movement/vibration.
The GPS antenna you showed was cool (like what you'd see on any token cellular site) - but for some antenna pron take a look at the Topcon PN-A5 to blow your mind, it's like a ground-reflection-eliminating choke ring GNSS antenna, but with a hemispherical studded choke so that you can still "see" satellites closer to the horizon.
This gets you to a GPS disciplined local oscillator. Useful. By using this solid time-base, you can discipline your host's now improved oscillators, and keep them tight to the Rubidium standard.
If you want to feed to NTP, there are a few things to keep in mind:
- 1PPS is edge-time critical. USB introduces jitter, use the native serial UART and a physical pin for the 1PPS interrupt to arrive in a deterministic manner. Run Linux with the pre-empt-RT kernel option to ensure that the critical path code and interrupts don't get blocked.
- Other host tasks can interfere with timing - lock the gpsd process to its own core on the RPi, keep everything else off it!!
- Speaking of things blocking, you can use more than one interface on the Ublox receivers, so use USB or I2C for GPS module configuration, and leave the fewest messages at the highest data-rate for the time data to gpsd
You already have a great article about hooking up the IEEE-1588 sync pin to 1PPS on CM4 to enable PTP - I don't need to explain anything to you there.
BTW, this project seems ripe for a recommendation of a mailing list you may wish to go down the rabbit hole on: https://groups.io/g/time-nuts In my experience these people are of a wide variety of experience, and frequently generous with their time (ha! time puns).