Failing to get BG from railway/nightscout

[rossens]

Describe the bug
OpenAPS rig no longer able to loop since migrating Nightscout from Heroku to Railway.

The rig is unable to get BG data from railway/nightscout (however it is able to send pump data to nightscout just fine).

/var/log/openaps/pump-loop.log: "Waiting up to 4 minutes for new BG.........." (but never gets BG)
/var/log/openaps/ns-loop.log: "Loading CGM data from Nightscout failed {"code":"CERT_HAS_EXPIRED"}

BG url: https://melaniejellis.up.railway.app/api/v1/entries/sgv.json?count=288&token=

Note: From Edison bash console I can get the data no problem using curl. Only seems to be an issue from node js
node js version: nodejs/unknown,now 8.17.0-1nodesource1 i386 [installed]

Note: railway uses "Lets Encrypt" for certification. Is node version too old to support "Lets Encrypt"? (I was reluctant to try upgrading node js in case it breaks something else.)

To Reproduce
Steps to reproduce the behavior:

1. Migrate nightscout from heroku to railway: https://nightscout.github.io/vendors/railway/migration/?fbclid=IwAR3PVqKdfU7rVO8n41LHfi83FgupH7ZO8rHvZs_ECIinnuYZZ5CJoKKaBRk
2. Check if rig is still looping

Expected behavior
Rig should continue looping after upgrading nightscout from heroku to railway

Smartphone:
model: Samsung S21
xdrip: 344deb4-2022.11.16

Setup Information:

• Pump type: Medtronic 722, v. 2.4A firmware
• CGM type: "xDrip-DexcomG6" with anubis transmitter
• Rig type: Edison/Explorer Board rig
• Linux version: 3.10.98-jubilinux-edison (robin@robin-i7) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #3 SMP PREEMPT Sun Aug 13 04:22:45 EDT 2017
• oref0 version: 0.7.1-dev (we updated from github to latest dev on 14/01/2023)

Additional context
I'm creating this bug report on behalf of @melaniejellis - who I've been helping get her openAPS rig looping again.
I'm not completely familiar with the openAPS system so apologies if this turns out this is not be a bug (but rather something we've missed in the setup.)

I implemented a temporary work around to get melanie looping again... which was to add the following lines to /usr/bin/oref0-get-ns-entries (https://github.com/openaps/oref0/blob/dev/bin/oref0-get-ns-entries.js). This allows the ns script to get the BG data, ignoring the certificate issue:

    headers["User-Agent"] = 'openaps';
    var uri = nsurl + '/api/v1/entries/sgv.json?count=' + records + tokenAuth;
    var options = {
      uri: uri
      , json: true
      , timeout: 90000
      , headers: headers
      , gzip : true
+     , agentOptions: {
+         rejectUnauthorized: false
+       }
    };

Look forward to hearing (if this is a real bug) what is the proper fix.

[scottleibrand]

According to ChatGPT, The main difference between certificate validation in curl and Node.js is that curl uses the operating system's root CA store, while Node.js uses its own copy of the root CA store, which is managed by the Node.js tls module.

Our working rigs are using Linux edison-eb 3.10.98-jubilinux-edison #3 SMP PREEMPT Sun Aug 13 04:22:45 EDT 2017 i686 GNU/Linux, which looks the same as yours. For node we have:

# nodejs -v
v8.17.0
# node -v
v9.11.2

Do you have a second Edison available that you could do a fresh flash and OpenAPS install on? I suspect that would fix the issue.

[sulkaharo]

Wonder what Node version is the latest that runs comfortably on the Edison. The chip is an x86 one, so theoretically even the latest LTS releases should work.

[melaniejellis]

Hi, thanks for your reply, I do have a few back ups. Ill do the flash and openAPS and report back.

[melaniejellis]

I've got :

nodejs -v

v8.17.0

node -v

v8.17.0

[sulkaharo]

Hmm I just tested loading the data and I don't think the issue is with Node. My browsers also fail to connect to the site and testing this with wget, the utility reports the certificate that's published doesn't match the domain name, which is a hard error and blocks SSL from connecting:

--2023-01-16 12:08:35--  https://melaniejellis.up.railway.app/
Resolving melaniejellis.up.railway.app (melaniejellis.up.railway.app)... 52.166.12.23
Connecting to melaniejellis.up.railway.app (melaniejellis.up.railway.app)|52.166.12.23|:443... connected.
ERROR: no certificate subject alternative name matches
	requested host name ‘melaniejellis.up.railway.app’.
To connect to melaniejellis.up.railway.app insecurely, use `--no-check-certificate'.

[rossens]

Hmm yeah agreed something strange is going on then.
I just checked myself accessing https://melaniejellis.up.railway.app/ from chrome (windows 10) and had no issues.
Also tried wget on a linux box, which also worked:

$ wget https://melaniejellis.up.railway.app
--2023-01-16 18:35:48--  https://melaniejellis.up.railway.app/
Resolving melaniejellis.up.railway.app (melaniejellis.up.railway.app)... 104.196.232.237
Connecting to melaniejellis.up.railway.app (melaniejellis.up.railway.app)|104.196.232.237|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 42973 (42K) [text/html]
Saving to: ‘index.html’

index.html          100%[===================>]  41.97K   263KB/s    in 0.2s

2023-01-16 18:35:49 (263 KB/s) - ‘index.html’ saved [42973/42973]

Notice a different ip address though... wondering if that changed over night (legit), or there is something odd going on with 2 versions of the same site or something?

[sulkaharo]

Right, so looks like Railway.app is very widely blacklisted in a variety of software intended for cybersecurity purposes. So if you're having issues loading data from Railway, please check the network you're using is not blocking Railway. I just discovered the corporate security solution my work has in place was blocking 100% of Railway.

[rossens]

Same issue on a different rig:

root@meleAPS:~# node /usr/bin/oref0-get-ns-entries ns-glucose.json https://melaniejellis.up.railway.app xxx

Connecting to 192.168.1.1, testing for xDrip API availability
Load from local xDrip timed out, likely not connected to xDrip hotspot
{"errno":-2,"code":"ENOENT","syscall":"open","path":"/root/ns-glucose.json"}
Loading CGM data from Nightscout failed {"code":"CERT_HAS_EXPIRED"}

Whereas curl, wget etc work fine.
Versions:

root@meleAPS:~# node -v
v8.17.0
root@meleAPS:~# nodejs -v
v8.17.0
root@meleAPS:~# oref0-version
0.7.0 [dev]
root@meleAPS:~# cd /root/src/oref0/ && git status
On branch dev
Your branch is up-to-date with 'origin/dev'.
nothing to commit, working tree clean

Note: I attempted re-flashing the rig as well - but failed miserably. The flashall.bat never detected the edison USB OTG device. (I noticed the OTG seemed flaky - it would constantly connect/disconnect in device manager every few seconds).

[rossens]

Update on this issue:
I use a simpler script to rule out as much other noise as possible:

#!/usr/bin/env node
'use strict';

const https = require('https');
https.get('https://melaniejellis.up.railway.app/api/v1/entries/sgv.json?count=1&token=xxx', (resp) => {
  let data = '';
  resp.on('data', (chunk) => { data += chunk; });
  resp.on('end', () => { console.log("Success: " + data); });

}).on("error", (err) => { console.log("Error: " + err.message); });

I tested this works on Ubuntu 22.04 (node v12.22.9)
Success: [{"_id":"...<trimmed>...}]
I tested this does not work on Edison openAPS rig (node v8.17.0)
Error: certificate has expired
I also tried updating node on Edison to "latest" (v9.11.2) - but this version also not working.

I had just about run out of ideas when I found this: node-fetch/node-fetch#568
Apparently these guys had encountered the identical looking issue with their node-fetch module vs Lets Encrypt certificates, for one user it was even also on a Heroku app.
They used the same temporary workaround as I suggested (rejectUnauthorized: false), and have since made a proper fix: "They merged https://github.com/electron/electron/pull/31218/files" which patches boringssl

I'm not yet sure if/how this same fix would be applicable in oref0 (does it use boring ssl?) - but passing on my find in the hope you guys may know.
Any thoughts?

[sulkaharo]

Sounds like the easy quick to deploy fix is to not host Nightscout in a service that uses Let's Encrypt certificates. Edisons haven't been supported for years in terms of software (by vendors) and if we can't get a newer Node version to run, there's relatively little we can do. Disabling the certificate checks is technically possible but sounds like something that shouldn't be done in the mainline version of openaps.

[rossens]

Could another option be to avoid node for the https work? e.g. I tried this which works:

#!/usr/bin/env node

const { exec } = require('child_process');
exec('curl https://melaniejellis.up.railway.app/api/v1/entries/sgv.json?count=1&token=xxx',
(error, stdout, stderr) => {
  console.log(stdout);
  if (error !== null) {
      console.log(`exec error: ${error}`);
  }
});

Interestingly when I first tried this it failed with the same certificate error - and I realised both curl & wget had the same issue as node. But this was an older Edison which hadn't been apt upgraded in a while. After an apt upgrade - curl & wget were both fixed (but node still broken).