v2.1.3 Gremsy Vio camera payload fails to download camera definition due to "unsupported protocol" error

[rayw-dronesense]

2024-11-26 12:38:58.851 20122-20563 Mavsdk                  com.dronesense.pilot.blue            I  Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-11-26 12:38:58.852 20122-20563 Mavsdk                  com.dronesense.pilot.blue            I  Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-11-26 12:38:58.853 20122-20563 Mavsdk                  com.dronesense.pilot.blue            E  Error while downloading text, curl error code: Unsupported protocol
2024-11-26 12:38:58.853 20122-20563 Mavsdk                  com.dronesense.pilot.blue            E  Failed to download camera definition.
2024-11-26 12:38:58.854 20122-20563 Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Error
2024-11-26 12:38:58.854 20122-20563 Mavsdk                  com.dronesense.pilot.blue            D  Failed to fetch camera definition!

[JonasVautherin]

We build curl without ssl, I think it won't support https: https://github.com/mavlink/MAVSDK/blob/main/third_party/curl/CMakeLists.txt#L12

Would you have a way to use http instead?

A weird thing in your case is that the camera definition is downloaded from the internet. Shouldn't it be served by the drone? Or is it somehow a simulator setup?

[julianoes]

I tried to add openssl with https support in mavlink/MAVSDK#2386 and I gave up trying to get it building for all platforms. If you want to waste a few days staring at CI playing with dependencies, feel free. I find it utterly frustrating.

[rayw-dronesense]

@JonasVautherin

We build curl without ssl, I think it won't support https: https://github.com/mavlink/MAVSDK/blob/main/third_party/curl/CMakeLists.txt#L12

Would you have a way to use http instead?

A weird thing in your case is that the camera definition is downloaded from the internet. Shouldn't it be served by the drone? Or is it somehow a simulator setup?

It's a real drone and payload. Nope, they for some reason decided it was a good idea to have the file fetched from Github every time; here's their documentation: https://docs.gremsy.com/payloads/vio/camera-setting-menu/camera-definition-file-download - pretty sure I have no way of changing this behavior...

For generate the camera setting menu, the QGroundControl app need to download the camera definition file from Gremsy github server. The download path will be sent by the Vio automatically.

@julianoes

I tried to add openssl with https support in mavlink/MAVSDK#2386 and I gave up trying to get it building for all platforms. If you want to waste a few days staring at CI playing with dependencies, feel free. I find it utterly frustrating.

Oh boy...sorry to hear about that. By any chance, were we able to get it to work on Android at least?

[julianoes]

Pretty sure Android was one of the broken ones:
https://github.com/mavlink/MAVSDK/actions/runs/11588668231/job/32262783180#step:6:1416

Is the file available with http though? In that case, you can just intercept (but not in Java) the message containing the URL and change https to http.

Or what we could also try is to just parse https and use http to download. If it works good, otherwise no hard done, I'd think...

[JonasVautherin]

Whoops it got resolved automatically because I mentioned it in the PR. Let me reopen it.

[JonasVautherin]

Trying to backport it to v2.12 so that we can maybe release it before v3: mavlink/MAVSDK#2454

[rayw-dronesense]

@julianoes

Pretty sure Android was one of the broken ones:
https://github.com/mavlink/MAVSDK/actions/runs/11588668231/job/32262783180#step:6:1416

That's unfortunate. Thanks for the info!

Is the file available with http though? In that case, you can just intercept (but not in Java) the message containing the URL and change https to http.

Or what we could also try is to just parse https and use http to download. If it works good, otherwise no hard done, I'd think...

Hmm... I don't think Github lets you download using HTTP anymore; here's what happens when I take that URL and do a wget with "HTTP" instead of "HTTPS":

$ wget http://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
--2024-12-03 09:50:55--  http://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
Resolving github.com (github.com)... 140.82.112.4
Connecting to github.com (github.com)|140.82.112.4|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml [following]
--2024-12-03 09:50:56--  https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
Connecting to github.com (github.com)|140.82.112.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/697574913/7809a88c-be05-4ff3-8f7e-5a72ae3e69ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241203T145056Z&X-Amz-Expires=300&X-Amz-Signature=953f4511aeadde975534734960e4d0aeedd6555d6f07228f74fd7e0b997a79d1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvio_camera_f1_def.xml&response-content-type=application%2Foctet-stream [following]
--2024-12-03 09:50:56--  https://objects.githubusercontent.com/github-production-release-asset-2e65be/697574913/7809a88c-be05-4ff3-8f7e-5a72ae3e69ee?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20241203%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20241203T145056Z&X-Amz-Expires=300&X-Amz-Signature=953f4511aeadde975534734960e4d0aeedd6555d6f07228f74fd7e0b997a79d1&X-Amz-SignedHeaders=host&response-content-disposition=attachment%3B%20filename%3Dvio_camera_f1_def.xml&response-content-type=application%2Foctet-stream
Resolving objects.githubusercontent.com (objects.githubusercontent.com)... 185.199.109.133, 185.199.108.133, 185.199.111.133, ...
Connecting to objects.githubusercontent.com (objects.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 16600 (16K) [application/octet-stream]
Saving to: ‘vio_camera_f1_def.xml’

@JonasVautherin Thanks!

[JonasVautherin]

Can you try with mavsdk_server:2.1.4? Just pushed it to MavenCentral. It should support HTTPS

[rayw-dronesense]

@JonasVautherin - Thanks for the updated release! With 2.1.4 it does get a bit further, but runs into this issue,

2024-12-03 10:55:50.389  2182-4091  Mavsdk                  com.dronesense.pilot.blue            I  Download file: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml using cURL...
2024-12-03 10:55:50.392  2182-4091  Mavsdk                  com.dronesense.pilot.blue            I  Downloading camera definition from: https://github.com/Gremsy/Vio-Camera-Definition/releases/download/v2.0.3/vio_camera_f1_def.xml
2024-12-03 10:55:50.452  2182-4091  Mavsdk                  com.dronesense.pilot.blue            E  Error while downloading text, curl error code: Problem with the SSL CA cert (path? access rights?)
2024-12-03 10:55:50.452  2182-4091  Mavsdk                  com.dronesense.pilot.blue            E  Failed to download camera definition.
2024-12-03 10:55:50.456  2182-4091  Mavsdk                  com.dronesense.pilot.blue            I  Downloaded file, result Error
2024-12-03 10:55:50.456  2182-4091  Mavsdk                  com.dronesense.pilot.blue            D  Failed to fetch camera definition!

[JonasVautherin]

Oh yeah, that's a good question: in order to properly leverage HTTPS, curl (openssl?) needs to have access to certificates. Usually they are found on the system, but I honestly don't know how we should do that with our static library 🤔.

An easy way would be to disable the verification (see e.g. here). We could justify it by saying that downloading a wrong camera_definition.xml is not a security issue 😅. Ideally we would have a way to feed certificates to mavsdk_server though?

Would you mind looking into libcurl to see if there is a way to give it certificates? Apparently the executable has some options for that (see here), but in our case we use libcurl (not the executable). The next question would be whether we can read system certificates on Android...

I had not realized before, but it isn't as easy as just enabling https in curl 🙈.

EDIT: here I see stuff like this:

    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 1);
    curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1);
    curl_easy_setopt(curl, CURLOPT_CAINFO, "./ca.crt");

So if we had read access to the system certificates on Android, we could maybe just use that? I checked on my device and I have certificates in /system/etc/security/cacerts, for instance.

[JonasVautherin]

Oh maybe we could just set CURLOPT_CAPATH to /system/etc/security/cacerts on Android (with an ifdef in mavsdk)?

And it seems like this answer does it at build time!

I got this to work on Android by recompiling libcurl and configuring the default search path for certificates. This can be done by passing the option:

--with-ca-path=/system/etc/security/cacerts to ./configure

or

-DCURL_CA_PATH=/system/etc/security/cacerts to cmake