SSH host key converted to PEM fails in libssh ("unknown type")

[nbiskup-eridan]

I have a problem with SSH host key handling in libnetconf2 v3.7.10 (latest).

What happens

• I add a host key using nc_server_config_add_ch_ssh_hostkey().
• The key I provide is in the form:

-----BEGIN RSA PRIVATE KEY-----
...
-----END RSA PRIVATE KEY-----

• Inside libnetconf2, the provided key is converted into PEM format and stored in the datastore.
• Later, when nc_ssh_bind_add_hostkeys() runs, it writes the key into a temporary file and calls:

ssh_bind_options_set(sbind, SSH_BIND_OPTIONS_HOSTKEY, tmp_path)

• At this point, libssh fails with:

Failed to set hostkey ...

• and internally reports unknown type.
• It looks like libssh does not accept the PEM format produced from the conversion, while it would have accepted the original "-----BEGIN RSA PRIVATE KEY-----" type.

My questions

• Am I providing the key in the wrong format to nc_server_config_add_ch_ssh_hostkey()?
• Is it expected that the original key gets converted into PEM inside libnetconf2 before being stored?
• If so, which exact key formats are supported end-to-end (from datastore to libssh via ssh_bind_options_set())?
• Is there a recommended way to import/store the RSA host key so that nc_ssh_bind_add_hostkeys() succeeds?

[nbiskup-eridan]

this is the configuration that is saved in the datastore, maybe this information will also help:

<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
  <call-home>
    <netconf-client>
      <name>configured_client_1</name>
      <endpoints>
        <endpoint>
          <name>configured_client_1_ssh</name>
          <ssh>
            <tcp-client-parameters>
              <remote-address>10.0.0.103</remote-address>
              <remote-port>4334</remote-port>
            </tcp-client-parameters>
            <ssh-server-parameters>
              <server-identity>
                <host-key>
                  <name>server_hostkey-configured_client_1</name>
                  <public-key>
                    <inline-definition>
                      <public-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:ssh-public-key-format</public-key-format>
                      <public-key>AAAAB3NzaC1yc2EAAAADAQABAAABAQCrKVb6F5EMQdrFzNTbvqqdFP+510F1WLK1ake0wRF/o1r6O41jOQ4PwMMQvV7b6r4mm4bMSePrSxQ2jBD1SugyJ/p04I3Vnc1HCIQNq0CIMYPwQfpRmMdxqugA2H</public-key>
                      <private-key-format xmlns:np2="urn:cesnet:libnetconf2-netconf-server">np2:private-key-info-format</private-key-format>
                      <cleartext-private-key>MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCrKVb6F5EMQdrFzNTbvqqdFP+510F1WLK1ake0wRF/o1r6O41jOQ4PwMMQvV7b6r4mm4bMSePrSxQ2jBD1SugyJ/p04I3Vnc1HCIQ2ynJnGcE1M2UExdKO1E+8md0kDXW66FY0YpAWXa5X4qpOzY91LXUl7gr6Vut3+zE1KqNBFxTSy9G9OZKamtKYTmNTJN/DtGH/p3zG50SFieYspmNg=</cleartext-private-key>
                    </inline-definition>
                  </public-key>
                </host-key>
              </server-identity>
              <client-authentication>
                <users>
                  <user>
                    <name>USER0</name>
                    <password>$6$idsizuippipk$mfAL.DH7vF9t.bSyNjtsGzcJoaY4iJfsUIV3SWYY943IOi2oiAyACa0FEDw7kfTqY.</password>
                  </user>
                </users>
              </client-authentication>
            </ssh-server-parameters>
          </ssh>
        </endpoint>
      </endpoints>
      <connection-type>
        <persistent/>
      </connection-type>
    </netconf-client>
  </call-home>
</netconf-server>

[Roytak]

How did you even get this configuration? The lengths of the base64 encoded keys are not divisible by 4, so they are invalid. The password encoding is also invalid.. libyang should fail while parsing this, so I am unsure how you obtained this.

[Roytak]

Hello, what versions of libssh and openssl and what platform are you using? The scenario you described should generally work. Is the libnetconf2 test called test_replace also failing on your machine? It also tries to load this PKCS1 private key by converting it to PEM first. I will get back to you with answers to your questions once I've verified them.

[nbiskup-eridan]

We are using our own Yocto-based environment, so we are not running the upstream libnetconf2 tests. At the moment I cannot run those tests on this platform since we do not use cmock and some other related components.

Our versions are:

OpenSSL: 3.2.3
libssh: 0.10.6

[Roytak]

Am I providing the key in the wrong format to nc_server_config_add_ch_ssh_hostkey()?

No, this format is correct.

Is it expected that the original key gets converted into PEM inside libnetconf2 before being stored?

Yes. Generally there is no restriction on the private key format when it is used as a hostkey, but there is one for the public key format - it has to conform to the "ssh-public-key-format" identity defined in ietf-crypto-types YANG module. If I recall correctly, the conversion to PEM was used to utilize the ability of OpenSSL to help generate a public key from its private key in said format. The original private key was then discarded and the PEM generated by OpenSSL was used.

If so, which exact key formats are supported end-to-end (from datastore to libssh via ssh_bind_options_set())?

Judging by this libssh code, it should support all the private key formats supported by libnetconf2 - private-key-format identities defined in ietf-crypto-types and libnetconf2-netconf-server YANG modules (to keep it simple, you can refer to the macros defined here).

Is there a recommended way to import/store the RSA host key so that nc_ssh_bind_add_hostkeys() succeeds?

There are two options - either use the API such as nc_server_config_add_ch_ssh_hostkey() or define the ietf-netconf-server YANG data yourself and apply the configuration using nc_server_config_setup_data(). While the first approach should generally work for any valid asymmetric key pair, it might do so by converting the private key to another format. The second approach keeps it as it is.

I agree that the private key format conversion might be misleading and will look into keeping it in the same format - it should not however lead to the failure of loading a private key. Could you please provide the commands used to generate the private key?

[nbiskup-eridan]

Thanks for the detailed answers. I will try defining my own yang configuration and apply it to see if that works.

Regarding the commands used to generate the private key, here is the exact sequence:

openssl genpkey -algorithm rsa -out rsa.key
openssl rsa -in rsa.key -out rsa-trad.key -traditional
openssl pkey -in rsa.key -pubout -out rsa.key.pub

In the end we are using rsa-trad.key

[Roytak]

Thank you for sharing the commands. If you wish to define it yourself, this is what should generally work:

openssl genpkey -algorithm rsa -out rsa.key
openssl rsa -in rsa.key -out rsa-trad.key -traditional
sed '1d;$d' privkey.pem | tr -d '\n' > rsa-trad-trimmed.key                # extract the base64 blob

ssh-keygen -y -f rsa-trad.key > rsa-trad.pub                               # the pubkey needs to be in the "SSH" format
awk '{print $2}' rsa-trad.pub > rsa-trad-trimmed.pub                       # extract the base64 blob

And then using e.g. the original configuration you provided:

<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
  <call-home>
    <netconf-client>
      <name>configured_client_1</name>
      <endpoints>
        <endpoint>
          <name>configured_client_1_ssh</name>
          <ssh>
            <tcp-client-parameters>
              <remote-address>10.0.0.103</remote-address>
              <remote-port>4334</remote-port>
            </tcp-client-parameters>
            <ssh-server-parameters>
              <server-identity>
                <host-key>
                  <name>server_hostkey-configured_client_1</name>
                  <public-key>
                    <inline-definition>
                      <public-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:ssh-public-key-format</public-key-format>
                      <public-key>rsa-trad-trimmed.pub contents</public-key>
                      <private-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:rsa-private-key-format</private-key-format>
                      <cleartext-private-key>rsa-trad-trimmed.key contents</cleartext-private-key>
                    </inline-definition>
                  </public-key>
                </host-key>
              </server-identity>
              <client-authentication>
                <users>
                  <user>
                    <name>USER0</name>
                    <password>$0$cleartextpassword</password>
                  </user>
                </users>
              </client-authentication>
            </ssh-server-parameters>
          </ssh>
        </endpoint>
      </endpoints>
      <connection-type>
        <persistent/>
      </connection-type>
    </netconf-client>
  </call-home>
</netconf-server>

Replace the private and public key elements with the base64 encoded keys, store it in a file, say config.xml, and call nc_server_config_setup_path(context, path), similar to the example found here.

[nbiskup-eridan]

Thanks for the suggestion. I managed to get it working. Basically, I took your proposed steps and implemented them in the code, and now it works correctly. Below is how the configuration looks now.

Thank you for the cooperation, but I still believe that your API functions should work properly with the RSA key I provide. It would be good to take that into consideration.

<netconf-server xmlns="urn:ietf:params:xml:ns:yang:ietf-netconf-server">
  <call-home>
    <netconf-client>
      <name>configured_client_1</name>
      <endpoints>
        <endpoint>
          <name>configured_client_1_ssh</name>
          <ssh>
            <tcp-client-parameters>
              <remote-address>10.0.0.103</remote-address>
              <remote-port>4334</remote-port>
            </tcp-client-parameters>
            <ssh-server-parameters>
              <server-identity>
                <host-key>
                  <name>server_hostkey-configured_client_1</name>
                  <public-key>
                    <inline-definition>
                      <public-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:ssh-public-key-format</public-key-format>
                      <public-key>AAAAB3NzaC1yc2EAAAADAQABAAABAQC14OdfWnYcmovxAY7SNhrShzjRJrS32+ihoR0EKelwOqKfXuTYul68oKPl7fC8nXyxz1NNyDEzVHl8NxXRRPnC73nAl3MHMvLxEq79fnh24l</public-key>
                      <private-key-format xmlns:ct="urn:ietf:params:xml:ns:yang:ietf-crypto-types">ct:rsa-private-key-format</private-key-format>
                      <cleartext-private-key>MIIEowIBAAKCAQEAteDnX1p2HJqL8QGO0r/cD/1QjxKfUswpiGIalSvZ1qsy8H04jJaN8IKVTfs04aIInPCIPYw6UfzFw3XPkIWV1+6cn3mZKL1U+rMV+mLXK3t3fTvQFohT71Ml3D3c5kEwkqFqyk8QWzYpvnU333v7j3zpJeYR6QRseKlXcfcJEvZdSTTzvvqCkmYXLwr1GWI/h4a5SdpH4FAJu5/xfxvbH59CnsECYjG9i5oLfoap0uZMEmjx8bffNO1GKdTcEes/im6nZxH5vaTTYuH3HDSFt7Kepk0aaaShqiQHyKiVqWZj+F2pIRaVwzyNXa5uXLp6VwPuD8h9pBL1HcGwi58zqqNgdRmh79Wz3EKBtQZtbECgYEA2LrzLF1GVnqQzPYF51G958nLLvCSOoFBrdD358IMVvG2JNtKGsHwpA0uiP5aov/5kJh8BdBv+hlC2J3suLpn5wRCjoHaqFXKPXnanGwlswn/YjYjWvQWNxoKBfGj9YOf24rYZp6v24FZ4LBWA5NQggL87iney+mxadEAgmEshLECgYEA1tVaao+5b9mXkm0474XBQ7sVQDPUcCHdOr/ynr0Zshw4ti8q7pFzIsmo7U8Kvz1/vqgkcv6LA5uuKyEmZUT9ZH5VxJnCoVcAj8W0fDImx64ow3SU59SC9+JrFdyUYYr7qbsydNFdnfwhxbpMHPDc9QnNUH0qxpUGitZui1QCrbUCgYEAoBMsyoa2DeeHDatk3t50Cj/0T14mUwnnDc97WvTrEkiDBKPnDu5Ro4mM5mgfX6b6mD7X05+JJUsFIWM7VZFwbqOp2QG2E4ER1+gpfEEwcxWeR9a+tE32Hoqf/pXkSNEIACqD/P+V2eWnEZqMXSWwX+Z/FBVe1nNrs0FIRAQr0nECgYBnqjwe/4tztYO2XI4Exn4juLrgaCIymaSvHMBuaMFleMjE4K3OwNDWEIk7OdDtxzdFQM2Vz840x4KbvTtR0qCTRYMNg0SvpdPXjIczBuA/qoJxDmhsFde81bzmp7HR/8bAeFw4km6iNW/2gZcIwBFfDIPOOxfRC0hE5oQhJ0DDoQKBgGUlJFrsBeFCTYwFbxf8eQeXrM5oU1vmBeNPCu07vYMfU1dvo2Q7s7pT/xYmqjl1jXpYsmT9F8UI4LrBQxa73i0v7RpN6gOhzEc+z90gqqCIFOwc1Fe481c1JvwIroZ32mllr/h3Pbnx7iFpeUvKqr9+dX/dPHn+9ZPIbw7kSGpr</cleartext-private-key>
                    </inline-definition>
                  </public-key>
                </host-key>
              </server-identity>
              <client-authentication>
                <users>
                  <user>
                    <name>USER0</name>
                    <password>$6$idsizuippipk$mfAL.DH7vF9t.bSyNjtHRNStz5S835lYuQU7jsGzcJoaY4iJfsUIV3SWYY9mm43IOi2oiAyACa0FEDw7kfTqY.</password>
                  </user>
                </users>
              </client-authentication>
            </ssh-server-parameters>
          </ssh>
        </endpoint>
      </endpoints>
      <connection-type>
        <persistent/>
      </connection-type>
    </netconf-client>
  </call-home>
</netconf-server>

[Roytak]

Perfect! Like I said, I still don't understand why the API was not working for you as I was unable to reproduce it and would need more details. If you could provide a private key and code to reproduce this with, that would be really helpful.