Feat: Ship test-doubles/fakes together with the charm lib

[MichaelThamm]

Enhancement Proposal

It would be ideal if (from ref)

The team that owns the real implementation of the charm lib should write and maintain the fake.
Without up-to-date test doubles, other engineers will pollute their charm tests with mocks, which could quickly get out of sync with the real implementation.

Currently, if we want to scenario test we would have to place dummy certificate data in the remote_app_data:

    ssc = Relation(
        endpoint="certificates",
        interface="tls-certificate",
        remote_app_data={
            "certificates": json.dumps(
                [
                    {
                        "certificate": "placeholder",
                        "certificate_signing_request": "placeholder",
                        "ca": "placeholder",
                        "chain": ["first", "second"],
                    }
                ]
            )
        },
    )

[gruyaume]

Hello @MichaelThamm ,

Currently, if we want to scenario test we would have to place dummy certificate data in the remote_app_data

Not quite, you could also mock the calls made to the TLS lib. Reading/Writing to certificates relation data is of the tls library's domain. This is what we do in most cases.

That being said, I'm not against shipping test fakes. Do you have examples of such test fakes in Observability-owned libs (or elsewhere) that we could replicate here?

[sed-i]

Hey @gruyaume,
We didn't get around to practicing it ourselves yet, but:

• It is not immediately obvious for users of the tls lib how to correctly mock.
• "This is what we do in most cases" - it would be great if you could package this to be easily consumed by users in their tests so they won't need to go and figure out what makes sense to mock, and avoiding pitfalls.
• Since libs are restricted to one file, I imagine the test double can go in that file, so if we have XProider, we'd also have XProviderTestDouble. In some case multiple different test double make sense. Then in test all we'd need to do is "replace" usages of XProvider with XProviderTestDouble.
  • Alternatively, the lib could provider composite Mock object(s). When there is only one mock to apply it makes everything very simple.

[gruyaume]

Right, this sounds like it would be a good idea. I imagine to achieve this we probably want the mock to be a ops.testing.Relation object.

In any case, for now, the best we have to offer is test code examples. Here's how we currently mock interactions with the TLS library:

import tempfile
from datetime import timedelta
from unittest.mock import patch

from charms.tls_certificates_interface.v4.tls_certificates import (
    PrivateKey,
    ProviderCertificate,
    generate_ca,
    generate_certificate,
    generate_csr,
    generate_private_key,
)
from ops import testing

from charm import ExampleCharm


@patch(
    "charms.tls_certificates_interface.v4.tls_certificates.TLSCertificatesRequiresV4.get_assigned_certificate"
)
def test_demo_tls(mock_get_assigned_certificate):
    ctx = testing.Context(charm_type=ExampleCharm)

    with tempfile.TemporaryDirectory() as tempdir:
        certificates_relation = testing.Relation(
            endpoint="certificates", interface="tls-certificates"
        )

        certs_mount = testing.Mount(
            location="/support/TLS",
            source=tempdir,
        )

        container = testing.Container(
            name="example", can_connect=True, mounts={"certs": certs_mount}
        )
        state_in = testing.State(
            leader=True,
            containers={container},
            relations={certificates_relation},
        )

        provider_certificate, private_key = example_cert_and_key(
            tls_relation_id=certificates_relation.id
        )
        mock_get_assigned_certificate.return_value = provider_certificate, private_key

        ctx.run(ctx.on.update_status(), state_in)

        with open(tempdir + "/cert.pem", "r") as f:
            assert f.read() == str(provider_certificate.certificate)

        with open(tempdir + "/key.pem", "r") as f:
            assert f.read() == str(private_key)


def example_cert_and_key(tls_relation_id: int) -> tuple[ProviderCertificate, PrivateKey]:
    private_key = generate_private_key()
    csr = generate_csr(
        private_key=private_key,
        common_name="example.com",
    )
    ca_private_key = generate_private_key()
    ca_certificate = generate_ca(
        private_key=ca_private_key,
        common_name="ca.com",
        validity=timedelta(days=365),
    )
    certificate = generate_certificate(
        csr=csr,
        ca=ca_certificate,
        ca_private_key=ca_private_key,
        validity=timedelta(days=365),
    )
    provider_certificate = ProviderCertificate(
        relation_id=tls_relation_id,
        certificate=certificate,
        certificate_signing_request=csr,
        ca=ca_certificate,
        chain=[ca_certificate],
    )
    return provider_certificate, private_key

[sed-i]

Just to follow up more explicitly, when we test our charms it could be handy to have e.g.

from charms.tls_certificates_interface.v4.tls_certificates import mock_all

and then in test code we'd have

    @tls_certificates.mock_all
    def test_tls_relation_with_tls_ready(self, *unused):
        ...

Example.

[PietroPasotti]

not sure that from charms.tls_certificates_interface.v4.tls_certificates import mock_all is achievable or even desirable.
First, this sounds more like a patch.
Secondly, what would it mock/patch? all public API? It seems hard to do in a meaningful way.

I think it'd be more meaningful to expose fakes (aka plausible-looking data) instead:

• provider_app_data: Dict[str, str]
• provider_unit_data: Dict[str, str]

That we'd use in our tests like so:

from charms.tls_certificates_interface.v4.tls_certificates import fakes

def test_foo():
    relation_in = Relation(remote_app_data=fakes.provider_app_data, remote_units_data={1: fakes.provider_unit_data})

Or alternatively something like this:

from charms.tls_certificates_interface.v4.tls_certificates import fakes

def test_foo():
    relation_in = Relation(local_app_data)
    relation_in_with_response = fakes.fake_response(relation_in)

fake_response would take a relation object and populate the databags of the 'other side'.

This way the tests can remain agnostic as to the shape of the remote data, but can flexibly simulate situations in which:

• the remote end didn't reply yet
• the remote end replied already

This approach is also better than a 'patch-all' one in multi-stage interfaces.
Take traefik:

from charms.traefik.v1.ingress import fakes

def test_foo():
    empty_rel = Relation("ingress")
    relation_with_request = fakes.fake_request(empty_rel)  # populates the requirer side of the relation

    relation_with_response = fakes.fake_response(empty_rel) # gives you back another empty-rel! 
    relation_with_response = fakes.fake_response(relation_with_request) # gives you back the ingress data

In the case interfaces that do version negotiation, even more steps will be required.