Add opentelemetry integration to Oban #6

indrekj · 2021-08-30T11:40:50Z

By default a new trace is automatically started when a job is processed
by monitoring these events:

[:oban, :job, :start] — at the point a job is fetched from the database and will execute
[:oban, :job, :stop] — after a job succeeds and the success is recorded in the database
[:oban, :job, :exception] — after a job fails and the failure is recorded in the database

To also record a span when a job is created and to link traces together
Oban.insert/2 has to be replaced by OpentelemetryOban.insert/2.

Before:

  %{id: 1, in_the: "business", of_doing: "business"}
  |> MyApp.Business.new()
  |> Oban.insert()

After:

  %{id: 1, in_the: "business", of_doing: "business"}
  |> MyApp.Business.new()
  |> OpentelemetryOban.insert()

Checklist

Acknowledgements

I certify that I am authorized to assign ownership of this code to the OpenTelemetry project.
I acknowledge that ownership of the Hex package must be given/transferred to the OpenTelemetry Hex Org

Project Type

Prior Art

Is this an existing project? Includes private libraries being open sourced.
- Yes
- No
Are there any other packages on Hex providing similar functionality? - I'm not aware of it.

Existing Project

Moving from an existing public repository? Provide a link in the description.
All authors who have contributed code contained in this PR have signed the CNCF CLA - no other authors at the moment

All Projects

Label created (maintainers)
labeler.yml updated
CI
- Elixir or Erlang language file is updated
- Test strategy should use the matrix strategy with all supported language/OTP combinations
- The job is scoped to only run for relevant code changes
Codeowners adds scoped ownership for project maintainers
License file (Apache2 - copyright to OpenTelemetry Authors)
All code complies with the Otel Specification and Otel Semantic Conventions - Trace Metrics

instrumentation/opentelemetry_oban/test/opentelemetry_oban_test.exs

bryannaegele · 2021-08-30T17:42:22Z

Thanks for opening this @indrekj. Once I get the phoenix migration complete, I will try to write up a guide in the next few days on migrating a library to here.

I couldn't find the original repo out on github or the package in hex. Do you have links to those?

Also, one thing to get a jump on is verify anyone who has contributed to the original codebase has signed the CLA https://identity.linuxfoundation.org/projects/cncf.

indrekj · 2021-08-30T18:17:17Z

I couldn't find the original repo out on github or the package in hex. Do you have links to those?

I deleted the repository. Basically ported the initial (non-merged) PR to here. There have been no releases yet. That's why you cannot find the repo nor the package. I can release the first version (v0.1) after this gets merged.

Also, one thing to get a jump on is verify anyone who has contributed to the original codebase has signed the CLA https://identity.linuxfoundation.org/projects/cncf.

Again, not an issue. There were no contributions before that. The current code is written by myself.

bryannaegele · 2021-08-30T18:37:57Z

We'll do a formal review later as part of this, but I had a couple of general questions on the implementation that could be considered while we work out CI and processes.

Propagation: There's currently no ability for propagating context to the jobs for linking spans to parents. In our implementation at work, we trace directly and aren't using telemetry. Is there a mechanism you can think of to propagate that ctx through job metadata that could be implemented?
Job creation: We have a process event here but no corresponding send or receive. Are there mechanisms or opportunities here we could leverage?
Batches: It would be nice to support batches with the batch id. This also relates to propagation in general.

indrekj · 2021-08-31T08:17:04Z

Propagation: There's currently no ability for propagating context to the jobs for linking spans to parents. In our implementation at work, we trace directly and aren't using telemetry. Is there a mechanism you can think of to propagate that ctx through job metadata that could be implemented?

I did look into this but didn't find a good transparent option. I initially thought that maybe using the Oban plugins mechanism could help here. But the jobs are inserted like this. This is basically a direct insertion to the database so there are no plugins involved.

There is a meta field on the job that could be used to propagate tracing context.

So if the user is willing to change the codebase then we could propagate the information through that.

E.g. before:

      %{account_id: 1, url: "https://example.com"}
      |> MyApp.Worker.new()
      |> Oban.insert()

after:

      %{account_id: 1, url: "https://example.com"}
      |> MyApp.Worker.new()
      |> OpentelemetryOban.enhance()
      |> Oban.insert()

where OpentelemetryOban.enhance(worker) would add propagation information to worker.meta field. Is this something you think would be a good solution?

Job creation: We have a process event here but no corresponding send or receive. Are there mechanisms or opportunities here we could leverage?

send: The whole send is basically just an insert call to the Postgres database. So there aren't any hooks/plugins that we can use (at least I don't see them). Again, if we'd use OpentelemetryOban.enhance() helper method then theoretically we could create a span there. But that seems very brittle because it doesn't actually wrap the insert call and if the user inserts the job later then the timings will be wrong. It would be better to wrap or replace the insert call somehow, but I'm not exactly sure how. I'm open to ideas.

receive: Correct me if I'm wrong but to my understanding, this is meant to be used in case of batch processing or when receive and process and done separately. I checked a couple of ruby libraries (1, 2, 3) and they all seemed to use process without the receive event. Oban seems to support batch workers but that's available only through the Pro version. I'd prefer to focus on the free version first.

bryannaegele · 2021-09-01T18:51:16Z

So if the user is willing to change the codebase then we could propagate the information through that.

I would strongly suggest supporting that and selection of metadata keys for inclusion as attributes.

This is from 0.4 code. We have this in our workers, but with the new span ctx format, I'd recommend using text_map_inject. Oban serializes metadata to json, so it's easier to just use propagators.

  defp decode_ctx("undefined"), do: :undefined

  defp decode_ctx(ctx) do
    Enum.map(ctx, fn
      el when is_binary(el) -> String.to_existing_atom(el)
      el -> el
    end)
    |> List.to_tuple()
  end

indrekj · 2021-09-02T13:04:56Z

@bryannaegele I did a bunch of changes here.

But basically now if the user uses OpentelemeryOban.insert/2 instead of Oban.insert/2 then a send span is created and also the context is propagated through the Oban job metadata. On the receiving side, we create a new trace and link them together.

Also added some instructions to the project README file.

I'll deal with CI after opentelemetry_phoenix gets merged to avoid duplicate work. The tests pass locally, but for CI (Github actions) I need to somehow set up Postgres.

tsloughter · 2021-09-03T11:50:59Z

But basically now if the user uses OpentelemeryOban.insert/2 instead of Oban.insert/2 then a send span is created and also the context is propagated through the Oban job metadata. On the receiving side, we create a new trace and link them together.

Nice!

And sadly this is the reality of libraries using telemetry for instrumentation related to tracing :(, especially if they don't offer any middleware/hooking functionality.

telemetry is filling the role that opentelemetry_api is meant to, but with less functionality -- the "role" being a way for everyone to instrument their code with a common library that by default does nothing, so users who don't want it can ignore it, and is completely pluggable in what it does do if the user chooses to enable it.

Not that I don't think that was completely reasonable to do :), just stating the state of things.

There is likely a need for a push for libraries that use telemetry and make these sort of external service calls to all include a pluggable way to extend those calls. Then when opentelemetry_oban starts it can register a function based on the configured opentelemetry propagators.

Because some users who want to enable tracing are going to be using Oban indirectly and won't be able to simply use the new function. Same issue exists for HTTP, database, MQ, Kafka, etc clients

dvic · 2021-09-13T07:28:55Z

I just came across this PR, FYI we have also an implementation available at https://github.com/qdentity/opentelemetry_oban. For this I had to get two PRs into Oban: https://github.com/sorentwo/oban/pulls?q=is%3Apr+author%3Advic+is%3Aclosed. We can have a look and, if needed, merge missing pieces into this PR.

indrekj · 2021-09-13T08:07:04Z

@tsloughter @bryannaegele Rebased my PR and addressed the checklist in the open checklist PR.

@dvic I did see that repository, though only after most of my code was already done. I would like to get this PR approved/merged before porting over your changes. This current PR takes care of job processing and trace propagation. I see you added tracing for circuits, plugins etc. It likely makes sense to port them over but as it needs some testing and making sure they follow semantic conventions I'd like to start with the basic features that this PR should take care of.

dvic · 2021-09-17T13:35:49Z

@indrekj I just tried your PR and I must say, it's looking good :) I like the way you organized the attributes and distinguished between process/send spans, while also supporting linking them to each other. I archived https://github.com/qdentity/opentelemetry_oban and referred to this PR.

Let me know if there's anything I can do to get this PR merged and published 👍

dvic · 2021-09-20T11:06:37Z

Another thing that you could add this PR: OpentelemetryOban.insert!/1, I noticed that one was missing (Oban has this one)

indrekj · 2021-09-20T12:20:07Z

Ack, I'll take a look tomorrow

dvic · 2021-09-20T12:37:29Z

Ack, I'll take a look tomorrow

👍 Another thing I just encountered that you might want tackle as well: ricardoccpaiva/opentelemetry_tesla#5 (it seems we have to do &__MODULE__.my_handle_fun/4 instead of &my_handle_fun/4)

indrekj · 2021-09-21T13:20:07Z

@dvic Updated

dvic · 2021-09-21T13:21:53Z

@dvic Updated

Perfect, thanks 👍

dvic · 2021-11-23T11:53:09Z

@tsloughter @bryannaegele Updated

Sampler has been removed

CI is now green - Postgres configuration was indeed missing

Span linking - I would like to add a propagation_style configuration option so the user can choose between Links and Child-Of style. Similar to Ruby libraries. But I would like to do that in a separate PR after this PR, because this PR is already huge and I don't want to make it even bigger.

Great :) Should we add to the docs a line about how to exclude the internal query (by tracing the plugin events and then using a root sampler)?

tsloughter · 2021-11-23T21:14:46Z

Agreed on a separate PR for adding propagation style.

dvic · 2021-11-24T14:29:41Z

I have used the following sampler:

defmodule MySampler do
  @behaviour :otel_sampler

  defstruct [:opts]

  @impl true
  def setup(opts), do: %__MODULE__{opts: opts}

  @impl true
  def description(%__MODULE__{}), do: "MySampler"

  @impl true
  def should_sample(ctx, _trace_id, _links, span_name, _span_kind, attributes, _opts) do
    drop =
      String.ends_with?(span_name, ".repo.query:oban_producers") ||
        String.ends_with?(span_name, ".repo.query:oban_jobs") ||
        String.starts_with?(span_name, "Elixir.Oban.Plugins") ||
        String.starts_with?(span_name, "Elixir.Oban.Pro.Plugins") ||
        Keyword.get(attributes, :"db.statement") in ["begin", "commit"]

    decision =
      if drop do
        :drop
      else
        :record_and_sample
      end

    attributes = []

    tracestate =
      OpenTelemetry.Tracer.current_span_ctx(ctx)
      |> OpenTelemetry.Span.tracestate()

    {decision, attributes, tracestate}
  end
end

and then in the config:

my_sampler = {MySampler, []}

config :opentelemetry,
  sampler:
    {:parent_based,
     %{
       root: my_sampler,
       local_parent_sampled: my_sampler,
       remote_parent_sampled: my_sampler
     }}

If you think it's useful we could add this to the docs as a starter.

indrekj · 2021-11-24T15:29:17Z

I'm personally not a fan of that approach. It requires the user to manually list every possible span there could be. If the underlying library changes and there's a new span then this has to be modified again. This can be quite cumbersome when there are many services.

In general, this seems to be a common problem and I think API/SDK should provide a way to start a non-recording span. E.g. In ruby there's a helper method untraced for instrumentation libraries. These are used in libraries like sidekiq.

dvic · 2021-11-24T15:30:54Z

I'm personally not a fan of that approach. It requires the user to manually list every possible span there could be. If the underlying library changes and there's a new span then this has to be modified again. This can be quite cumbersome when there are many services.

In general, this seems to be a common problem and I think API/SDK should provide a way to start a non-recording span. E.g. In ruby there's a helper method [untraced](https://github.com/open-telemetry/opentelemetry-ruby/blob/2bf0e6aed060867bc65560a92e2c464d48b26a02/common/lib/opentelemetry/common/utilities.rb#L75-L77) for instrumentation libraries. These are used in libraries like sidekiq.

Interesting, but children of a non-recording span are still sampled right? How would this pan out with the ecto queries problem?

indrekj · 2021-11-24T15:32:31Z

Interesting, but children of a non-recording span are still sampled right?

No, they're not. The decision is propagated to all the children spans.

The sidekiq library had exactly the same problem. The Poller was running in the background and that caused Redis spans to be created without any context/parents. So creating a non recording span/trace removed them.

dvic · 2021-11-24T15:33:59Z

Interesting, but children of a non-recording span are still sampled right?

No, they're not. The decision is propagated to all the children spans.

Ah I see, didn't know that, thanks!

tsloughter · 2021-11-28T16:22:25Z

@indrekj so you've got the tests against postgres working in CI? Making this good to merge now?

indrekj · 2021-11-28T16:44:35Z

Yup I think so

…

On Sun, Nov 28, 2021, 18:22 Tristan Sloughter ***@***.***> wrote: @indrekj <https://github.com/indrekj> so you've got the tests against postgres working in CI? Making this good to merge now? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAA23X5SULDQK7VEUKDXHDUOJJMZANCNFSM5DBWAPHQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

dvic · 2021-11-30T09:07:58Z

instrumentation/opentelemetry_oban/lib/opentelemetry_oban/job_handler.ex

+      kind: :consumer,
+      links: links
+    })
+    |> Span.set_attributes(attributes)


@indrekj why is this done with Span.set_attributes? It's not possible to set the attributes directly in the start_opts here? The reason why I'm asking is that I don't see the attributes in my sampler (but they do appear in Honeycomb). I'm receiving this span without the attributes in my sampler. @tsloughter is this intended behaviour of the sampler / span design?

I think I used some other project as an example. The start_opts works with attributes key indeed: https://github.com/open-telemetry/opentelemetry-erlang-contrib/compare/9ff9f64044d47306c0764dd9e12472a1b7eb3349..5b50df46ca8c0847f89e595cf74d4a1e2892c6e8 . I'm also using set_attributes in two other places (insert functions) but there I don't have access to all attributes immediately so there it makes sense to use set_attributes.

With this change now I do get the attributes in the sampler, but I'm not sure if this is a bug or not, I was expecting this to work also with Span.set_attributes.

I think it might be intentional. Imagine you have this flow:

Start span A

Send outbound request to service B

Set attributes to span A

Do something and finish span A

In the step 2, the sampling decision has to be known. Otherwise, service B doesn't know if the spans should be sampled or not. So the sampler has to be run in the step 1.

dvic · 2021-11-30T13:21:31Z

instrumentation/opentelemetry_oban/lib/opentelemetry_oban/job_handler.ex

+    :otel_propagator_text_map.extract(Map.to_list(job_meta))
+    parent = OpenTelemetry.Tracer.current_span_ctx()
+    links = if parent == :undefined, do: [], else: [OpenTelemetry.link(parent)]
+    OpenTelemetry.Tracer.set_current_span(:undefined)


@indrekj Due to the force pushes on this PR I lost a bit track when this was introduced (maybe it's a good idea to do one force push at the end ;)), but currently I'm having the problem that the span started inside a job are not children of the job span, is this intended? Are you also seeing this behaviour? The consequence of this is that my custom sampler is not that useful anymore :)

I mean, I can always start another wrapper span inside the job, but I just wanted to double check and understand that this is intended behavior.

We're not seeing this. I also added a test to ensure it: https://github.com/open-telemetry/opentelemetry-erlang-contrib/compare/db0e1f1a67bbd51992b10a0020da3b21119f6edb..ddf26187cee991e329c9dcbeaa42ad50eacc0c5d

We however had a similar issue when we were using def timeout(_job) do in the jobs. If timeout function is defined then oban uses perform_timed to execute the worker, see: https://github.com/sorentwo/oban/blob/60274b76ff2c59d660ae1bf7721a3ff14854700b/lib/oban/queue/executor.ex#L257-L269 . This creates a new process so the current span information is lost. We only had one job in the entire system with that method, so we used:

# We use our own `with_timeout` instead of Oban.Worker.timeout/1 because if # the latter is used then the trace context is not propagated. defp with_timeout(fun) do task = Task.async(OpentelemetryFunction.wrap(fun)) case Task.yield(task, @timeout) || Task.shutdown(task) do {:ok, reply} -> reply nil -> {:error, :timeout} end end

I'd like to get this PR merged to start addressing these kind of rare edge cases. As you said, the PR is huge and it's hard to follow the changes now.

Ahhh, yes in this case we also used Oban.Worker.timeout/1, thanks! 👍

I'd like to get this PR merged to start addressing these kind of rare edge cases. As you said, the PR is huge and it's hard to follow the changes now.

I agree, maybe we can track these open issues / edge cases in the description so we can work on it when the PR is merged.

Yea, I can merge and then you should start opening issues.

I saw some question about set_attributes and sampling but can't find it again, so answering here: The sampler won't see attributes set with set_attributes because it has already been sampled at that point. A span is started and the attributes passed to the function that starts the span will be available but any attributes set after starting the span are of course not available.

Not sure what the code was, maybe it was something that used a pipe to look like it was using the "builder pattern" to construct a span? But we don't use that pattern, so there is no "build" function that would come after doing something like span("name") |> set_attributes(...) |> start(). So yea, must pass any attributes that need to be available for sampling to the start call.

Thanks. It was here: https://github.com/open-telemetry/opentelemetry-erlang-contrib/pull/6/files/9ff9f64044d47306c0764dd9e12472a1b7eb3349#diff-ca8cd6f4f23c284564be4b34cb5caf868177e4190300720bc3c6fdb255c18b90 . For that particular case we were able to avoid set_attributes

Yea, I can merge and then you should start opening issues.

👍

By default a new trace is automatically started when a job is processed by monitoring these events: * `[:oban, :job, :start]` — at the point a job is fetched from the database and will execute * `[:oban, :job, :stop]` — after a job succeeds and the success is recorded in the database * `[:oban, :job, :exception]` — after a job fails and the failure is recorded in the database To also record a span when a job is created and to link traces together `Oban.insert/2` has to be replaced by `OpentelemetryOban.insert/2`. Before: ```elixir %{id: 1, in_the: "business", of_doing: "business"} |> MyApp.Business.new() |> Oban.insert() ``` After: ```elixir %{id: 1, in_the: "business", of_doing: "business"} |> MyApp.Business.new() |> OpentelemetryOban.insert() ```

indrekj · 2021-12-08T15:34:16Z

PR has been approved for some time now, and there doesn't seem to be more comments. Could somebody merge it?

tsloughter · 2021-12-08T15:40:07Z

Thanks for the persistence :). I keep preparing to merge it, hit the update button, checks start running again... and I move on to something else and forget :(. Checks are again running after I hit the update button, this time I'll try not to move on to something else.

tsloughter · 2021-12-08T15:45:23Z

Merged.

I looked up the package on hex and it appears to be owned by a company/org "salemove". Is that you?

We need the opentelemetry org/team added to it on hex so we can also publish if need be -- as in, we won't be publishing the new release but in case the other owner went away or wasn't available and there was a security vulnerability, stuff like that, we'd need to be able to publish.

indrekj · 2021-12-08T16:00:08Z

@tsloughter Thx.

Ownership has been transferred to opentelemetry now. See https://hex.pm/packages/opentelemetry_oban . I think I lost all permissions myself now, but that doesn't really matter I think. Probably releasing new versions should be part of the github actions flow anyway.

tsloughter · 2021-12-08T18:58:27Z

Thanks, you should stay as an owner to be able to publish. It is much better to release manually and not github actions. Unless github actions has a way to require manual intervention during an action which may make it a viable option.

dvic · 2021-12-08T20:04:47Z

Thanks, you should stay as an owner to be able to publish. It is much better to release manually and not github actions. Unless github actions has a way to require manual intervention during an action which may make it a viable option.

Yes, I believe you can do this in public repos (and github enterprise). See https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments

indrekj · 2021-12-09T11:57:55Z

Thanks, you should stay as an owner to be able to publish.

Could you run mix hex.owner add opentelemetry_oban indrek in that case? All my permissions were lost when I transferred this between organizations.

tsloughter · 2021-12-09T12:14:59Z

Hm, that didn't work, but I swear you can have both a user and an owner as the owner. It returned "cannot add owner that is not a member of the organization"

github-actions bot added elixir instrumentation scope-ci labels Aug 30, 2021

indrekj marked this pull request as ready for review August 30, 2021 11:41

indrekj requested a review from a team August 30, 2021 11:41

indrekj commented Aug 30, 2021

View reviewed changes

instrumentation/opentelemetry_oban/test/opentelemetry_oban_test.exs Show resolved Hide resolved

indrekj force-pushed the opentelemetry_oban branch from 7bbe628 to ef1a06f Compare August 30, 2021 11:47

indrekj force-pushed the opentelemetry_oban branch from f4170a0 to 2958c39 Compare August 30, 2021 18:21

bryannaegele marked this pull request as draft August 30, 2021 18:22

indrekj force-pushed the opentelemetry_oban branch from 2958c39 to baf26b9 Compare September 2, 2021 12:59

indrekj force-pushed the opentelemetry_oban branch from baf26b9 to 41852f3 Compare September 3, 2021 07:14

indrekj force-pushed the opentelemetry_oban branch from 41852f3 to c20a9ad Compare September 13, 2021 07:56

indrekj force-pushed the opentelemetry_oban branch 2 times, most recently from a38db18 to 763b5d5 Compare September 21, 2021 13:20

indrekj marked this pull request as ready for review September 27, 2021 10:52

tsloughter approved these changes Nov 23, 2021

View reviewed changes

dvic approved these changes Nov 23, 2021

View reviewed changes

indrekj force-pushed the opentelemetry_oban branch from 483ed85 to 9ff9f64 Compare November 28, 2021 12:21

dvic reviewed Nov 30, 2021

View reviewed changes

indrekj force-pushed the opentelemetry_oban branch 3 times, most recently from d7e0794 to db0e1f1 Compare November 30, 2021 09:28

dvic reviewed Nov 30, 2021

View reviewed changes

indrekj force-pushed the opentelemetry_oban branch from db0e1f1 to ddf2618 Compare November 30, 2021 13:51

Merge branch 'main' into opentelemetry_oban

21f6007

tsloughter merged commit eecb238 into open-telemetry:main Dec 8, 2021

Add opentelemetry integration to Oban #6

Add opentelemetry integration to Oban #6

Uh oh!

Conversation

indrekj commented Aug 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

bryannaegele commented Aug 30, 2021

Uh oh!

indrekj commented Aug 30, 2021

Uh oh!

bryannaegele commented Aug 30, 2021

Uh oh!

indrekj commented Aug 31, 2021

Uh oh!

bryannaegele commented Sep 1, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indrekj commented Sep 2, 2021

Uh oh!

tsloughter commented Sep 3, 2021

Uh oh!

dvic commented Sep 13, 2021

Uh oh!

indrekj commented Sep 13, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dvic commented Sep 17, 2021

Uh oh!

dvic commented Sep 20, 2021

Uh oh!

indrekj commented Sep 20, 2021

Uh oh!

dvic commented Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indrekj commented Sep 21, 2021

Uh oh!

dvic commented Sep 21, 2021

Uh oh!

dvic commented Nov 23, 2021

Uh oh!

tsloughter commented Nov 23, 2021

Uh oh!

dvic commented Nov 24, 2021

Uh oh!

indrekj commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dvic commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

indrekj commented Nov 24, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dvic commented Nov 24, 2021

Uh oh!

tsloughter commented Nov 28, 2021

Uh oh!

indrekj commented Nov 28, 2021 via email

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

indrekj commented Aug 30, 2021 •

edited

Loading

bryannaegele commented Sep 1, 2021 •

edited

Loading

indrekj commented Sep 13, 2021 •

edited

Loading

dvic commented Sep 20, 2021 •

edited

Loading

indrekj commented Nov 24, 2021 •

edited

Loading

dvic commented Nov 24, 2021 •

edited

Loading

indrekj commented Nov 24, 2021 •

edited

Loading