feat: add Charmed Kafka plugins

[marcoppenheimer]

---

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

• Is the commit message split over multiple lines and hard-wrapped at 72 characters?
• Is the subject and message clear and concise?
• Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
• Does the commit contain a Signed-off-by: First Lastname email@example.com?
• Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
• Are all passwords or private data gathered by this PR obfuscated?

[packit-as-a-service]

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

• sudo dnf install -y 'dnf*-command(copr)'
• dnf copr enable packit/sosreport-sos-4214
• And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

[arif-ali]

other than the review comments, a few other things

1. There is no trigger for these plugins, which means these plugins will always run on any Ubuntu platform. Please add packages, services, files tuples depending on the plugin, so that the plugins are only enabled as required. This probably the reason why the CI has failed on only on the Ubuntu runs.
2. The 2 options imho don't add value here. Typically sos is collected for logs and data from a date to the current date. Also collecting logs by default from 1970-01-01 doesn't make sense. This would mean you're collecting all the data by default. It would be preferred, that this was behind the all_logs option. You can look at the maas plugin which parses the since option, which can then allow you to utilise this. So I would work this in this way.

[arif-ali]

would it be worth using the all_logs option to get all logs, and then have a default of 10000 lines?

[arif-ali]

again same comment as below

[arif-ali]

same comment here around all_logs, and maybe have a conditional here for that. There are examples of this in other plugins if you wanted to look

[arif-ali]

same comment I made below

[arif-ali]

same as above on all_logs option

[arif-ali]

snap logs by default only captures 10 lines, is it intended that you only want 10 lines

 snap logs --help
Usage:
  snap logs [logs-OPTIONS] <service>...

The logs command fetches logs of the given services and displays them in
chronological order.

[logs command options]
          --abs-time   Display absolute times (in RFC 3339 format). Otherwise, display relative times up to 60 days, then YYYY-MM-DD.
      -n=              Show only the given number of lines, or 'all'. (default: 10)
      -f               Wait for new lines and print them as they come in.

[logs command arguments]
  <service>:           A service specification, which can be just a snap name (for all services in the snap), or <snap>.<app> for a single service.

What I meant with all_logs option was that we could do something similar to

if self.getOption('all_logs'):
        self.add_cmd_output(
            "snap logs -n 50000 charmed-zookeeper.daemon",
            suggest_filename="snap_logs_charmed-zookeeper_daemon",
        )
else:
        self.add_cmd_output(
            "snap logs -n 500 charmed-zookeeper.daemon",
            suggest_filename="snap_logs_charmed-zookeeper_daemon",
        )

[marcoppenheimer]

@arif-ali - Thanks for the speedy review! I think I've addressed all points, let me know!

[TurboTurtle]

Thanks for the PR!

Some notes on these plugins below. Overall the plugins look beneficial and useful, but need some alignment with sos norms.

[TurboTurtle]

These can all be cleaned up, and made to fit line lengths:

endpoints = {
    'cruise-control-state': 'state?super_verbose=true',
    'cluster-state': 'kafka_cluster_state?verbose=true',
    'partition_load': 'partition_load',
    ...etc...
}

url = 'localhost:9090/kafkacruisecontrol'

for fname, api in endpoints.items():
    self.add_cmd_output(
        f"curl {self.credentials_args} {url}/{api}",
        suggest_filename=fname
    )

[marcoppenheimer]

I don't see issue with the verbosity here, as having each step explicit helps with potential future post-processing for individual steps if needed.

[marcoppenheimer]

@TurboTurtle - Have addressed the majority of your points, let me know if all is OK!

[marcoppenheimer]

Hey, gentle bump on this. I'm not sure why the CI is failing, any ideas?

[pmoravec]

Here are the backtraces that caused the AllPluginSmokeTest.test_no_exceptions_during_execution test to fail:

Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/sos/report/__init__.py", line 1270, in setup
    plug.setup()
  File "/tmp/cirrus-ci-build/sos/report/plugins/charmed_cruise_control.py", line 92, in setup
    f"curl {self.credentials_args} localhost:9090/kafkacruisecontrol/state?super_verbose=true",  # noqa: E501 # pylint:disable=line-too-long
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/tmp/cirrus-ci-build/sos/report/plugins/charmed_cruise_control.py", line 31, in credentials_args
    with open(
FileNotFoundError: [Errno 2] No such file or directory: '/var/snap/charmed-kafka/current/etc/cruise-control/cruisecontrol.credentials'

Traceback (most recent call last):
  File "/tmp/cirrus-ci-build/sos/report/__init__.py", line 1270, in setup
    plug.setup()
  File "/tmp/cirrus-ci-build/sos/report/plugins/charmed_kafka.py", line 103, in setup
    f"charmed-kafka.topics --describe {self.default_bin_args}",
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/tmp/cirrus-ci-build/sos/report/plugins/charmed_kafka.py", line 46, in default_bin_args
    if not self.bootstrap_server:
  File "/usr/lib/python3.10/functools.py", line 981, in __get__
    val = self.func(instance)
  File "/tmp/cirrus-ci-build/sos/report/plugins/charmed_kafka.py", line 34, in bootstrap_server
    with open(f"{PATHS['CONF']}/client.properties", encoding="utf-8") as f:
FileNotFoundError: [Errno 2] No such file or directory: '/var/snap/charmed-kafka/current/etc/kafka/client.properties'

Plus also:

[plugin:charmed_zookeeper] Error during plugin collections: list index out of range

that I commented directly in the code.

[marcoppenheimer]

@pmoravec - Thank you! CI fixed I think, we good to go?

[arif-ali]

@marcoppenheimer added some comments on log collection, otherwise looks good to me