[juju] Add plugin option for Juju state reporting #3803

MichaelThamm · 2024-10-10T18:25:59Z

This change modifies the Juju plugin to optionally capture Juju state (controllers, models, applications, units) information. The capture assumes that juju is installed on the machine where sos is called, and that the juju user has superuser privilege to the current (or requested) controllers.

This option is disabled by default because, while the information can be very useful, the collection acts on the live Juju state.

Use the plugin with the feature:

sos report --only-plugin juju -k juju.juju-state=True

Run with a specific user (who has access to the controllers specified). This defaults to ubuntu:

-k juju.juju-user=super_admin

If necessary, you can filter by controllers or models with cluster options.

-k juju.controllers="controller_a controller_b"
-k juju.models="controller_a:model_x controller_b:model_y"

If nothing is supplied for either the controllers or models options, the report will include all state information for them respectively.

Please place an 'X' inside each '[]' to confirm you adhere to our Contributor Guidelines

Is the commit message split over multiple lines and hard-wrapped at 72 characters?
Is the subject and message clear and concise?
Does the subject start with [plugin_name] if submitting a plugin patch or a [section_name] if part of the core sosreport code?
Does the commit contain a Signed-off-by: First Lastname [email protected]?
Are any related Issues or existing PRs properly referenced via a Closes (Issue) or Resolved (PR) line?
Are all passwords or private data gathered by this PR obfuscated?
Add tests to ensure results are scrubbed / functioning

packit-as-a-service · 2024-10-10T18:28:40Z

Congratulations! One of the builds has completed. 🍾

You can install the built RPMs by following these steps:

sudo yum install -y dnf-plugins-core on RHEL 8
sudo dnf install -y dnf-plugins-core on Fedora
dnf copr enable packit/sosreport-sos-3803
And now you can install the packages.

Please note that the RPMs should be used only in a testing environment.

sos/report/plugins/juju.py

arif-ali

As well as turboturtle comments, can we also look at the flake/pylint issues as well is any updated PR

sos/report/plugins/juju.py

jcastill · 2024-10-11T19:06:03Z

A couple of very small comments from me.
The DCO is still failing, I think you are missing '-s' while running 'git commit'.
Last thing, please make sure you squash the two commits you have at the moment into one.

jcastill · 2024-10-13T17:44:09Z

@MichaelThamm remember to squash all the commits into one and sign it

MichaelThamm · 2024-10-14T17:29:54Z

@arif-ali or @jcastill I imagine that I need to obfuscate some of the following information which is included in the juju controllers command output:

        "lxd": {
            "user": "admin",
            "recent-server": IP:Port,
            "uuid": "ac03821c-361e-4daf-83c9-a712cd56b8d4",
            "api-endpoints": [IP:Port],
            "ca-cert": "-----BEGIN CERTIFICATE-----
            ...
            -----END CERTIFICATE-----",
        },

Is there a need to obfuscate these things or is it just passwords, tokens, keys as mentioned in obfuscate?

arif-ali · 2024-10-14T17:34:56Z

@MichaelThamm The IP addresses and endpoints, not so, as they would be typically handled by --clean or sos clean, if indeed these needed to be obfuscated.

In terms of the certs, you can get the example of the sunbeam.py plugin, where we just needs self.do_cmd_private_sub('juju controllers') in the postproc() function. This will automatically obfuscate any command that showed the cert.

sos/report/plugins/juju.py

MichaelThamm · 2024-10-15T14:18:55Z

@arif-ali

In terms of the certs, you can get the example of the sunbeam.py plugin, where we just needs self.do_cmd_private_sub('juju controllers') in the postproc() function. This will automatically obfuscate any command that showed the cert.

When I run: self.do_cmd_private_sub('juju controllers --format=json') I see this in the logs:

2024-10-15 10:13:54,360 DEBUG: [plugin:juju] substituting '-----SCRUBBED' for '----(?:-| )BEGIN.*?----(?:-| )END' in commands matching '*juju controllers --format=json*'

However, I do not see results in the commands dir of the sos report tar.xz file

This method does not have a filename parameter either. Not sure how to continue with this?

arif-ali · 2024-10-15T14:27:31Z

When I run: self.do_cmd_private_sub('juju controllers --format=json') I see this in the logs:
2024-10-15 10:13:54,360 DEBUG: [plugin:juju] substituting '-----SCRUBBED' for '----(?:-| )BEGIN.*?----(?:-| )END' in commands matching '*juju controllers --format=json*'
However, I do not see results in the commands dir of the sos report tar.xz file

This method does not have a filename parameter either. Not sure how to continue with this?

You can just specify juju controllers in the function, and it will do a match. In the sunbeam.py, I did the same with doing a self.collect_cmd_output("juju controllers --format json"), and then used the juju controllers in the self.do_cmd_private_sub()

Signed-off-by: Michael Thamm <[email protected]>

MichaelThamm · 2024-10-15T14:56:00Z

I think the PR is almost ready for review, but I noticed we have tests for the Juju plugin. Is there any need to update this since the test for scrubbing seems to be sufficient? I also tested the scrubbing for juju controllers manually and it passes.

TurboTurtle · 2024-10-15T22:28:51Z

sos/report/plugins/juju.py

+        )
+        if controllers_json["status"] == 0:
+            desired_controllers = set(
+                self.get_option("controllers").split(" ")


We generally have plugins use : as a delimiter in plugin options that can take a list (as the use of commas gets murky with argparse and how plugin options are....parsed.

The only problem with this is for the Juju model synatx it can be:

The current controller's model -> model_a

Or it can be a specific controller and model -> controller_x:model_a

For this reason I chose to split via empty space since models and controllers are not allowed to have spaces in their names according to Juju.

i.e. controller one -> not valid name

Would this be a reasonable exception to the Sos plugin standard?

If it's an expectation that users would pass something like -k juju.controllers="foo:bar", then yeah it's totally reasonable to use something else. We should just make note of that, ideally in both the plugin option description (for sos report -l output) and the docstring for the plugin (as the docstring gets used in sos help output, e.g. sos help report.plugins.juju - see the kernel plugin docstring for a good example).

I guess there is no reason why we couldn't use a colon delimiter for the controllers option. However, it is required for the models option (as I mentioned above) so I think it makes sense to be consistent within the scope of this plugin to keep the white-space delimiter rather than colon. What do you think?

I added a docstring and the PluginOpts have description args. Is this enough to show up in the sos report -l output? I tested this locally and I could not get my Juju plugin options to show up in this output and I have no idea why.

Reference

This is what I get when I run your changes:

# bin/sos report -l | grep juju juju Juju orchestration tool juju.juju-state off Include Juju state in the report juju.juju-user ubuntu Juju client user. juju.controllers Collect Juju state for specified controllers. Uses a whitespace delimited list. juju.models Collect Juju state for specified models. Uses a whitespace delimited list.

And this is what I get when running sos help:

Plugin Options These options may be toggled or changed using '-k juju.option_name=$value' Option Name Default Description juju-state False/Off Include Juju state in the report juju-user ubuntu Juju client user. controllers None/Unset Collect Juju state for specified controllers. Uses a whitespace delimited list. models None/Unset Collect Juju state for specified models. Uses a whitespace delimited list.

Regarding why the options are not showing up in the output locally, perhaps it's because the machine where you are running doesn't have the /var/log/juju file present? Since that's the activation condition for this plugin:

files = ('/var/log/juju',)

As far as I understand it, if it's not present then the plugin should show as inactive and the options won't be printed.

TurboTurtle · 2024-10-15T22:31:36Z

sos/report/plugins/juju.py

+
+        # Specific models
+        if self.get_option("models"):
+            for model in self.get_option("models").split(" "):


Ditto, we prefer plugin options to use : as a delimiter. We may need to add this to the plugin guide, I'll check this shortly.

sos/report/plugins/juju.py

TurboTurtle · 2024-10-15T22:46:32Z

sos/report/plugins/juju.py

+        if self.get_option("controllers") and self.get_option("models"):
+            self._log_warn(
+                "Options: controllers, models are mutually exclusive. "
+                "Will not collect Juju information."
+            )
+            return


I'm assuming these are mutually exclusive within the Juju stack, but would we not anticipate some (admittedly niche) use case where an sos end user isn't sure if a particular host is using one or the other, and wants to pass e.g. -k juju.controllers=foo,juju.models=bar to get either without needing to know the specific host configuration? I'm mainly thinking of some sort of automation-driven sos collection here, but I could be way off with that idea.

According to the the Juju CLI:

juju status --help

-m, --model (= "") Model to operate in. Accepts [<controller name>:]<model name>|<model UUID>

There is no controller option elsewhere so I think it is reasonable to be opinionated and keep the 2 options mutually exclusive. As an added bonus, this simplifies the logic in the plugin.

Note: I tested this with model UUID and model name. Both work fine.

jcastill · 2024-11-18T21:26:37Z

@MichaelThamm remember to squash the three commits into one, and to sign the commit (that is what's causing the DCO failure).

TurboTurtle reviewed Oct 10, 2024

View reviewed changes

sos/report/plugins/juju.py Outdated Show resolved Hide resolved

arif-ali requested changes Oct 11, 2024

View reviewed changes

sos/report/plugins/juju.py Outdated Show resolved Hide resolved

jcastill suggested changes Oct 11, 2024

View reviewed changes

sos/report/plugins/juju.py Outdated Show resolved Hide resolved

sos/report/plugins/juju.py Show resolved Hide resolved

jcastill approved these changes Oct 13, 2024

View reviewed changes

MichaelThamm force-pushed the report-juju branch from 60d0c0e to 9d36d66 Compare October 14, 2024 15:30

MichaelThamm changed the title ~~Feat: Report on juju environment~~ [Juju] Add plugin option for Juju state reporting Oct 14, 2024

MichaelThamm changed the title ~~[Juju] Add plugin option for Juju state reporting~~ [juju] Add plugin option for Juju state reporting Oct 14, 2024

arif-ali reviewed Oct 14, 2024

View reviewed changes

[juju] Add plugin option for Juju state reporting

f389bfa

Signed-off-by: Michael Thamm <[email protected]>

MichaelThamm force-pushed the report-juju branch from 045cef0 to f389bfa Compare October 15, 2024 14:49

MichaelThamm marked this pull request as ready for review October 15, 2024 15:41

TurboTurtle reviewed Oct 15, 2024

View reviewed changes

MichaelThamm added 2 commits November 18, 2024 15:34

Add docstring and cleanup for Juju plugin

04fc3ce

chore: lint

9e4c546

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[juju] Add plugin option for Juju state reporting #3803

[juju] Add plugin option for Juju state reporting #3803

MichaelThamm commented Oct 10, 2024 •

edited

Loading

packit-as-a-service bot commented Oct 10, 2024

arif-ali left a comment

jcastill commented Oct 11, 2024

jcastill commented Oct 13, 2024

MichaelThamm commented Oct 14, 2024

arif-ali commented Oct 14, 2024

MichaelThamm commented Oct 15, 2024

arif-ali commented Oct 15, 2024

MichaelThamm commented Oct 15, 2024

TurboTurtle Oct 15, 2024

MichaelThamm Oct 16, 2024 •

edited

Loading

TurboTurtle Oct 16, 2024

MichaelThamm Nov 18, 2024

MichaelThamm Nov 18, 2024

jcastill Nov 18, 2024

jcastill Nov 18, 2024

TurboTurtle Oct 15, 2024

TurboTurtle Oct 15, 2024

MichaelThamm Nov 18, 2024 •

edited

Loading

jcastill commented Nov 18, 2024

[juju] Add plugin option for Juju state reporting #3803

Are you sure you want to change the base?

[juju] Add plugin option for Juju state reporting #3803

Conversation

MichaelThamm commented Oct 10, 2024 • edited Loading

packit-as-a-service bot commented Oct 10, 2024

arif-ali left a comment

Choose a reason for hiding this comment

jcastill commented Oct 11, 2024

jcastill commented Oct 13, 2024

MichaelThamm commented Oct 14, 2024

arif-ali commented Oct 14, 2024

MichaelThamm commented Oct 15, 2024

arif-ali commented Oct 15, 2024

MichaelThamm commented Oct 15, 2024

Choose a reason for hiding this comment

MichaelThamm Oct 16, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelThamm Nov 18, 2024 • edited Loading

Choose a reason for hiding this comment

jcastill commented Nov 18, 2024

MichaelThamm commented Oct 10, 2024 •

edited

Loading

MichaelThamm Oct 16, 2024 •

edited

Loading

MichaelThamm Nov 18, 2024 •

edited

Loading