Make ParsedFile provider work with composite namevars #9130

ekohl · 2023-10-23T19:58:09Z

Previously it was assumed that the name was the key. This changes the implementation to instead match on the key attributes. This allows it to work with composite namevars. While there was previously code for multiple target support, this makes it possible to use the same key in multiple files.

It also gets rid of some code that wasn't needed.

To achieve this, it also needs to use resource titles as identifiers in preloading

When using composite namevars for resources then resource.name always nil. Prior to this it was used as a key in a hash when prefetching resources. By using the title it should work, since that's guaranteed to be set and unique.

This can break providers where the name is different from the title.

Right now there are no tests, but I tested it locally with puppetlabs-postgresql.

When using composite namevars for resources then `resource.name` always `nil`. Prior to this it was used as a key in a hash when prefetching resources. By using the title it should work, since that's guaranteed to be set and unique. This can break providers where the name is different from the title.

Previously it was assumed that the name was the key. This changes the implementation to instead match on the key attributes. This allows it to work with composite namevars. While there was previously code for multiple target support, this makes it possible to use the same key in multiple files. It also gets rid of some code that wasn't needed.

puppetlabs-jenkins · 2023-10-23T19:58:13Z

Can one of the admins verify this patch?

ekohl · 2023-10-23T20:05:12Z

lib/puppet/provider/parsedfile.rb

@@ -449,7 +440,7 @@ def flush
    # If the target isn't set, then this is our first modification, so
    # mark it for flushing.
    unless @property_hash[:target]
-      @property_hash[:target] = @resource.should(:target) || self.class.default_target


I think this may be because I started to use target as a parameter instead of a property.

ekohl · 2023-10-23T20:05:31Z

lib/puppet/provider/parsedfile.rb

@@ -429,17 +422,15 @@ def create
      end
    end
    mark_target_modified
-    (@resource.class.name.to_s + "_created").intern


AFAIK create is supposed to not return anything.

ekohl · 2023-10-23T20:05:52Z

lib/puppet/provider/parsedfile.rb

  end

  def destroy
    # We use the method here so it marks the target as modified.
    self.ensure = :absent
-    (@resource.class.name.to_s + "_deleted").intern


AFAIK destroy it not supposed to return anything

ekohl · 2023-10-23T20:07:39Z

lib/puppet/provider/parsedfile.rb

@@ -465,13 +456,7 @@ def initialize(record)
    # The 'record' could be a resource or a record, depending on how the provider
    # is initialized.  If we got an empty property hash (probably because the resource
    # is just being initialized), then we want to set up some defaults.
-    @property_hash = self.class.record?(resource[:name]) || {:record_type => self.class.name, :ensure => :absent} if @property_hash.empty?


In my testing it's not needed to set the properties when a Puppet::Resource is passed in, but it is needed to set the record_type to skip things. Perhaps skip_record? isn't needed at all anymore and can be dropped. Then this line can also be dropped.

ekohl · 2023-10-23T20:08:17Z

lib/puppet/provider/parsedfile.rb

  end

-  attr_accessor :property_hash


Not sure why it needs to be an accessor. I think it can only break things.

I agree in principal, but it's referenced by subclasses of ParsedFile like https://github.com/puppetlabs/puppetlabs-sshkeys_core/blob/fbd0c6dc9ca7314a567b6873cb66e2adf52af30b/lib/puppet/provider/sshkey/parsed.rb#L32C8-L32C21 and is therefore part of the parsed file API.

SimonHoenscheid · 2023-10-23T20:23:24Z

@ekohl I have just a question out of curiosity: I tried to patch the postgresql_conf provider to work with multiple files and was told that the cron type, which is also based on ParsedFile is able to do this. But I never found the code which does it. Is there a working patch in the cron type/provider?

ekohl · 2023-10-23T22:19:56Z

I did not think about that, but just to be explicit: you're talking about https://github.com/puppetlabs/puppetlabs-cron_core? I see it does override some methods, like resource_for_record, match and targets. It also implements prefetch_hook. Those do appear to be the broken methods and it may be possible to apply the same in puppetlabs-postgresql.

It would also be an interesting test to see how much of that can be dropped with this patch.

SimonHoenscheid · 2023-10-24T07:38:17Z

Yes, I was referring to https://github.com/puppetlabs/puppetlabs-cron_core

joshcooper · 2023-10-24T17:59:12Z

lib/puppet/transaction.rb

@@ -361,7 +361,7 @@ def resources_by_provider(type_name, provider_name)
      @catalog.vertices.each do |resource|
        if resource.class.attrclass(:provider)
          prov = resource.provider && resource.provider.class.name
-          @resources_by_provider[resource.type][prov][resource.name] = resource
+          @resources_by_provider[resource.type][prov][resource.title] = resource


Since resource title doesn't always equal name, I'm pretty sure this will be break something.

Looks like puppet originally prefetched using title (see commit c35d07b)

prefetchers[provider.class][resource.title] = resource

Later there was a bug reported in the mount provider redmine #2013 And puppet was changed to use name in 9577d3a as "ParsedFile types seem to be the main one that suffers from (mismatch between type and title)"

- prefetchers[provider.class][resource.title] = resource + prefetchers[provider.class][resource.name] = resource

I'm not convinced that changing the transaction to use name was the right fix (as opposed to a bug in the mount provider), but that ship sailed awhile ago. We'll need a different way for parsed file providers to opt-into the new behavior.

joshcooper · 2023-10-24T18:00:39Z

lib/puppet/provider/parsedfile.rb

-    return nil unless @records
-    @records.find { |r| r[:name] == name }
-  end
-


This is called in

puppet/lib/puppet/provider/parsedfile.rb

Line 468 in 945beb6

@property_hash = self.class.record?(resource[:name]) || {:record_type => self.class.name, :ensure => :absent} if @property_hash.empty?

joshcooper · 2024-05-29T19:55:47Z

I just ran into this issue while investigating composite name var support for host_core to enable managing both ipv4 and v6 entries. I'll take a closer look.

ekohl

@joshcooper This originally came up in puppetlabs-postgresql, but I don't have time to finish this so as with my other PRs you should feel free to take over. I'll happily review.

ekohl · 2024-06-21T15:06:06Z

lib/puppet/provider/parsedfile.rb

+    resources&.each do |name, resource|
+      value = resource[:target]
+      if value
+        targets << value
      end
    end


This can probably be optimized by relying on Ruby 2.7+

targets += resources.filter_map { |_name, resource| resource[:target] } if resources

ekohl added 2 commits October 23, 2023 21:49

ekohl mentioned this pull request Oct 23, 2023

Extend postgresql_conf to support multiple config files using ParsedFile puppetlabs/puppetlabs-postgresql#1542

Draft

ekohl commented Oct 23, 2023

View reviewed changes

joshcooper reviewed Oct 24, 2023

View reviewed changes

joshcooper added the enhancement New feature or request label Nov 15, 2023

ekohl commented Jun 21, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make ParsedFile provider work with composite namevars #9130

Make ParsedFile provider work with composite namevars #9130

ekohl commented Oct 23, 2023

puppetlabs-jenkins commented Oct 23, 2023

ekohl Oct 23, 2023

ekohl Oct 23, 2023

ekohl Oct 23, 2023

ekohl Oct 23, 2023

ekohl Oct 23, 2023

joshcooper Oct 24, 2023

SimonHoenscheid commented Oct 23, 2023

ekohl commented Oct 23, 2023

SimonHoenscheid commented Oct 24, 2023

joshcooper Oct 24, 2023

joshcooper May 30, 2024

joshcooper Oct 24, 2023

joshcooper commented May 29, 2024

ekohl left a comment

ekohl Jun 21, 2024

Make ParsedFile provider work with composite namevars #9130

Are you sure you want to change the base?

Make ParsedFile provider work with composite namevars #9130

Conversation

ekohl commented Oct 23, 2023

puppetlabs-jenkins commented Oct 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonHoenscheid commented Oct 23, 2023

ekohl commented Oct 23, 2023

SimonHoenscheid commented Oct 24, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshcooper commented May 29, 2024

ekohl left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment