"Could not set 'directory' on ensure: Permission denied" errors with rspec-puppet > 1.0.1 + puppet 3.7.3 future parser

[jhoblitt]

I made a big push yesterday to add the future parser to the travis matrix of all of my published puppet forge modules and discovered that the two modules that use rspec-puppet-augeas fail under the future parser with error messages like: Evaluation Error: Use of 'import' has been discontinued in favor of a manifest directory.

Direct links to travis runs for both modules with future parser errors:

https://travis-ci.org/jhoblitt/puppet-mdadm/builds/41427168
https://travis-ci.org/jhoblitt/puppet-autofsck/builds/41423244

Example (truncated) rspec errors with future parser enabled:

[future_parser] ~/github/puppet-mdadm $ FUTURE_PARSER="yes" bundle exec rake spec
HEAD is now at f8ce04a Merge branch 'release/master/three_dot_oh_dot_oh'
/home/jhoblitt/.rvm/rubies/ruby-2.1.5/bin/ruby -S rspec spec/unit/classes/mdadm_params_spec.rb spec/unit/classes/mdadm_spec.rb spec/unit/facts/mdadm_arrays_spec.rb spec/unit/facts/mdadm_spec.rb spec/unit/facts/mdadmversion_spec.rb --color
FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF..........

Failures:

  1) mdadm::params for osfamily RedHat 6.x 
     Failure/Error: it { should contain_class('mdadm::params') }
     Puppet::Error:
       Evaluation Error: Use of 'import' has been discontinued in favor of a manifest directory. See http://links.puppetlabs.com/puppet-import-deprecation at line 2:1 on node shinything.lan
     # ./spec/unit/classes/mdadm_params_spec.rb:15:in `block (4 levels) in <top (required)>'

  2) mdadm::params for osfamily RedHat 5.x 
     Failure/Error: it { should contain_class('mdadm::params') }
     Puppet::Error:
       Evaluation Error: Use of 'import' has been discontinued in favor of a manifest directory. See http://links.puppetlabs.com/puppet-import-deprecation at line 2:1 on node shinything.lan
     # ./spec/unit/classes/mdadm_params_spec.rb:21:in `block (4 levels) in <top (required)>'

  3) mdadm::params for osfamily RedHat 4.x should fail
     Failure/Error: expect { should contain_class('mdadm::params') }.
       expected Puppet::Error with message matching /not supported on Scientific 4/, got #<Puppet::Error: Evaluation Error: Use of 'import' has been discontinued in favor of a manifest directory. See http://links.puppetlabs.com/puppet-import-deprecation at line 2:1 on node shinything.lan> with backtrace:
         # ./spec/unit/classes/mdadm_params_spec.rb:28:in `block (5 levels) in <top (required)>'
         # ./spec/unit/classes/mdadm_params_spec.rb:28:in `block (4 levels) in <top (required)>'
     # ./spec/unit/classes/mdadm_params_spec.rb:28:in `block (4 levels) in <top (required)>'

  4) mdadm::params unsupported osfamily should fail
     Failure/Error: expect { should contain_class('mdadm::params') }.
       expected Puppet::Error with message matching /not supported on Debian/, got #<Puppet::Error: Evaluation Error: Use of 'import' has been discontinued in favor of a manifest directory. See http://links.puppetlabs.com/puppet-import-deprecation at line 2:1 on node shinything.lan> with backtrace:
         # ./spec/unit/classes/mdadm_params_spec.rb:43:in `block (4 levels) in <top (required)>'
         # ./spec/unit/classes/mdadm_params_spec.rb:43:in `block (3 levels) in <top (required)>'
     # ./spec/unit/classes/mdadm_params_spec.rb:43:in `block (3 levels) in <top (required)>'
...

[jhoblitt]

I just realized this may be an id10t error. I noticed that the rspec-puppet version pin in 0.3.0 was forcing bundler to downgrade it's version. I thought I'd handled that...

$ bundle list rspec-puppet
/home/jhoblitt/.rvm/gems/ruby-2.1.5/gems/rspec-puppet-0.1.6

[jhoblitt]

OK - don't know where my mind was yesterday...

Switching to my fork of rspec-puppet-augeas from last May that removes the rspec-puppet version pin and actually using a version of rspec-puppet > 1.0.1 (https://github.com/jhoblitt/puppet-mdadm/pull/2/files) resolves the "import" errors.

Sorry for the noise...

However, there's now a new error that occurs with and without the future parser enabled. It looks like a problem with test dir path construction.

Failures:

  1) mdadm on osfamily RedHat config_file_options => { mailaddr => foo } Augeas[mdadm.conf mailaddr] should change MAILADDR
     Failure/Error: should execute.with_change
     RuntimeError:
       Got 2 failure(s) while initializing: File[/etc/puppet/ssl]: change from absent to directory failed: Could not set 'directory' on ensure: Permission denied - /etc/puppet/ssl; File[/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/mdadm/lib:/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/stdlib/lib]: change from absent to directory failed: Cannot create /home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/mdadm/lib:/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/stdlib/lib; parent directory /home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/mdadm/lib:/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/stdlib does not exist
     # ./spec/unit/classes/mdadm_spec.rb:81

  2) mdadm on osfamily RedHat config_file_options => { mailaddr => foo } Augeas[mdadm.conf mailaddr] should change MAILADDR
     Failure/Error: should execute.with_change
     RuntimeError:
       Got 2 failure(s) while initializing: File[/etc/puppet/ssl]: change from absent to directory failed: Could not set 'directory' on ensure: Permission denied - /etc/puppet/ssl; File[/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/mdadm/lib:/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/stdlib/lib]: change from absent to directory failed: Cannot create /home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/mdadm/lib:/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/stdlib/lib; parent directory /home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/mdadm/lib:/home/jhoblitt/github/puppet-mdadm/spec/fixtures/modules/stdlib does not exist
     # ./spec/unit/classes/mdadm_spec.rb:90

Finished in 3.08 seconds
50 examples, 2 failures

[domcleal]

Those errors are puzzling. It seems r-p-a hits them because it's actually applying a catalog, which rspec-puppet doesn't do. This triggers a new code path inside Puppet to initialise settings...

/home/dcleal/.rvm/gems/ruby-2.0.0-p353@tmp/gems/puppet-3.7.3/lib/puppet/settings.rb:999:in `block in use'
/home/dcleal/.rvm/gems/ruby-2.0.0-p353@tmp/gems/puppet-3.7.3/lib/puppet/resource/catalog.rb:181:in `apply'
/home/dcleal/.rvm/gems/ruby-2.0.0-p353@tmp/gems/puppet-3.7.3/lib/puppet/settings.rb:989:in `use'
/home/dcleal/.rvm/gems/ruby-2.0.0-p353@tmp/gems/puppet-3.7.3/lib/puppet/util/storage.rb:45:in `load'
/home/dcleal/.rvm/gems/ruby-2.0.0-p353@tmp/gems/puppet-3.7.3/lib/puppet/resource/catalog.rb:163:in `apply'
/home/dcleal/.rvm/gems/ruby-2.0.0-p353@tmp/bundler/gems/rspec-puppet-augeas-6152e283a65f/lib/rspec-puppet-augeas/fixtures.rb:47:in `apply'

• I wonder if File[/etc/puppet/ssl] could be resolved in rspec-puppet by setting confdir or ssldir to a temporary directory in the same way as https://github.com/rodjek/rspec-puppet/blob/master/lib/rspec-puppet/support.rb#L119-L120?
• the directory error is due to https://github.com/rodjek/rspec-puppet/blob/master/lib/rspec-puppet/support.rb#L140 which sets up a colon-separated list of libdirs, which seems to be accepted in some parts of Puppet, but I suppose it's not parsed in Puppet settings and is handled as a single path. Puppet bug perhaps?

[jhoblitt]

I suspect this might be related to puppetlabs/puppet#3114 but I haven't had the chance to investigate (holiday weekend in the US).

[joshcooper]

the directory error is due to https://github.com/rodjek/rspec-puppet/blob/master/lib/rspec-puppet/support.rb#L140 which sets up a colon-separated list of libdirs, which seems to be accepted in some parts of Puppet, but I suppose it's not parsed in Puppet settings and is handled as a single path. Puppet bug perhaps?

Yep, puppet is not consistent about handling a list of libdirs. I posted to puppet-dev about this. And this issue shows that if you ever do set libdir to multiple directories, and actually try to "use" settings, like when running puppet apply, then puppet will interpret :libdir as a single directory, try to create/manage it as part of its settings catalog, and fail.

[domcleal]

I've pushed a fix for the permission error (which is the same as #14 I reckon) to the 'future' branch, which I'll merge to master as soon as the rspec-puppet 2.0.0 gem is pushed and do a 0.4.0 release. That should get most things running again.

Once @joshcooper's LOAD_PATH fix is also merged and released in rspec-puppet, then it'll fix the multi-libdir case too (as in your mdadm module).

[domcleal]

0.4.0's released now which should fix the permissions problem, just leaving the multi-libdir case as the rspec-puppet PR's still open.

[jhoblitt]

I think this is real progress, I'm down to errors which I assume are bubbling up from trying to write to /dev/null.

     Failure/Error: should execute.with_change
     NoMethodError:
       Could not create resources for managing Puppet's files and directories in sections [:main]: undefined method `exists?' for nil:NilClass
       undefined method `exists?' for nil:NilClass
     Shared Example Group: "debian" called from ./spec/classes/autofsck_spec.rb:77
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/type/user.rb:382:in `exists?'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings.rb:800:in `service_user_available?'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings/file_setting.rb:52:in `safe_to_use_settings_value?'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings/file_setting.rb:43:in `value'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings/file_setting.rb:106:in `owner'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings/file_setting.rb:160:in `to_resource'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings.rb:916:in `block in to_catalog'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings.rb:912:in `each'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings.rb:912:in `to_catalog'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/settings.rb:982:in `use'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/util/storage.rb:45:in `load'
     # ./.bundle/ruby/2.1.0/gems/puppet-3.7.5/lib/puppet/resource/catalog.rb:163:in `apply'
     # ./.bundle/ruby/2.1.0/gems/rspec-puppet-augeas-0.4.0/lib/rspec-puppet-augeas/fixtures.rb:51:in `apply'
     # ./.bundle/ruby/2.1.0/gems/rspec-puppet-augeas-0.4.0/lib/rspec-puppet-augeas/resource.rb:17:in `initialize'
     # ./.bundle/ruby/2.1.0/gems/rspec-puppet-augeas-0.4.0/lib/rspec-puppet-augeas/example/run_augeas_example_group.rb:69:in `new'
     # ./.bundle/ruby/2.1.0/gems/rspec-puppet-augeas-0.4.0/lib/rspec-puppet-augeas/example/run_augeas_example_group.rb:69:in `subject'
     # ./spec/classes/autofsck_spec.rb:32:in `block (4 levels) in <top (required)>'

[jhoblitt]

I get the same failure with @joshcooper 's rspec-puppet PR. Eg.

gem 'rspec-puppet',
  :git => 'https://github.com/joshcooper/rspec-puppet.git',
  :ref => '9a178275bbe7d6ed5fb985ea11ae56e2d12f1a26'

It looks like PUP-3336 was merged into master but not stable.

@joshcooper - is there any hope of that getting back merged for 3.7.x?

[joshcooper]

@jhoblitt I'm not sure we can backport PUP-3336 to 3.7.x. It changes how puppet interprets its libdir setting, which I think is a breaking change for module developers that are used to testing their modules by specifying multiple directories.

The error undefined method 'exists?' usually means you're running as non-root, your system user/group binaries, e.g. useradd, have permissions 0750, and so there is no suitable user/group provider. This is a problem for puppet because it needs a user/group provider to manage its internal file/directory settings, e.g. setting permissions on the ssldir containing your private key. This issue is filed as https://tickets.puppetlabs.com/browse/PUP-1162. Can you try running the test as root?

Ideally when puppet is running as non-root, it would only attempt to manage the mode of its internal settings, and not owner & group (so it wouldn't require a user & group provider).

[jhoblitt]

@joshcooper I suspected that would be the concern but isn't libdir handling already a regression in 3.x? I don't see how rspec-puppet could be changed to work with both 3.7.x and 4.x without modifications on the puppet side.

The spec tests do indeed pass as root.

Ps - bundler gives this nice warning...

Don't run Bundler as root. Bundler can ask for sudo if it is needed, and
installing your bundle as root will break this application for all non-root
users on this machine.

[joshcooper]

isn't libdir handling already a regression in 3.x

It is broken if you specify multiple directories in libdir and then try to run puppet, e.g. puppet apply. But it has historically "sort of worked" if you are testing modules, and are using the setting to specify the autoloader search path for loading custom types, providers, etc. I say "sort of" because it works if you have a simple type, etc, but not if your type tries to load helper code that is not in $LOAD_PATH.

For the 3.x vs 4.x question, I think rodjek/rspec-puppet#237 should work for both. As in, so long as every module's lib dir is in the ruby $LOAD_PATH, then all versions of puppet can load transitive dependencies for custom types, providers, etc.

[choffee]

As a bit of a workaround running the tests with fakeroot seems to work fine for me.

fakeroot bundle exec rake spec