sensu_agent resource attempting to do impossible reloads on upgrade

[wduncanfraser]

It looks like #132 has caused this cookbook to start failing converging for me when Sensu is being upgraded.

There is no reload operation defined in the systemd unit file shipped with the Sensu agent (At least on CentOS 7), which causes this to fail every time.

It looks like the original intent of the above PR was to fix the unit file itself not getting reloaded. This needs to go through a daemon-reload or through systemd generators. "reload" on a systemd unit has to be implemented in the unit, and is for the unit to reload itself if it can, without restarting.

Chef Version

Chef: 15

Input

sensu_agent 'default' do
version node['ac_sensu_agent']['version']
end

Output

* yum_package[sensu-go-agent] action install
      - install version 6.2.7-4449 of package sensu-go-agent
    * systemd_unit[sensu-agent] action reload

      ================================================================================
      Error executing action `reload` on resource 'systemd_unit[sensu-agent]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Expected process to exit with [0], but received '3'
      ---- Begin output of ["/usr/bin/systemctl", "--system", "reload", "sensu-agent"] ----
      STDOUT:
      STDERR: Failed to reload sensu-agent.service: Job type reload is not applicable for unit sensu-agent.service.
      See system logs and 'systemctl status sensu-agent.service' for details.
      ---- End output of ["/usr/bin/systemctl", "--system", "reload", "sensu-agent"] ----
      Ran ["/usr/bin/systemctl", "--system", "reload", "sensu-agent"] returned 3

      Resource Declaration:
      ---------------------
      # In /var/chef/cache/cookbooks/sensu-go/resources/agent.rb

       59:     systemd_unit 'sensu-agent' do
       60:       action :nothing
       61:     end
       62:

      Compiled Resource:
      ------------------
      # Declared in /var/chef/cache/cookbooks/sensu-go/resources/agent.rb:59:in `block in class_from_file'

      systemd_unit("sensu-agent") do
        action [:nothing]
        default_guard_interpreter :default
        declared_type :systemd_unit
        cookbook_name "ac_sensu_agent"
        unit_name "sensu-agent"
      end

      System Info:
      ------------
      chef_version=15.10.12
      platform=centos
      platform_version=7.9.2009
      ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
      program_name=/usr/bin/chef-client
      executable=/opt/chef/bin/chef-client


    ================================================================================
    Error executing action `install` on resource 'sensu_agent[default]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    systemd_unit[sensu-agent] (/var/chef/cache/cookbooks/sensu-go/resources/agent.rb line 59) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '3'
    ---- Begin output of ["/usr/bin/systemctl", "--system", "reload", "sensu-agent"] ----
    STDOUT:
    STDERR: Failed to reload sensu-agent.service: Job type reload is not applicable for unit sensu-agent.service.
    See system logs and 'systemctl status sensu-agent.service' for details.
    ---- End output of ["/usr/bin/systemctl", "--system", "reload", "sensu-agent"] ----
    Ran ["/usr/bin/systemctl", "--system", "reload", "sensu-agent"] returned 3

    Resource Declaration:
    ---------------------
    suppressed sensitive resource output

    Compiled Resource:
    ------------------
    suppressed sensitive resource output

    System Info:
    ------------
    chef_version=15.10.12
    platform=centos
    platform_version=7.9.2009
    ruby=ruby 2.6.6p146 (2020-03-31 revision 67876) [x86_64-linux]
    program_name=/usr/bin/chef-client
    executable=/opt/chef/bin/chef-client

Impact

Cookbook failing to converge on agent upgrade.

Expected Behavior

Succesful upgrade of Sensu agents.

Actual Behavior

Converge failure due to undefined reload operation in systemd unit.

[kmf]

Also affected

[ValkyrieOps]

For some reason I have not run into this when upgrading agents. Does it make sense to revert the change from #132 and replace it with something like this?

  package 'sensu-go-agent' do
      if latest_version?(new_resource.version)
        action :upgrade
        notifies :run, 'execute[daemon-reload]', :immediately
        notifies :restart, 'service[sensu-agent]', :delayed
      else
        version new_resource.version
        action :install
      end
    end

    execute 'daemon-reload' do
      command 'systemctl daemon-reload'
      action :nothing
    end