Not clear failure descritpion: Failed to create temporary directory

[zhukovgreen]

SUMMARY

I have two very similar VMs and for one VM the ansible connects normally, but for another raising the following error. The traceback is not helpful to me.

The command and output is the following:

ansible-playbook  --extra-vars='repo_root=/var/tmp/tmt/run-070/plans/integration/inhibit-if-kmods-is-not-supported/vm_oracle8/good/discover/default/tests' -c community.libvirt.libvirt_qemu -i c2r_oracle8_template-clone, /home/azhukov/Documents/convert2rhel/tests/ansible_collections/basic_setup.yml -vvv

ansible-playbook 2.10.7
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/azhukov/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/azhukov/.pyenv/versions/3.9.1/lib/python3.9/site-packages/ansible
  executable location = /home/azhukov/.pyenv/versions/3.9.1/bin/ansible-playbook
  python version = 3.9.1 (default, Jan 14 2021, 15:48:15) [GCC 10.2.0]
Using /etc/ansible/ansible.cfg as config file
setting up inventory plugins
Parsed c2r_oracle8_template-clone, inventory source with host_list plugin
Loading callback plugin default of type stdout, v2.0 from /home/azhukov/.pyenv/versions/3.9.1/lib/python3.9/site-packages/ansible/plugins/callback/default.py
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: basic_setup.yml ******************************************************
Positional arguments: /home/azhukov/Documents/convert2rhel/tests/ansible_collections/basic_setup.yml
verbosity: 4
connection: community.libvirt.libvirt_qemu
timeout: 10
become_method: sudo
tags: ('all',)
inventory: ('c2r_oracle8_template-clone,',)
extra_vars: ('repo_root=/var/tmp/tmt/run-070/plans/integration/inhibit-if-kmods-is-not-supported/vm_oracle8/good/discover/default/tests',)
forks: 5
1 plays in /home/azhukov/Documents/convert2rhel/tests/ansible_collections/basic_setup.yml

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
task path: /home/azhukov/Documents/convert2rhel/tests/ansible_collections/basic_setup.yml:1
Loading collection community.libvirt from /home/azhukov/.pyenv/versions/3.9.1/lib/python3.9/site-packages/ansible_collections/community/libvirt
<c2r_oracle8_template-clone> CONNECT TO qemu:///system
<c2r_oracle8_template-clone> FIND DOMAIN c2r_oracle8_template-clone
<c2r_oracle8_template-clone> ESTABLISH community.libvirt.libvirt_qemu CONNECTION
<c2r_oracle8_template-clone> EXEC /bin/sh -c 'echo ~ && sleep 0'
<c2r_oracle8_template-clone> GA send: {"execute": "guest-exec", "arguments": {"path": "/bin/sh", "capture-output": true, "arg": ["-c", "echo ~ && sleep 0"]}}
<c2r_oracle8_template-clone> GA return: {'return': {'pid': 1007}}
<c2r_oracle8_template-clone> GA send: {"execute": "guest-exec-status", "arguments": {"pid": 1007}}
<c2r_oracle8_template-clone> GA return: {'return': {'exited': False}}
<c2r_oracle8_template-clone> GA return: {'return': {'exitcode': 0, 'out-data': 'L3Jvb3QK', 'exited': True}}
<c2r_oracle8_template-clone> GA stdout: /root
<c2r_oracle8_template-clone> GA stderr: 
<c2r_oracle8_template-clone> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1617899822.4112039-61006-63082701834685 `" && echo ansible-tmp-1617899822.4112039-61006-63082701834685="` echo /root/.ansible/tmp/ansible-tmp-1617899822.4112039-61006-63082701834685 `" ) && sleep 0'
<c2r_oracle8_template-clone> GA send: {"execute": "guest-exec", "arguments": {"path": "/bin/sh", "capture-output": true, "arg": ["-c", "( umask 77 && mkdir -p \"` echo /root/.ansible/tmp `\"&& mkdir \"` echo /root/.ansible/tmp/ansible-tmp-1617899822.4112039-61006-63082701834685 `\" && echo ansible-tmp-1617899822.4112039-61006-63082701834685=\"` echo /root/.ansible/tmp/ansible-tmp-1617899822.4112039-61006-63082701834685 `\" ) && sleep 0"]}}
<c2r_oracle8_template-clone> GA return: {'return': {'pid': 1008}}
<c2r_oracle8_template-clone> GA send: {"execute": "guest-exec-status", "arguments": {"pid": 1008}}
<c2r_oracle8_template-clone> GA return: {'return': {'exited': False}}
<c2r_oracle8_template-clone> GA return: {'return': {'exitcode': 1, 'err-data': 'bWtkaXI6IGNhbm5vdCBjcmVhdGUgZGlyZWN0b3J5IOKAmC9yb290Ly5hbnNpYmxl4oCZOiBQZXJtaXNzaW9uIGRlbmllZAo=', 'exited': True}}
<c2r_oracle8_template-clone> GA stdout: 
<c2r_oracle8_template-clone> GA stderr: mkdir: cannot create directory ‘/root/.ansible’: Permission denied
fatal: [c2r_oracle8_template-clone]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo /root/.ansible/tmp `\"&& mkdir \"` echo /root/.ansible/tmp/ansible-tmp-1617899822.4112039-61006-63082701834685 `\" && echo ansible-tmp-1617899822.4112039-61006-63082701834685=\"` echo /root/.ansible/tmp/ansible-tmp-1617899822.4112039-61006-63082701834685 `\" ), exited with result 1, stderr output: mkdir: cannot create directory ‘/root/.ansible’: Permission denied\n",
    "unreachable": true
}

PLAY RECAP *********************************************************************
c2r_oracle8_template-clone : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0   

I see from the systemctl status qemu-guest-agent, that the command was called:

[root@vase inhibit-if-kmods-is-not-supported]# systemctl status qemu-guest-agent
● qemu-guest-agent.service - QEMU Guest Agent
   Loaded: loaded (/usr/lib/systemd/system/qemu-guest-agent.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2021-04-08 16:29:07 UTC; 10min ago
 Main PID: 666 (qemu-ga)
    Tasks: 2 (limit: 10636)
   Memory: 2.9M
   CGroup: /system.slice/qemu-guest-agent.service
           └─666 /usr/bin/qemu-ga --method=virtio-serial --path=/dev/virtio-ports/org.qemu.guest_agent.0 --blacklist= ->

Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec called: "/bin/sh -c echo ~ && sleep 0"
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1007
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1007
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1007
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec called: "/bin/sh -c ( umask 77 && mkdir -p "`>
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1008
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1008
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1008
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1008
Apr 08 16:37:02 vase.domaci.sit.hvfree.net qemu-ga[666]: info: guest-exec-status called, pid: 1008

I also ensured, that in qemu-ga config I have commented the following line:

# BLACKLIST_RPC=guest-file-open,guest-file-close,guest-file-read,guest-file-write,guest-file-seek,guest-file-flush,guest-exec,guest-exec-status

Also the machine has in its configuration:

    <channel type="unix">
      <source mode="bind" path="/var/lib/libvirt/qemu/f16x86_64.agent"/>
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>

ISSUE TYPE

• Bug Report

COMPONENT NAME

libvirt-qemu

ANSIBLE VERSION

➞  ansible --version                                                                                                   ansible 2.10.7  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/azhukov/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/azhukov/.pyenv/versions/3.9.1/lib/python3.9/site-packages/ansible
  executable location = /home/azhukov/.pyenv/versions/3.9.1/bin/ansible
  python version = 3.9.1 (default, Jan 14 2021, 15:48:15) [GCC 10.2.0]

CONFIGURATION

null

OS / ENVIRONMENT

kvm VM, using ansible libvirt plugin to connect

STEPS TO REPRODUCE

- hosts: all
  vars:
    # to be overridden by the tmt
    repo_root:
  tasks:

    - name: Building rpms
      delegate_to: localhost
      shell: make rpms
      args:
        chdir: ../..

    - name: Copying rpms to kvm remote
      copy:
        dest: "{{ repo_root }}/.rpms"
        src: ../../.rpms/
      when: ansible_facts['virtualization_type'] == "kvm"

    - name: Install el8 convert2rhel package
      yum:
        name: "{{ repo_root }}/.rpms/convert2rhel-0.20-1.el8.noarch.rpm"
        state: present
        disable_gpg_check: true

EXPECTED RESULTS

In another machine, it works, but I can't find the difference with the faulty one

ACTUAL RESULTS

Already provided

[zhukovgreen]

What else needs to be checked? Thanks for any hint

[zhukovgreen]

I found the reason for the failure - SELinux was in enforcing mode. I believe this could be easily checked and clear exception might be provided