Handle CML API errors on cml_lab_facts.py

[jillesca]

Before anything, thanks for this repo, is excellent to build topologies in CML using ansible.

Issue

While creating a lab and using the cisco.cml.build along the -e startup='host' -e wait='yes' options, the playbook is failing as an CML API is hitting a 500 error. The playbook continues to check if the lab is fully Booted (which is it is) but due to this error, the localhost is marked as failed.

Below is a output of the error:

fatal: [localhost]: FAILED! => {
    "attempts": 40,
    "changed": false,
    "module_stderr": "SSL Verification disabled\nAPI Error: {\n  \"description\": \"The API at /api/v0/labs/cb1dfc/layer3_addresses encountered an unexpected error.  Please report this problem to support.\",\n  \"code\": 500\n}\nTraceback (most recent call last):\n  File \"/root/.ansible/tmp/ansible-tmp-1670068947.166718-3941-209521240526074/AnsiballZ_cml_lab_facts.py\", line 107, in <module>\n    _ansiballz_main()\n  File \"/root/.ansible/tmp/ansible-tmp-1670068947.166718-3941-209521240526074/AnsiballZ_cml_lab_facts.py\", line 99, in _ansiballz_main\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\n  File \"/root/.ansible/tmp/ansible-tmp-1670068947.166718-3941-209521240526074/AnsiballZ_cml_lab_facts.py\", line 47, in invoke_module\n    runpy.run_module(mod_name='ansible_collections.cisco.cml.plugins.modules.cml_lab_facts', init_globals=dict(_module_fqn='ansible_collections.cisco.cml.plugins.modules.cml_lab_facts', _modlib_path=modlib_path),\n  File \"/usr/local/lib/python3.10/runpy.py\", line 224, in run_module\n    return _run_module_code(code, init_globals, run_name, mod_spec)\n  File \"/usr/local/lib/python3.10/runpy.py\", line 96, in _run_module_code\n    _run_code(code, mod_globals, init_globals,\n  File \"/usr/local/lib/python3.10/runpy.py\", line 86, in _run_code\n    exec(code, run_globals)\n  File \"/tmp/ansible_cisco.cml.cml_lab_facts_payload_kj9vlw6l/ansible_cisco.cml.cml_lab_facts_payload.zip/ansible_collections/cisco/cml/plugins/modules/cml_lab_facts.py\", line 146, in <module>\n  File \"/tmp/ansible_cisco.cml.cml_lab_facts_payload_kj9vlw6l/ansible_cisco.cml.cml_lab_facts_payload.zip/ansible_collections/cisco/cml/plugins/modules/cml_lab_facts.py\", line 142, in main\n  File \"/tmp/ansible_cisco.cml.cml_lab_facts_payload_kj9vlw6l/ansible_cisco.cml.cml_lab_facts_payload.zip/ansible_collections/cisco/cml/plugins/modules/cml_lab_facts.py\", line 110, in run_module\n  File \"/usr/local/lib/python3.10/site-packages/virl2_client/models/interface.py\", line 137, in discovered_ipv4\n    self.node.lab.sync_l3_addresses_if_outdated()\n  File \"/usr/local/lib/python3.10/site-packages/virl2_client/models/lab.py\", line 152, in sync_l3_addresses_if_outdated\n    self.sync_layer3_addresses()\n  File \"/usr/local/lib/python3.10/site-packages/virl2_client/models/lab.py\", line 1202, in sync_layer3_addresses\n    response.raise_for_status()\n  File \"/usr/local/lib/python3.10/site-packages/requests/models.py\", line 1021, in raise_for_status\n    raise HTTPError(http_error_msg, response=self)\nrequests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://10.10.20.161/api/v0/labs/cb1dfc/layer3_addresses\n",
    "module_stdout": "",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

PLAY RECAP ************************************************************************************************
bridge-to-sandbox          : ok=1    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
host1-20.31                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host2-20.32                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
host3-20.33                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
leaf1-20.21                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
leaf2-20.22                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
leaf3-20.23                : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
localhost                  : ok=4    changed=1    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
sandbox-backend            : ok=1    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0   
spine1-20.11               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   
spine2-20.12               : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

To summarize the traceback the relevant parts are:

File \"/tmp/ansible_cisco.cml.cml_lab_facts_payload_kj9vlw6l/ansible_cisco.cml.cml_lab_facts_payload.zip/ansible_collections/cisco/cml/plugins/modules/cml_lab_facts.py\", 
- line 110, in run_module\n  
File \"/usr/local/lib/python3.10/site-packages/virl2_client/models/interface.py\", 
- line 137, in discovered_ipv4\n    self.node.lab.sync_l3_addresses_if_outdated()\n  
File \"/usr/local/lib/python3.10/site-packages/virl2_client/models/lab.py\", 
- line 152, in sync_l3_addresses_if_outdated\n    self.sync_layer3_addresses()\n  
File \"/usr/local/lib/python3.10/site-packages/virl2_client/models/lab.py\", 
- line 1202, in sync_layer3_addresses\n    response.raise_for_status()\n  
File \"/usr/local/lib/python3.10/site-packages/requests/models.py\", 
- line 1021, in raise_for_status\n    
- raise HTTPError(http_error_msg, response=self)\n
- requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: https://10.10.20.161/api/v0/labs/cb1dfc/layer3_addresses\n"

The line which is causing the issue in cml_lab_facts.py plugin is

'ipv4_addresses': interface.discovered_ipv4,

I suspect it could be possible to add a logic that in case an error is returned from virl2_client, give an empty value so the playbook continue.

is worth to mention that I confirmed this issue is coming from CML API, I tested with the swagger client from CML and the issue was present.

inventory

I'm my case, in my inventory I'm defining the ansible_host ip, so the interface.discovered_ipv4 is not adding value for me. for reference this is my inventory:

---
all:
  vars:
    ansible_user: "{{ lookup('env', 'ANSIBLE_USER') | default('cisco', true) }}"
    ansible_password: "{{ lookup('env', 'ANSIBLE_PASSWORD') | default('cisco', true) }}"
    ansible_network_os: ios
    ansible_ssh_pass: cisco
  children:
    network:
      children:
        iosxe_routers:
          hosts:
            spine1-20.11:
              ansible_host: 10.10.20.11
            spine2-20.12:
              ansible_host: 10.10.20.12
            leaf1-20.21:
              ansible_host: 10.10.20.21
            leaf2-20.22:
              ansible_host: 10.10.20.22
            leaf3-20.23:
              ansible_host: 10.10.20.23
    clients:
      children:
        linux_hosts:
          hosts:
            host1-20.31:
              ansible_host: 10.10.20.31
            host2-20.32:
              ansible_host: 10.10.20.32
            host3-20.33:
              ansible_host: 10.10.20.33

Is worth to mention that cisco.cml.inventory playbook, fails due to this same error.

I'm also bringing this 500 error to the CML API folks to see why this is happening, but I consider is good to add logic here to handle issues outside of the module.

setup

This is my setup:

~ # ansible --version
ansible [core 2.13.5]
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.10/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/local/bin/ansible
  python version = 3.10.8 (main, Oct 13 2022, 23:21:19) [GCC 11.2.1 20220219]
  jinja version = 3.1.2
  libyaml = False
~ # ansible-galaxy collection list cisco.cml

# /root/.ansible/collections/ansible_collections
Collection Version
---------- -------
cisco.cml  1.1.0  
~ # 

And I'm using virl2-client==2.2.1.post2 since this is the version that works currently with the devnet sandbox.

Thanks again for this great repo.

[jasonking3]

@jillesca - we had seen this API error in CML v2.2 with L3 addresses as well. I don't think we are seeing it in v2.4, however. I think your logic is sound though. It is probably better to just not report an IP address, rather than error out like that. I will look into it.