Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: nb_inventory is taking several minutes to run due to the "limit&offset" behaviour. Can we turn that off? #1370

Open
alice-rc opened this issue Jan 14, 2025 · 6 comments
Open

[Bug]: nb_inventory is taking several minutes to run due to the "limit&offset" behaviour. Can we turn that off? #1370

alice-rc opened this issue Jan 14, 2025 · 6 comments
Labels
bug Something isn't working

Comments

@alice-rc
Copy link

Ansible NetBox Collection version

v3.20.0

Ansible version

ansible [core 2.17.7]
  config file = <REDACTED>/ansible.cfg
  configured module search path = ['<REDACTED>/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = <REDACTED>/lib64/python3.11/site-packages/ansible
  ansible collection location = /<REDACTED>/collections
  executable location = <REDACTED>/bin/ansible
  python version = 3.11.9 (main, Sep 11 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2.0.1)] (<REDACTED>/bin/python3.11)
  jinja version = 3.1.4
  libyaml = True

NetBox version

NetBox Community v4.1.11

Python version

3.11

Steps to Reproduce

$ date; ansible-inventory -i inventory/nb_test_vms.yml --list -vvv 2>&1 >testout; date

Content of inventory config file:

---
plugin: netbox.netbox.nb_inventory
api_endpoint: "https://netbox.\<REDACTED>"
validate_certs: false
flatten_custom_fields: false
plurals: false
group_by:
  - tenant
group_names_raw: true
query_filters:
  - tenant__n: \<REDACTED\>
device_query_filters:
  - id: -1
prefixes: false
services: false
fetch_all: true
racks: false
rename_variables:
  - pattern: custom_fields
    repl: nb_inventory_custom_fields
  - pattern: is_virtual
    repl: nb_inventory_is_virtual

Expected Behavior

The plugin should not be taking several minutes to run. Need an option to turn off the "limit" behavior.

Using just a single limit, takes 14 seconds

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?limit=0&tenant__n=<REDACTED>&exclude=config_context" -H "Authorization: Token $NETBOX_TOKEN" > /dev/null; date
Tue 14 Jan 16:33:58 UTC 2025
Tue 14 Jan 16:34:12 UTC 2025

Using no limit, takes 2 seconds

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?tenant__n=<REDACTED>&exclude=config_context" -H "Authorization: Token $NETBOX_TOKEN" > /dev/null; date
Tue 14 Jan 16:35:54 UTC 2025
Tue 14 Jan 16:35:56 UTC 2025

Observed Behavior

Output from the above inventory command:
Inventory plugin takes almost 7 minutes

Tue 14 Jan 01:14:36 UTC 2025
Tue 14 Jan 01:21:27 UTC 2025

Beginning of the output file:

...
Fetching: https://netbox.<REDACTED>/api/status
Fetching: https://netbox.<REDACTED>/api/dcim/devices/?limit=0&tenant__n=<REDACTED>&id=-1&exclude=config_context
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?limit=0&tenant__n=<REDACTED>&exclude=config_context
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=1000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=2000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=3000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=4000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=5000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=6000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=7000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=8000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=9000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=10000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=11000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=12000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=13000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=14000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=15000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=16000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=17000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=18000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=19000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=20000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=21000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=22000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=23000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=24000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=25000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=26000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=27000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=28000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=29000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=30000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/virtualization/virtual-machines/?exclude=config_context&limit=1000&offset=31000&tenant__n=<REDACTED>
Fetching: https://netbox.<REDACTED>/api/dcim/sites/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/regions/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/site-groups/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/locations/?limit=0
Fetching: https://netbox.<REDACTED>/api/tenancy/tenants/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/device-roles/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/platforms/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/device-types/?limit=0
Fetching: https://netbox.<REDACTED>/api/dcim/manufacturers/?limit=0
Fetching: https://netbox.<REDACTED>/api/virtualization/clusters/?limit=0
Parsed <REDACTED>/inventory/nb_test_vms.yml inventory source with auto plugin
...
@alice-rc alice-rc added the bug Something isn't working label Jan 14, 2025
@sc68cal
Copy link
Contributor

sc68cal commented Jan 15, 2025

Can you also do a test where you do the API call without the limit, then with the limit to just make sure it's not the case where the first API call warmed up a cache?

@alice-rc
Copy link
Author

Curl doesn't cache anything, but incase there could be caching on netbox itself....

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?tenant__n=<REDACTED>&exclude=config_context&foo=$RANDOM" -H "Authorization: Token $NETBOX_TOKEN" -H "Cache-Control: no-cache" > test_nolimit; date
Wed 15 Jan 18:24:31 UTC 2025
Wed 15 Jan 18:24:33 UTC 2025

$ date; curl -s "https://netbox.<REDACTED>/api/virtualization/virtual-machines/?limit=0&tenant__n=<REDACTED>&exclude=config_context&foo=$RANDOM" -H "Authorization: Token $NETBOX_TOKEN" -H "Cache-Control: no-cache" > test_limit; date
Wed 15 Jan 18:24:54 UTC 2025
Wed 15 Jan 18:25:07 UTC 2025

$ cat test_nolimit | jq . | grep \"count\"
  "count": 30978,
$ cat test_limit | jq . | grep \"count\"
  "count": 30978,

@alice-rc
Copy link
Author

But I'm not so concerned with the performance of the raw curl, but more with the performance of the curl commands vs the nb_inventory plugin. 2 seconds -> 13 seconds is far less concerning than 2/13 seconds to almost 7 minutes

@sc68cal
Copy link
Contributor

sc68cal commented Jan 15, 2025

I understand. However looking at your logs it appears that you have over 31k virtual machines.

Thank you for checking that we are getting the same amount of machines with the limit vs non limit query because that was going to be my next suspicion as well.

We need to investigate why the inventory plugin is pulling in chunks, perhaps at one point it was needed in order to not hammer NetBox but it appears that forcing NetBox to do pagination is making performance worse now.

@alice-rc
Copy link
Author

alice-rc commented Jan 16, 2025 •
edited
Loading

So doing more investigation on this it seems that there are some NetBox configuration settings that come into play here. The big one is that, by default, MAX_PAGE_SIZE is set to 1000 which is what seems to be the underlying reason for the limit&offset. That being the case I'm not sure how we can get around the limit&offset restriction other than to use the GraphQL API instead of the regular API as it does not have that page limit restriction. That introduces a whole other complexity as there seems to be current limitations on filtering when using the the GraphQL API (netbox-community/netbox#17688).
Our biggest issue is that the nb_inventory plugin is unusable given that it takes 7 minutes to run.

We are currently investigating how we could write a custom inventory plugin using the GraphQL API as we don't have a broad range of filter or field retrieval requirements.

@sc68cal
Copy link
Contributor

sc68cal commented Jan 19, 2025

Have you enabled caching?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@sc68cal @alice-rc