Plugin dont work after update netbox to 2.9.1 #129

SerhiiZahuba · 2020-08-23T13:58:52Z

Environment

Python version: 3.6.8
NetBox version: 2.9.1
vCenter version: 6.7.0 -->

vCenter NetBox Sync version:

Steps to Reproduce

After run plugin i see

[root@netbox]# /bin/python3.6 /opt/vcenter-netbox-sync/run.py >/var/log/vcentersync.log
2020-08-23 15:56:17,690 [INFO] Created new HTTP Session for NetBox.
2020-08-23 15:56:17,785 [INFO] Detected NetBox API v2.9.
2020-08-23 15:56:17,786 [INFO] Verifying all prerequisite objects exist in NetBox.
2020-08-23 15:56:17,838 [INFO] NetBox manufacturers object 'VMware' match current values. Moving on.
2020-08-23 15:56:17,866 [INFO] NetBox platforms object 'VMware ESXi' match current values. Moving on.
2020-08-23 15:56:17,895 [INFO] NetBox platforms object 'Windows' match current values. Moving on.
2020-08-23 15:56:17,919 [INFO] NetBox platforms object 'Linux' match current values. Moving on.
2020-08-23 15:56:18,032 [INFO] NetBox sites object 'vCenter' do not match current values.
Traceback (most recent call last):
File "/opt/vcenter-netbox-sync/run.py", line 1601, in
main()
File "/opt/vcenter-netbox-sync/run.py", line 244, in main
nb.verify_dependencies()
File "/opt/vcenter-netbox-sync/run.py", line 1540, in verify_dependencies
self.obj_exists(nb_obj_type=dep_type, vc_data=dep)
File "/opt/vcenter-netbox-sync/run.py", line 1105, in obj_exists
set(vc_data["tags"] + nb_data["tags"])
TypeError: unhashable type: 'dict'

Expected Behavior

Observed Behavior

Djf2884 · 2020-08-24T06:45:16Z

Same here, even after a cleanup, collect seem to work and stop with unexpected behavior.

also it seem not working properly with vcenter/esxi 7.0 which have been updated on API level.

gstorme · 2020-08-24T09:38:42Z

Probably related to netbox-community/netbox#3703
Can you paste a debug run log + results of GET /api/dcim/sites/?name=vCenter?

SerhiiZahuba · 2020-08-24T11:42:16Z

2020-08-24 10:00:02,237 [INFO] Verifying all prerequisite objects exist in NetBox.
2020-08-24 10:00:02,268 [INFO] NetBox manufacturers object 'VMware' match current values. Moving on.
2020-08-24 10:00:02,304 [INFO] NetBox platforms object 'VMware ESXi' match current values. Moving on.
2020-08-24 10:00:02,329 [INFO] NetBox platforms object 'Windows' match current values. Moving on.
2020-08-24 10:00:02,358 [INFO] NetBox platforms object 'Linux' match current values. Moving on.
2020-08-24 10:00:02,426 [INFO] NetBox sites object 'vCenter' do not match current values.
2020-08-24 12:00:01,821 [INFO] Created new HTTP Session for NetBox.
2020-08-24 12:00:01,866 [INFO] Detected NetBox API v2.9.
2020-08-24 12:00:01,868 [INFO] Verifying all prerequisite objects exist in NetBox.
2020-08-24 12:00:01,898 [INFO] NetBox manufacturers object 'VMware' match current values. Moving on.
2020-08-24 12:00:01,932 [INFO] NetBox platforms object 'VMware ESXi' match current values. Moving on.
2020-08-24 12:00:01,969 [INFO] NetBox platforms object 'Windows' match current values. Moving on.
2020-08-24 12:00:01,999 [INFO] NetBox platforms object 'Linux' match current values. Moving on.
2020-08-24 12:00:02,069 [INFO] NetBox sites object 'vCenter' do not match current values.

`HTTP 200 OK
Allow: GET, POST, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
"count": 1,
"next": null,
"previous": null,
"results": [
{
"id": 12,
"url": "http://netbox..com/api/dcim/sites/12/",
"name": "vCenter",
"slug": "vcenter",
"status": {
"value": "active",
"label": "Active"
},
"region": null,
"tenant": null,
"facility": "",
"asn": null,
"time_zone": null,
"description": "",
"physical_address": "",
"shipping_address": "",
"latitude": null,
"longitude": null,
"contact_name": "",
"contact_phone": "",
"contact_email": "",
"comments": "A default virtual site created to house objects that have been synced from vCenter.",
"tags": [
{
"id": 10,
"url": "http://netbox..com/api/extras/tags/10/",
"name": "Synced",
"slug": "synced",
"color": "9e9e9e"
},
{
"id": 9,
"url": "http://netbox.**.com/api/extras/tags/9/",
"name": "vCenter",
"slug": "vcenter",
"color": "9e9e9e"
}
],
"custom_fields": {},
"created": "2020-05-13",
"last_updated": "2020-05-13T20:34:40.272068+03:00",
"circuit_count": null,
"device_count": null,
"prefix_count": null,
"rack_count": null,
"virtualmachine_count": null,
"vlan_count": null
}
]
}`

gstorme · 2020-08-24T12:41:49Z

Yes so this script is incompatible with Netbox >= 2.9.0 due to changes in the way tags are created.
Best to stay on 2.8 until a PR is submitted by someone with the necessary changes.

SerhiiZahuba · 2020-08-24T12:59:33Z

@synackray can you help ?

gstorme · 2020-09-12T16:49:29Z

Not just the tags, but also the API changes in 2.9.0 for ipam.IPAddress cause the mapping between IP & virtual-machine to not work anymore.

ipam.IPAddress: Removed interface field; replaced with assigned_object generic foreign key. This may represent either a device interface or a virtual machine interface. Assign an object by setting assigned_object_type and assigned_object_id.

bb-Ricardo · 2020-09-17T15:13:38Z

this will fix the tag issue: https://github.com/bb-Ricardo/vcenter-netbox-sync/commit/2d3052fd14bc3947b24e1da3927e93e4e44b6355

Will have a look at ip ipam/ipaddress next week.

bb-Ricardo · 2020-09-23T12:27:26Z

Well, we got some progress: https://github.com/bb-Ricardo/vcenter-netbox-sync/commit/9447109a12e18703954cd843c7bc2a6adb936285

Now all my items sync again as before.

Please check it out. All changes are also in #132

SerhiiZahuba · 2020-09-25T15:30:15Z

I have a problem after updating the plug-in, all virtual servers were transferred to Standalone ESXi Host, they were previously synchronized with their server, for example, if the virtual machine is on the 192.168.50.170 server, then in the Cluster it will also be synchronized there. Now all VMs are in Standalone ESXi Host.

SerhiiZahuba · 2020-09-25T16:23:00Z

also, as you can see in the screenshot, some virtual servers are in their clusters, but the main part has moved to Standalone

bb-Ricardo · 2020-09-25T20:43:09Z

That's very interesting as I didn't touch this part of code: https://github.com/bb-Ricardo/vcenter-netbox-sync/blame/9447109a12e18703954cd843c7bc2a6adb936285/run.py#L557

"Standalone ESXi Host" is used if the hosts cluster name is the same as the hosts name:

                    cluster = obj.parent.name
                    # We throw the cluster away if it matches the ESXi host
                    # name as it is standalone.
                    if cluster == obj_name:
                        # Store the host so that we can check VMs against it
                        self.standalone_hosts.append(cluster)
                        cluster = "Standalone ESXi Host"

and Line: https://github.com/bb-Ricardo/vcenter-netbox-sync/blob/develop/run.py#L647

                elif vc_obj_type == "virtual_machines":
                    # Virtual Machines
                    log.debug(
                        "Collecting info for virtual machine '%s'", obj_name
                        )
                    # Cluster
                    cluster = obj.runtime.host.parent.name
                    if cluster in self.standalone_hosts:
                        log.debug(
                            "VM is assigned to a standalone ESXi host. Setting "
                            "cluster to 'Standalone ESXi Host'."
                            )
                        cluster = "Standalone ESXi Host"

In my opinion the whole sync needs a complete rewrite. Unfortunately this code is only used in a test environment. It was a POC and after the last NetBox API FU I'm not sure if I'm willing to spend my time on changing code and adapting APIs every time a new NetBox major release is issued.

I have loads of ideas but (https://github.com/bb-Ricardo/vcenter-netbox-sync/issues) but not sure if time allows it.

MarcSN311 · 2020-10-06T10:02:13Z

@bb-Ricardo I keep getting errors like this: [DEBUG] NetBox 400 status reason: {"tags":[["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: Synced"],["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: vCenter"]]}
I am on your develop branch

bb-Ricardo · 2020-10-06T11:40:40Z

You're on NetBox version 2.9.X?

MarcSN311 · 2020-10-06T11:45:59Z

I am on 2.9.4

fjasboka · 2020-10-06T11:46:53Z

@MarcSN311 Not sure if it the exact same problem I ran into, but I had to delete the vCenter and Synced tag before running it.
I am running 2.9.4

MarcSN311 · 2020-10-06T11:48:19Z

I am on a fresh install, I did not have any tags before running the sync.

EDIT: Nevermind, I found the issue on my end.

SerhiiZahuba · 2020-10-06T13:26:38Z

@fjasboka

do you have a problem with clusters? like mine #129 (comment)

or don't you use them?

fjasboka · 2020-10-07T06:45:53Z

@SerhiiZahuba I did not run into that issue. It placed all the VMs in the correct cluster and also the ESXi servers.

SerhiiZahuba · 2020-10-07T12:59:06Z

Strange, I don’t understand what I’m doing wrong. All my virtual machines are migrated to the Standalone ESXi Host section.

ludvik78 · 2020-10-25T08:44:16Z

@bb-Ricardo I keep getting errors like this: [DEBUG] NetBox 400 status reason: {"tags":[["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: Synced"],["Related objects must be referenced by numeric ID or by dictionary of attributes. Received an unrecognized value: vCenter"]]}
I am on your develop branch

Any progress on this issue?

bb-Ricardo · 2020-10-26T15:42:22Z

currently trying to rewrite this whole process.

Hope I get all of this covered in here: https://github.com/bb-Ricardo/netbox-sync

bb-Ricardo · 2020-11-06T19:37:10Z

I went down a rabbit hole and finally can see the light at the end of the tunnel again. This took way longer then expected (as usual).

Here you go: https://github.com/bb-Ricardo/netbox-sync

please test as much as you can: Debug logs are very welcome.

nathan-march · 2020-11-18T22:40:07Z

Just gave it a first try and looks very promising! Only thing I've noticed so far is my vlans in vmware are named "123-name-10.0.0.0/24" and they get imported into netbox with the url encoded %2f instead of a / (when looking at the name column on the VM interface)

bb-Ricardo · 2020-11-18T22:53:36Z

Great to hear.

Can you please open an issue for that. Would like to have a look at it.

Thank you.

SerhiiZahuba added Status: Pending Type: Bug labels Aug 23, 2020

bb-Ricardo mentioned this issue Sep 23, 2020

Add Support for NetBox API version 2.9 #132

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Plugin dont work after update netbox to 2.9.1 #129

Plugin dont work after update netbox to 2.9.1 #129

SerhiiZahuba commented Aug 23, 2020

Djf2884 commented Aug 24, 2020

gstorme commented Aug 24, 2020

SerhiiZahuba commented Aug 24, 2020

gstorme commented Aug 24, 2020

SerhiiZahuba commented Aug 24, 2020

gstorme commented Sep 12, 2020

bb-Ricardo commented Sep 17, 2020

bb-Ricardo commented Sep 23, 2020

SerhiiZahuba commented Sep 25, 2020 •

edited

Loading

SerhiiZahuba commented Sep 25, 2020

bb-Ricardo commented Sep 25, 2020

MarcSN311 commented Oct 6, 2020

bb-Ricardo commented Oct 6, 2020

MarcSN311 commented Oct 6, 2020

fjasboka commented Oct 6, 2020

MarcSN311 commented Oct 6, 2020 •

edited

Loading

SerhiiZahuba commented Oct 6, 2020

fjasboka commented Oct 7, 2020

SerhiiZahuba commented Oct 7, 2020

ludvik78 commented Oct 25, 2020

bb-Ricardo commented Oct 26, 2020

bb-Ricardo commented Nov 6, 2020

nathan-march commented Nov 18, 2020 •

edited

Loading

bb-Ricardo commented Nov 18, 2020

Plugin dont work after update netbox to 2.9.1 #129

Plugin dont work after update netbox to 2.9.1 #129

Comments

SerhiiZahuba commented Aug 23, 2020

Environment

Steps to Reproduce

Expected Behavior

Observed Behavior

Djf2884 commented Aug 24, 2020

gstorme commented Aug 24, 2020

SerhiiZahuba commented Aug 24, 2020

gstorme commented Aug 24, 2020

SerhiiZahuba commented Aug 24, 2020

gstorme commented Sep 12, 2020

bb-Ricardo commented Sep 17, 2020

bb-Ricardo commented Sep 23, 2020

SerhiiZahuba commented Sep 25, 2020 • edited Loading

SerhiiZahuba commented Sep 25, 2020

bb-Ricardo commented Sep 25, 2020

MarcSN311 commented Oct 6, 2020

bb-Ricardo commented Oct 6, 2020

MarcSN311 commented Oct 6, 2020

fjasboka commented Oct 6, 2020

MarcSN311 commented Oct 6, 2020 • edited Loading

SerhiiZahuba commented Oct 6, 2020

fjasboka commented Oct 7, 2020

SerhiiZahuba commented Oct 7, 2020

ludvik78 commented Oct 25, 2020

bb-Ricardo commented Oct 26, 2020

bb-Ricardo commented Nov 6, 2020

nathan-march commented Nov 18, 2020 • edited Loading

bb-Ricardo commented Nov 18, 2020

SerhiiZahuba commented Sep 25, 2020 •

edited

Loading

MarcSN311 commented Oct 6, 2020 •

edited

Loading

nathan-march commented Nov 18, 2020 •

edited

Loading