Skip to content

Commit

Permalink
Merge pull request #100 from networktocode/develop-2.0
Browse files Browse the repository at this point in the history
Release 2.0
  • Loading branch information
dgarros authored Oct 19, 2020
2 parents 9cdb6d2 + 3cd7b02 commit 986fe8b
Show file tree
Hide file tree
Showing 39 changed files with 2,320 additions and 1,070 deletions.
28 changes: 21 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,10 @@ pip install ntc-netbox-plugin-onboarding
systemctl restart netbox netbox-rq
```

> The plugin is compatible with NetBox 2.8.1 and higher
> The ntc-netbox-plugin-onboarding v1.3 is compatible with NetBox 2.8
> The ntc-netbox-plugin-onboarding v2 is compatible with NetBox 2.8 and NetBox 2.9
To ensure NetBox Onboarding plugin is automatically re-installed during future upgrades, create a file named `local_requirements.txt` (if not already existing) in the NetBox root directory (alongside `requirements.txt`) and list the `ntc-netbox-plugin-onboarding` package:

```no-highlight
Expand Down Expand Up @@ -64,26 +66,38 @@ The plugin behavior can be controlled with the following list of settings
- `default_device_role_color` string (default FF0000), color assigned to the device role if it needs to be created.
- `default_management_interface` string (default "PLACEHOLDER"), name of the management interface that will be created, if one can't be identified on the device.
- `default_management_prefix_length` integer ( default 0), length of the prefix that will be used for the management IP address, if the IP can't be found.
- `skip_device_type_on_update` boolean (default False), If True, an existing NetBox device will not get its device type updated. If False, device type will be updated with one discovered on a device.
- `skip_manufacturer_on_update` boolean (default False), If True, an existing NetBox device will not get its manufacturer updated. If False, manufacturer will be updated with one discovered on a device.
- `platform_map` (dictionary), mapping of an **auto-detected** Netmiko platform to the **NetBox slug** name of your Platform. The dictionary should be in the format:
```python
{
<Netmiko Platform>: <NetBox Slug>
<Netmiko Platform>: <NetBox Slug>
}
```
- `onboarding_extensions_map` (dictionary), mapping of a NAPALM driver name to the loadable Python module used as an onboarding extension. The dictionary should be in the format:
```python
{
<Napalm Driver Name>: <Loadable Python Module>
}
```
- `object_match_strategy` (string), defines the method for searching models. There are
currently two strategies, strict and loose. Strict has to be a direct match, normally
using a slug. Loose allows a range of search criteria to match a single object. If multiple
objects are returned an error is raised.

## Usage

### Preparation

To work properly the plugin needs to know the Site, Platform, Device Type, Device Role of each
device as well as its primary IP address or DNS Name. It's recommended to create these objects in
NetBox ahead of time and to provide them when you want to start the onboarding process.
To properly onboard a device, the plugin needs to only know the Site as well as device's primary IP address or DNS Name.

> For DNS Name Resolution to work, the instance of NetBox must be able to resolve the name of the
> device to IP address.

Providing other attributes (`Platform`, `Device Type`, `Device Role`) is optional - if any of these attributes is provided, plugin will use provided value for the onboarded device.
If `Platform`, `Device Type` and/or `Device Role` are not provided, the plugin will try to identify these information automatically and, based on the settings, it can create them in NetBox as needed.
> If the Platform is provided, it must contains a valid Napalm driver available to the worker in Python
> If the Platform is provided, it must point to an existing NetBox Platform. NAPALM driver of this platform will be used only if it is defined for the platform in NetBox.
> To use a preferred NAPALM driver, either define it in NetBox per platform or in the plugins settings under `platform_map`

### Onboard a new device

Expand Down
15 changes: 14 additions & 1 deletion development/base_configuration.py
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,20 @@

# Enable custom logging. Please see the Django documentation for detailed guidance on configuring custom logs:
# https://docs.djangoproject.com/en/1.11/topics/logging/
LOGGING = {}
LOGGING = {
"version": 1,
"disable_existing_loggers": False,
"formatters": {"rq_console": {"format": "%(asctime)s %(message)s", "datefmt": "%H:%M:%S",},},
"handlers": {
"rq_console": {
"level": "DEBUG",
"class": "rq.utils.ColorizingStreamHandler",
"formatter": "rq_console",
"exclude": ["%(asctime)s"],
},
},
"loggers": {"rq.worker": {"handlers": ["rq_console"], "level": "DEBUG"},},
}

# Setting this to True will permit only authenticated users to access any part of NetBox. By default, anonymous users
# are permitted to access most data in NetBox (excluding secrets) but not make any changes.
Expand Down
96 changes: 96 additions & 0 deletions docs/examples/example_ios_set_device_role.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,96 @@
"""Example of custom onboarding class.
(c) 2020 Network To Code
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""

from netbox_onboarding.netbox_keeper import NetboxKeeper
from netbox_onboarding.onboarding.onboarding import Onboarding


class MyOnboardingClass(Onboarding):
"""Custom onboarding class example.
Main purpose of this class is to access and modify the onboarding_kwargs.
By accessing the onboarding kwargs, user gains ability to modify
onboarding parameters before the objects are created in NetBox.
This class adds the get_device_role method that does the static
string comparison and returns the device role.
"""

def run(self, onboarding_kwargs):
"""Ensures network device."""
# Access hostname from onboarding_kwargs and get device role automatically
device_new_role = self.get_device_role(hostname=onboarding_kwargs["netdev_hostname"])

# Update the device role in onboarding kwargs dictionary
onboarding_kwargs["netdev_nb_role_slug"] = device_new_role

nb_k = NetboxKeeper(**onboarding_kwargs)
nb_k.ensure_device()

self.created_device = nb_k.device

@staticmethod
def get_device_role(hostname):
"""Returns the device role based on hostname data.
This is a static analysis of hostname string content only
"""
hostname_lower = hostname.lower()
if ("rtr" in hostname_lower) or ("router" in hostname_lower):
role = "router"
elif ("sw" in hostname_lower) or ("switch" in hostname_lower):
role = "switch"
elif ("fw" in hostname_lower) or ("firewall" in hostname_lower):
role = "firewall"
elif "dc" in hostname_lower:
role = "datacenter"
else:
role = "generic"

return role


class OnboardingDriverExtensions:
"""This is an example of a custom onboarding driver extension.
This extension sets the onboarding_class to MyOnboardingClass,
which is an example class of how to access and modify the device
role automatically through the onboarding process.
"""

def __init__(self, napalm_device):
"""Inits the class."""
self.napalm_device = napalm_device
self.onboarding_class = MyOnboardingClass
self.ext_result = None

def get_onboarding_class(self):
"""Return onboarding class for IOS driver.
Currently supported is Standalone Onboarding Process
Result of this method is used by the OnboardingManager to
initiate the instance of the onboarding class.
"""
return self.onboarding_class

def get_ext_result(self):
"""This method is used to store any object as a return value.
Result of this method is passed to the onboarding class as
driver_addon_result argument.
:return: Any()
"""
return self.ext_result
24 changes: 24 additions & 0 deletions docs/release-notes/version-2.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# ntc-netbox-plugin-onboarding v2.0 Release Notes

## v2.0

### Enhancements

* NetBox 2.9 support - Supported releases 2.8 and 2.9
* Onboarding extensions - Customizable onboarding process through Python modules.
* Onboarding details exposed in a device view - Date, Status, Last success and Latest task id related to the onboarded device are presented under the device view.
* Onboarding task view - Onboarding details exposed in a dedicated view, including NetBox's ChangeLog.
* Onboarding Changelog - Onboarding uses NetBox's ChangeLog to display user and changes made to the Onboarding Task object.
* Skip onboarding feature - New attribute in the OnboardingDevice model allows to skip the onboarding request on devices with disabled onboarding setting.

### Bug Fixes

* Fixed race condition in `worker.py`
* Improved logging

### Additional Changes

* Platform map now includes NAPALM drivers as defined in NetBox
* Tests have been refactored to inherit NetBox's tests
* Onboarding process will update the Device found by the IP-address lookup. In case of no existing device with onboarded IP-address is found in NetBox, onboarding might update the existing NetBox' looking up by network device's hostname.
* Onboarding will raise Exception when `create_device_type_if_missing` is set to `False` for existing Device with DeviceType mismatch (behaviour pre https://github.com/networktocode/ntc-netbox-plugin-onboarding/issues/74)
6 changes: 5 additions & 1 deletion netbox_onboarding/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
limitations under the License.
"""

__version__ = "1.3.0"
__version__ = "2.0.0"

from extras.plugins import PluginConfig

Expand All @@ -39,7 +39,11 @@ class OnboardingConfig(PluginConfig):
"default_management_prefix_length": 0,
"default_device_status": "active",
"create_management_interface_if_missing": True,
"skip_device_type_on_update": False,
"skip_manufacturer_on_update": False,
"platform_map": {},
"onboarding_extensions_map": {"ios": "netbox_onboarding.onboarding_extensions.ios",},
"object_match_strategy": "loose",
}
caching_config = {}

Expand Down
2 changes: 1 addition & 1 deletion netbox_onboarding/admin.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,5 +32,5 @@ class OnboardingTaskAdmin(admin.ModelAdmin):
"failed_reason",
"port",
"timeout",
"created_on",
"created",
)
2 changes: 2 additions & 0 deletions netbox_onboarding/choices.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,12 +22,14 @@ class OnboardingStatusChoices(ChoiceSet):
STATUS_PENDING = "pending"
STATUS_RUNNING = "running"
STATUS_SUCCEEDED = "succeeded"
STATUS_SKIPPED = "skipped"

CHOICES = (
(STATUS_FAILED, "failed"),
(STATUS_PENDING, "pending"),
(STATUS_RUNNING, "running"),
(STATUS_SUCCEEDED, "succeeded"),
(STATUS_SKIPPED, "skipped"),
)


Expand Down
3 changes: 2 additions & 1 deletion netbox_onboarding/constants.py
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
"""Constants for netbox_onboarding plugin."""
NETMIKO_TO_NAPALM = {

NETMIKO_TO_NAPALM_STATIC = {
"cisco_ios": "ios",
"cisco_nxos": "nxos_ssh",
"arista_eos": "eos",
Expand Down
39 changes: 39 additions & 0 deletions netbox_onboarding/exceptions.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
"""Exceptions.
(c) 2020 Network To Code
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""


class OnboardException(Exception):
"""A failure occurred during the onboarding process.
The exception includes a reason "slug" as defined below as well as a humanized message.
"""

REASONS = (
"fail-config", # config provided is not valid
"fail-connect", # device is unreachable at IP:PORT
"fail-execute", # unable to execute device/API command
"fail-login", # bad username/password
"fail-dns", # failed to get IP address from name resolution
"fail-general", # other error
)

def __init__(self, reason, message, **kwargs):
"""Exception Init."""
super(OnboardException, self).__init__(kwargs)
self.reason = reason
self.message = message

def __str__(self):
"""Exception __str__."""
return f"{self.__class__.__name__}: {self.reason}: {self.message}"
6 changes: 2 additions & 4 deletions netbox_onboarding/filters.py
Original file line number Diff line number Diff line change
Expand Up @@ -26,8 +26,6 @@ class OnboardingTaskFilter(NameSlugSearchFilterSet):

q = django_filters.CharFilter(method="search", label="Search",)

site_id = django_filters.ModelMultipleChoiceFilter(queryset=Site.objects.all(), label="Site (ID)",)

site = django_filters.ModelMultipleChoiceFilter(
field_name="site__slug", queryset=Site.objects.all(), to_field_name="slug", label="Site (slug)",
)
Expand All @@ -46,7 +44,7 @@ class Meta: # noqa: D106 "Missing docstring in public nested class"
model = OnboardingTask
fields = ["id", "site", "site_id", "platform", "role", "status", "failed_reason"]

def search(self, queryset, name, value):
def search(self, queryset, name, value): # pylint: disable=unused-argument, no-self-use
"""Perform the filtered search."""
if not value.strip():
return queryset
Expand All @@ -55,7 +53,7 @@ def search(self, queryset, name, value):
| Q(ip_address__icontains=value)
| Q(site__name__icontains=value)
| Q(platform__name__icontains=value)
| Q(device__icontains=value)
| Q(created_device__name__icontains=value)
| Q(status__icontains=value)
| Q(failed_reason__icontains=value)
| Q(message__icontains=value)
Expand Down
21 changes: 15 additions & 6 deletions netbox_onboarding/forms.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,11 +13,11 @@
"""

from django import forms
from django.db import transaction
from django_rq import get_queue

from utilities.forms import BootstrapMixin
from utilities.forms import BootstrapMixin, CSVModelForm
from dcim.models import Site, Platform, DeviceRole, DeviceType
from extras.forms import CustomFieldModelCSVForm

from .models import OnboardingTask
from .choices import OnboardingStatusChoices, OnboardingFailChoices
Expand All @@ -33,7 +33,7 @@ class OnboardingTaskForm(BootstrapMixin, forms.ModelForm):
required=True, label="IP address", help_text="IP Address/DNS Name of the device to onboard"
)

site = forms.ModelChoiceField(required=True, queryset=Site.objects.all(), to_field_name="slug")
site = forms.ModelChoiceField(required=True, queryset=Site.objects.all())

username = forms.CharField(required=False, help_text="Device username (will not be stored in database)")
password = forms.CharField(
Expand Down Expand Up @@ -106,7 +106,7 @@ class Meta: # noqa: D106 "Missing docstring in public nested class"
fields = ["q", "site", "platform", "status", "failed_reason"]


class OnboardingTaskFeedCSVForm(CustomFieldModelCSVForm):
class OnboardingTaskFeedCSVForm(CSVModelForm):
"""Form for entering CSV to bulk-import OnboardingTask entries."""

site = forms.ModelChoiceField(
Expand Down Expand Up @@ -149,12 +149,21 @@ class OnboardingTaskFeedCSVForm(CustomFieldModelCSVForm):

class Meta: # noqa: D106 "Missing docstring in public nested class"
model = OnboardingTask
fields = OnboardingTask.csv_headers
fields = [
"site",
"ip_address",
"port",
"timeout",
"platform",
"role",
]

def save(self, commit=True, **kwargs):
"""Save the model, and add it and the associated credentials to the onboarding worker queue."""
model = super().save(commit=commit, **kwargs)
if commit:
credentials = Credentials(self.data.get("username"), self.data.get("password"), self.data.get("secret"))
get_queue("default").enqueue("netbox_onboarding.worker.onboard_device", model.pk, credentials)
transaction.on_commit(
lambda: get_queue("default").enqueue("netbox_onboarding.worker.onboard_device", model.pk, credentials)
)
return model
18 changes: 18 additions & 0 deletions netbox_onboarding/metrics.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
"""Plugin additions to the NetBox navigation menu.
(c) 2020 Network To Code
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
"""
from prometheus_client import Counter

onboardingtask_results_counter = Counter(
name="onboardingtask_results_total", documentation="Count of results for Onboarding Task", labelnames=("status",)
)
Loading

0 comments on commit 986fe8b

Please sign in to comment.