Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release 2.0.1 #282

Merged
merged 15 commits into from
Oct 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .readthedocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,12 @@ sphinx:
configuration: "docs/source/conf.py"
fail_on_warning: false

build:
os: "ubuntu-22.04"
tools:
python: "3.11"

python:
version: "3.7"
install:
- requirements: "docs/requirements.txt"
- method: "pip"
Expand Down
15 changes: 15 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,21 @@ All notable changes to this project will be documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/).

## [2.0.1]

### Changed

- #276 - Removed upper version boud for `structlog` dependency

### Fixed

- #281 - Properly deprecated `DiffSync` class name
- #273 - Properly capitalized `DiffSync` in documentation
- #273 - Removed more mentions of `DiffSync` in favor of `Adapter`
- #274 - Fixed doc section title for getting started
- #269 - Fixed wording for a couple of docstrings
- #265 - Fixed readthedocs build

## [2.0.0]

### Changed
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,16 @@ DiffSync is at its most useful when you have multiple sources or sets of data to

DiffSync acts as an intermediate translation layer between all of the data sets you are diffing and/or syncing. In practical terms, this means that to use DiffSync, you will define a set of data models as well as the “adapters” needed to translate between each base data source and the data model. In Python terms, the adapters will be subclasses of the `Adapter` class, and each data model class will be a subclass of the `DiffSyncModel` class.

![Diffsync Components](https://raw.githubusercontent.com/networktocode/diffsync/develop/docs/images/diffsync_components.png "Diffsync Components")
![DiffSync Components](https://raw.githubusercontent.com/networktocode/diffsync/develop/docs/images/diffsync_components.png "DiffSync Components")


Once you have used each adapter to load each data source into a collection of data model records, you can then ask DiffSync to “diff” the two data sets, and it will produce a structured representation of the difference between them. In Python, this is accomplished by calling the `diff_to()` or `diff_from()` method on one adapter and passing the other adapter as a parameter.

![Diffsync Diff Creation](https://raw.githubusercontent.com/networktocode/diffsync/develop/docs/images/diffsync_diff_creation.png "Diffsync Diff Creation")
![DiffSync Diff Creation](https://raw.githubusercontent.com/networktocode/diffsync/develop/docs/images/diffsync_diff_creation.png "DiffSync Diff Creation")

You can also ask DiffSync to “sync” one data set onto the other, and it will instruct your adapter as to the steps it needs to take to make sure that its data set accurately reflects the other. In Python, this is accomplished by calling the `sync_to()` or `sync_from()` method on one adapter and passing the other adapter as a parameter.

![Diffsync Sync](https://raw.githubusercontent.com/networktocode/diffsync/develop/docs/images/diffsync_sync.png "Diffsync Sync")
![DiffSync Sync](https://raw.githubusercontent.com/networktocode/diffsync/develop/docs/images/diffsync_sync.png "DiffSync Sync")

# Simple Example

Expand Down
21 changes: 8 additions & 13 deletions diffsync/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@
Any,
Set,
)
import warnings
from typing_extensions import deprecated

from pydantic import ConfigDict, BaseModel, PrivateAttr
import structlog # type: ignore
Expand Down Expand Up @@ -425,7 +425,7 @@ def remove_child(self, child: "DiffSyncModel") -> None:


class Adapter: # pylint: disable=too-many-public-methods
"""Class for storing a group of DiffSyncModel instances and diffing/synchronizing to another DiffSync instance."""
"""Class for storing a group of DiffSyncModel instances and diffing/synchronizing to another Adapter instance."""

# In any subclass, you would add mapping of names to specific model classes here:
# modelname1 = MyModelClass1
Expand Down Expand Up @@ -480,7 +480,7 @@ def __init_subclass__(cls) -> None:
raise AttributeError(f'top_level references attribute "{name}" but it is not a DiffSyncModel subclass!')

def __str__(self) -> StrType:
"""String representation of a DiffSync."""
"""String representation of an Adapter."""
if self.type != self.name:
return f'{self.type} "{self.name}"'
return self.type
Expand Down Expand Up @@ -526,7 +526,7 @@ def dict(self, exclude_defaults: bool = True, **kwargs: Any) -> Dict[str, Dict[s
return data

def str(self, indent: int = 0) -> StrType:
"""Build a detailed string representation of this DiffSync."""
"""Build a detailed string representation of this Adapter."""
margin = " " * indent
output = ""
for modelname in self.top_level:
Expand Down Expand Up @@ -835,7 +835,7 @@ def remove(self, obj: DiffSyncModel, remove_children: bool = False) -> None:
def get_or_instantiate(
self, model: Type[DiffSyncModel], ids: Dict, attrs: Optional[Dict] = None
) -> Tuple[DiffSyncModel, bool]:
"""Attempt to get the object with provided identifiers or instantiate it with provided identifiers and attrs.
"""Attempt to get the object with provided identifiers or instantiate and add it with provided identifiers and attrs.

Args:
model: The DiffSyncModel to get or create.
Expand All @@ -848,7 +848,7 @@ def get_or_instantiate(
return self.store.get_or_instantiate(model=model, ids=ids, attrs=attrs)

def get_or_add_model_instance(self, obj: DiffSyncModel) -> Tuple[DiffSyncModel, bool]:
"""Attempt to get the object with provided obj identifiers or instantiate obj.
"""Attempt to get the object with provided obj identifiers or add obj.

Args:
obj: An obj of the DiffSyncModel to get or add.
Expand Down Expand Up @@ -894,15 +894,10 @@ def count(self, model: Union[StrType, "DiffSyncModel", Type["DiffSyncModel"], No
return self.store.count(model=model)


def DiffSync(*args: Any, **kwargs: Any) -> Adapter: # noqa pylint: disable=invalid-name
@deprecated("'diffsync.DiffSync' is deprecated and will be removed with 2.1, use 'diffsync.Adapter' instead.")
class DiffSync(Adapter):
"""For backwards-compatibility, keep around the old name."""

warnings.warn(
"'diffsync.DiffSync' is deprecated and will be removed with 2.1, use 'diffsync.Adapter' instead.",
DeprecationWarning,
)
return Adapter(*args, **kwargs)


# DiffSyncModel references Adapter and Adapter references DiffSyncModel. Break the typing loop:
DiffSyncModel.model_rebuild()
6 changes: 3 additions & 3 deletions docs/requirements.txt
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
m2r2==0.2.7
m2r2==0.3.3.post2
mistune==0.8.4
sphinx==4.5.0
sphinx==6.2.1
toml==0.10.2
sphinx-rtd-theme==1.0.0
sphinx-rtd-theme==2.0.0
8 changes: 4 additions & 4 deletions docs/source/core_engine/01-flags.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,8 +24,8 @@ diff = nautobot.diff_from(local, flags=flags)
| Name | Description | Binary Value |
|---|---|---|
| CONTINUE_ON_FAILURE | Continue synchronizing even if failures are encountered when syncing individual models. | 0b1 |
| SKIP_UNMATCHED_SRC | Ignore objects that only exist in the source/"from" DiffSync when determining diffs and syncing. If this flag is set, no new objects will be created in the target/"to" DiffSync. | 0b10 |
| SKIP_UNMATCHED_DST | Ignore objects that only exist in the target/"to" DiffSync when determining diffs and syncing. If this flag is set, no objects will be deleted from the target/"to" DiffSync. | 0b100 |
| SKIP_UNMATCHED_SRC | Ignore objects that only exist in the source/"from" adapter when determining diffs and syncing. If this flag is set, no new objects will be created in the target/"to" adapter. | 0b10 |
| SKIP_UNMATCHED_DST | Ignore objects that only exist in the target/"to" adapter when determining diffs and syncing. If this flag is set, no objects will be deleted from the target/"to" adapter. | 0b100 |
| SKIP_UNMATCHED_BOTH | Convenience value combining both SKIP_UNMATCHED_SRC and SKIP_UNMATCHED_DST into a single flag | 0b110 |
| LOG_UNCHANGED_RECORDS | If this flag is set, a log message will be generated during synchronization for each model, even unchanged ones. | 0b1000 |

Expand Down Expand Up @@ -57,8 +57,8 @@ class MyAdapter(Adapter):
|---|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---|
| IGNORE | Do not render diffs containing this model; do not make any changes to this model when synchronizing. Can be used to indicate a model instance that exists but should not be changed by DiffSync. | 0b1 |
| SKIP_CHILDREN_ON_DELETE | When deleting this model, do not recursively delete its children. Can be used for the case where deletion of a model results in the automatic deletion of all its children. | 0b10 |
| SKIP_UNMATCHED_SRC | Ignore the model if it only exists in the source/"from" DiffSync when determining diffs and syncing. If this flag is set, no new model will be created in the target/"to" DiffSync. | 0b100 |
| SKIP_UNMATCHED_DST | Ignore the model if it only exists in the target/"to" DiffSync when determining diffs and syncing. If this flag is set, the model will not be deleted from the target/"to" DiffSync. | 0b1000 |
| SKIP_UNMATCHED_SRC | Ignore the model if it only exists in the source/"from" adapter when determining diffs and syncing. If this flag is set, no new model will be created in the target/"to" adapter. | 0b100 |
| SKIP_UNMATCHED_DST | Ignore the model if it only exists in the target/"to" adapter when determining diffs and syncing. If this flag is set, the model will not be deleted from the target/"to" adapter. | 0b1000 |
| SKIP_UNMATCHED_BOTH | Convenience value combining both SKIP_UNMATCHED_SRC and SKIP_UNMATCHED_DST into a single flag | 0b1100 |
| NATURAL_DELETION_ORDER | When deleting, delete children before instances of this model. | 0b10000 |

Expand Down
4 changes: 2 additions & 2 deletions docs/source/core_engine/03-store.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
# Store backends

By default, `Diffsync` supports a local memory storage. All the loaded models from the adapters will be stored in memory, and become available for the diff calculation and sync process. This default behavior works well when executing all the steps in the same process, having access to the same memory space. However, if you want to scale out the execution of the tasks, running it in different processes or in totally different workers, a more distributed memory support is necessary.
By default, DiffSync supports a local memory storage. All the loaded models from the adapters will be stored in memory, and become available for the diff calculation and sync process. This default behavior works well when executing all the steps in the same process, having access to the same memory space. However, if you want to scale out the execution of the tasks, running it in different processes or in totally different workers, a more distributed memory support is necessary.

The `store` is a class attribute in the `Adapter` class, but all the store operations in that class are abstracted in the following methods: `get_all_model_names`, `get`, `get_by_uids`, `add`, `update`, `remove`, `get_or_instantiate`, `update_or_instantiate` and `count`.

## Use the `LocalStore` Backend

When you initialize the `Diffsync` Adapter class, there is an optional keyed-argument, `internal_storage_engine`, defaulting to the `LocalStore` class.
When you initialize the `Adapter` class, there is an optional keyed-argument, `internal_storage_engine`, defaulting to the `LocalStore` class.

```python
>> > from diffsync import Adapter
Expand Down
14 changes: 7 additions & 7 deletions docs/source/getting_started/01-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ class Cable(DiffSyncModel):
[...]


class Nautobot(DiffSync):
class Nautobot(Adapter):
site = Site
device = Device
interface = Interface
Expand All @@ -121,15 +121,15 @@ Would result in processing in the following order for each element until there i
- ip_address
- cable

> Note: This applies to the actual diff sync (`Diffsync.sync_from/Diffsync.sync_to`), and not the loading of the data (`Diffsync.load`), which is up to the developer to determine the order.
> Note: This applies to the actual diff sync (`Adapter.sync_from` and `Adapter.sync_to`), and not the loading of the data (`Adapter.load`), which is up to the developer to determine the order.

This can be visualized here in the included diagram.

![Preorder Tree Traversal](../../images/preorder-tree-traversal.drawio.png "Preorder Tree Traversal")

### Mapping Tree Traversal with `get_tree_traversal` method

For your convenience, there is a helper method that will provide a mapping of the order. The `DiffSync.get_tree_traversal()` class method will return a tree-like string, or optionally a dictionary when passing the `as_dict=True` parameter.
For your convenience, there is a helper method that will provide a mapping of the order. The `Adapter.get_tree_traversal()` class method will return a tree-like string, or optionally a dictionary when passing the `as_dict=True` parameter.

```python
>>> from nautobot_device_onboarding.network_importer.adapters.network_device.adapter import NetworkImporterAdapter
Expand All @@ -150,7 +150,7 @@ NetworkImporterAdapter
To add a site to the local cache/store, you need to pass a valid `DiffSyncModel` object to the `add()` function.

```python
class BackendA(DiffSync):
class BackendA(Adapter):
[...]

def load(self):
Expand Down Expand Up @@ -203,14 +203,14 @@ class Device(DiffSyncModel):
If you prefer to update the entire remote system with the final state after performing all individual create/update/delete operations (as might be the case if your "remote system" is a single YAML or JSON file), the easiest place to implement this logic is in the `sync_complete()` callback method that is automatically invoked by DiffSync upon completion of a sync operation.

```python
class BackendA(DiffSync):
class BackendA(Adapter):
[...]

def sync_complete(self, source: DiffSync, diff: Diff, flags: DiffSyncFlags, logger: structlog.BoundLogger):
def sync_complete(self, source: Adapter, diff: Diff, flags: DiffSyncFlags, logger: structlog.BoundLogger):
## TODO add your own logic to update the remote system now.
# The various parameters passed to this method are for your convenience in implementing more complex logic, and
# can be ignored if you do not need them.
#
# The default DiffSync.sync_complete() method does nothing, but it's always a good habit to call super():
# The default Adapter.sync_complete() method does nothing, but it's always a good habit to call super():
super().sync_complete(source, diff, flags, logger)
```
2 changes: 1 addition & 1 deletion docs/source/getting_started/index.rst
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#########
Upgrading
Getting started
#########

.. mdinclude:: 01-getting-started.md
4 changes: 2 additions & 2 deletions docs/source/upgrading/01-upgrading-to-2.0.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@

With diffsync 2.0, there a couple of breaking changes. What they are and how to deal with them is described in this document.

## Rename of the `diffsync.Diffsync` class to `diffsync.Adapter`
## Rename of the `diffsync.DiffSync` class to `diffsync.Adapter`

The main diffsync class `diffsync.Diffsync` has been renamed to `diffsync.Adapter` as we have found that this is the verbiage that is most often used by users and explains the intent of the class clearer. The old name will still be around until 2.1, but is considered deprecated at this point.
The main diffsync class `diffsync.DiffSync` has been renamed to `diffsync.Adapter` as we have found that this is the verbiage that is most often used by users and explains the intent of the class clearer. The old name will still be around until 2.1, but is considered deprecated at this point.

As a consequence, a lot of fields have been renamed all across diffsync. To the end user, this will most prominently appear in the signature of the `create` method, where you will have to rename the `diffsync` parameter to `adapter`.

Expand Down
2 changes: 1 addition & 1 deletion examples/03-remote-system/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ export NAUTOBOT_TOKEN = "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa"

The first time you run this example, a lot of changes should be reported between Nautobot and the local data because by default the demo instance doesn't have the subregion defined.
After the first sync, on subsequent runs, the diff should show no changes.
At this point, `Diffsync` will be able to identify and fix all changes in Nautobot. You can try to add/update or delete any country in Nautobot and DiffSync will automatically catch it and it will fix it with running in sync mode.
At this point, DiffSync will be able to identify and fix all changes in Nautobot. You can try to add/update or delete any country in Nautobot and DiffSync will automatically catch it and it will fix it with running in sync mode.

```
### DIFF Compare the data between Nautobot and the local JSON file.
Expand Down
2 changes: 1 addition & 1 deletion examples/05-nautobot-peeringdb/adapter_nautobot.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Diffsync adapter class for Nautobot."""
"""DiffSync adapter class for Nautobot."""
# pylint: disable=import-error,no-name-in-module
import pynautobot
from models import RegionModel, SiteModel
Expand Down
2 changes: 1 addition & 1 deletion examples/05-nautobot-peeringdb/adapter_peeringdb.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
"""Diffsync adapter class for PeeringDB."""
"""DiffSync adapter class for PeeringDB."""
# pylint: disable=import-error,no-name-in-module
import os
import requests
Expand Down
Loading
Loading