-
Notifications
You must be signed in to change notification settings - Fork 7
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feat/poc add dao layer #2248
base: dev
Are you sure you want to change the base?
Feat/poc add dao layer #2248
Changes from all commits
ae86cdf
c2847a7
fbc2094
dd40cf3
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,94 @@ | ||
# Copyright (c) 2024, RTE (https://www.rte-france.com) | ||
# | ||
# See AUTHORS.txt | ||
# | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
# | ||
# SPDX-License-Identifier: MPL-2.0 | ||
# | ||
# This file is part of the Antares project. | ||
|
||
from typing import List | ||
|
||
from antarest.study.business.link.LinkDAO import LinkDAO | ||
from antarest.study.business.model.link_model import LinkDTO | ||
from antarest.study.model import Study | ||
|
||
|
||
class CompositeLinkDAO(LinkDAO): | ||
""" | ||
CompositeLinkDAO acts as a composite data access object (DAO) for managing study links. | ||
It delegates operations to two underlying DAOs: one for cache and one for persistent storage. | ||
This class ensures that the cache and storage are kept in sync during operations. | ||
|
||
Attributes: | ||
cache_dao (LinkDAO): The DAO responsible for managing links in the cache. | ||
storage_dao (LinkDAO): The DAO responsible for managing links in persistent storage. | ||
""" | ||
|
||
def __init__(self, cache_dao: LinkDAO, storage_dao: LinkDAO): | ||
""" | ||
Initializes the CompositeLinkDAO with a cache DAO and a storage DAO. | ||
|
||
Args: | ||
cache_dao (LinkDAO): DAO for managing links in the cache. | ||
storage_dao (LinkDAO): DAO for managing links in persistent storage. | ||
""" | ||
self.cache_dao = cache_dao | ||
self.storage_dao = storage_dao | ||
|
||
def get_all_links(self, study: Study) -> List[LinkDTO]: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. about naming: I think we should reserve DTO for the web layer. The DAO layer should have its own suffix, for example Data. |
||
""" | ||
Retrieves all links for a given study. | ||
|
||
This method first tries to retrieve links from the cache. If the cache is empty, | ||
it fetches the links from the persistent storage, updates the cache with the retrieved links, | ||
and then returns the list of links. | ||
|
||
Args: | ||
study (Study): The study for which to retrieve the links. | ||
|
||
Returns: | ||
List[LinkDTO]: A list of all links associated with the study. | ||
""" | ||
links = self.cache_dao.get_all_links(study) | ||
if not links: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There is a difficulty with using the We should have an
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Or maybe a third alternative !
It will rely on a quite "hidden" convention but why not ! Relying on exceptions is quite pythonic from what I know ... |
||
links = self.storage_dao.get_all_links(study) | ||
for link in links: | ||
self.cache_dao.create_link(study, link) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think we should use the create_link method here, it's not the same thing to actually create a link and to just put an existing link in the cache. |
||
return links | ||
|
||
def create_link(self, study: Study, link_dto: LinkDTO) -> LinkDTO: | ||
""" | ||
Creates a new link for a given study. | ||
|
||
This method adds the link to both the persistent storage and the cache | ||
to ensure that they remain consistent. | ||
|
||
Args: | ||
study (Study): The target study. | ||
link_dto (LinkDTO): The link to be added. | ||
|
||
Returns: | ||
LinkDTO: The link that was added. | ||
""" | ||
self.storage_dao.create_link(study, link_dto) | ||
self.cache_dao.create_link(study, link_dto) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I am not sure it's a good idea that the cache implementation has to implement the modifications methods: In general, we populate a cache from what we read from the underlying storage, so we could just do nothing with the cache here except invalidate it (remove the link from the cache). Then on the next read it would be put in the cache again. |
||
return link_dto | ||
|
||
def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None: | ||
""" | ||
Deletes a specific link for a given study. | ||
|
||
This method removes the link from both the persistent storage and the cache | ||
to ensure consistency between the two. | ||
|
||
Args: | ||
study (Study): The study containing the link to be deleted. | ||
area1_id (str): The ID of the source area of the link. | ||
area2_id (str): The ID of the target area of the link. | ||
""" | ||
self.storage_dao.delete_link(study, area1_id, area2_id) | ||
self.cache_dao.delete_link(study, area1_id, area2_id) |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,57 @@ | ||
# Copyright (c) 2024, RTE (https://www.rte-france.com) | ||
# | ||
# See AUTHORS.txt | ||
# | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
# | ||
# SPDX-License-Identifier: MPL-2.0 | ||
# | ||
# This file is part of the Antares project. | ||
from abc import ABC, abstractmethod | ||
from typing import List | ||
|
||
from antarest.study.business.model.link_model import LinkDTO | ||
from antarest.study.model import Study | ||
|
||
|
||
class LinkDAO(ABC): | ||
""" | ||
DAO interface for managing the links of a study. | ||
Provides methods to access, add, save, and delete links. | ||
""" | ||
|
||
@abstractmethod | ||
def get_all_links(self, study: Study) -> List[LinkDTO]: | ||
""" | ||
Retrieves all the links associated with a study. | ||
Args: | ||
study (Study): The study for which to retrieve the links. | ||
Returns: | ||
List[LinkInternal]: A list of links associated with the study. | ||
""" | ||
pass | ||
|
||
@abstractmethod | ||
def create_link(self, study: Study, link: LinkDTO) -> LinkDTO: | ||
""" | ||
Adds an individual link to a study. | ||
Args: | ||
study (Study): The target study. | ||
link (LinkInternal): The link to be added. | ||
""" | ||
pass | ||
|
||
@abstractmethod | ||
def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None: | ||
""" | ||
Deletes a specific link associated with a study. | ||
|
||
Args: | ||
study (Study): The study containing the link to be deleted. | ||
area1_id (str): The ID of the source area of the link. | ||
area2_id (str): The ID of the target area of the link. | ||
""" | ||
pass | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,124 @@ | ||
# Copyright (c) 2024, RTE (https://www.rte-france.com) | ||
# | ||
# See AUTHORS.txt | ||
# | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
# | ||
# SPDX-License-Identifier: MPL-2.0 | ||
# | ||
# This file is part of the Antares project. | ||
|
||
import json | ||
from typing import List | ||
|
||
from antarest.core.interfaces.cache import ICache | ||
from antarest.study.business.link.LinkDAO import LinkDAO | ||
from antarest.study.business.model.link_model import LinkDTO | ||
from antarest.study.model import Study | ||
|
||
|
||
class LinkFromCacheDAO(LinkDAO): | ||
""" | ||
LinkFromCacheDAO is responsible for managing study links in the cache. | ||
It provides methods to retrieve, create, and delete links stored in the cache. | ||
|
||
Attributes: | ||
redis (ICache): The cache interface used for storing and retrieving links. | ||
""" | ||
|
||
def __init__(self, redis: ICache): | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We should not name it redis, the cache implementation is not always based on redis |
||
""" | ||
Initializes the LinkFromCacheDAO with a cache interface. | ||
|
||
Args: | ||
redis (ICache): The cache interface used for managing links. | ||
""" | ||
self.redis = redis | ||
|
||
def _get_cache_key(self, study_id: str) -> str: | ||
""" | ||
Generates a unique key for storing the links of a study. | ||
|
||
Args: | ||
study_id (str): The ID of the studyy. | ||
|
||
Returns: | ||
str: A unique cache key for the study. | ||
""" | ||
return f"study:{study_id}:links" | ||
|
||
def get_all_links(self, study: Study) -> List[LinkDTO]: | ||
""" | ||
Retrieves all links for a given study from the cache. | ||
|
||
This method first checks if the cache contains any links for the given study. | ||
If no links are found, it returns an empty list. Otherwise, it deserializes the | ||
links and converts them into instances of LinkDTO. | ||
|
||
Args: | ||
study (Study): The study for which to retrieve the links. | ||
|
||
Returns: | ||
List[LinkDTO]: A list of links associated with the study. | ||
""" | ||
cache_key = self._get_cache_key(study.id) | ||
cached_links = self.redis.get(cache_key) | ||
if not cached_links: | ||
return [] | ||
links_data = cached_links["links"] | ||
return [LinkDTO.model_validate(link_data) for link_data in links_data] | ||
|
||
def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None: | ||
""" | ||
Deletes all links associated with a given study from the cache. | ||
|
||
This method invalidates the cache for the specified study, ensuring | ||
that any subsequent requests will require fetching fresh data. | ||
|
||
Args: | ||
study (Study): The target study containing the links to delete. | ||
area1_id (str): The source area of the link to delete. | ||
area2_id (str): The target area of the link to delete. | ||
""" | ||
cache_key = self._get_cache_key(study.id) | ||
self.redis.invalidate(cache_key) | ||
|
||
def create_link(self, study: Study, link_dto: LinkDTO) -> LinkDTO: | ||
""" | ||
Adds a new link to the study in the cache. | ||
|
||
This method retrieves the current links from the cache, appends the new link, | ||
and updates the chache with the modified list. If the cache does not exist | ||
or is invalid, it initialises a new cache entry. | ||
|
||
Args: | ||
study (Study): The study to which the link belongs. | ||
link_dto (LinkDTO): The link to add. | ||
|
||
Returns: | ||
LinkDTO: The newly added link. | ||
""" | ||
cache_key = self._get_cache_key(study.id) | ||
cached_links = self.redis.get(cache_key) | ||
|
||
if isinstance(cached_links, str): | ||
try: | ||
cached_links = json.loads(cached_links) | ||
except json.JSONDecodeError: | ||
cached_links = {"links": []} | ||
elif not isinstance(cached_links, dict): | ||
cached_links = {"links": []} | ||
|
||
links_data = cached_links.get("links", []) | ||
if not isinstance(links_data, list): | ||
links_data = [] | ||
|
||
link_data = link_dto.model_dump(by_alias=True, exclude_unset=True) | ||
links_data.append(link_data) | ||
|
||
cached_links["links"] = links_data | ||
self.redis.put(cache_key, cached_links) | ||
|
||
return link_dto |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,121 @@ | ||
# Copyright (c) 2024, RTE (https://www.rte-france.com) | ||
# | ||
# See AUTHORS.txt | ||
# | ||
# This Source Code Form is subject to the terms of the Mozilla Public | ||
# License, v. 2.0. If a copy of the MPL was not distributed with this | ||
# file, You can obtain one at http://mozilla.org/MPL/2.0/. | ||
# | ||
# SPDX-License-Identifier: MPL-2.0 | ||
# | ||
# This file is part of the Antares project. | ||
import typing as t | ||
from typing import List | ||
|
||
from antares.study.version import StudyVersion | ||
|
||
from antarest.study.business.link.LinkDAO import LinkDAO | ||
from antarest.study.business.model.link_model import LinkDTO, LinkInternal | ||
from antarest.study.business.utils import execute_or_add_commands | ||
from antarest.study.model import Study | ||
from antarest.study.storage.variantstudy.model.command.create_link import CreateLink | ||
from antarest.study.storage.variantstudy.model.command.remove_link import RemoveLink | ||
|
||
|
||
class LinkFromStorageDAO(LinkDAO): | ||
""" | ||
LinkFromStorageDAO is responsible for managing study links in persistent storage. | ||
It provides methods to retrieve, create, and delete links directly in the underlying storage. | ||
|
||
Attributes: | ||
storage_service: The service used to interact with the persistent storage. | ||
""" | ||
|
||
def __init__(self, storage_service) -> None: | ||
""" | ||
Initializes the LinkFromStorageDAO with a storage service. | ||
|
||
Args: | ||
storage_service: The service responsible for interacting with persistent storage. | ||
""" | ||
self.storage_service = storage_service | ||
|
||
def get_all_links(self, study: Study) -> List[LinkDTO]: | ||
""" | ||
Retrieves all links for a given study from persistent storage. | ||
|
||
This method reads the study configuration and retrieves all links between areas | ||
defined in the study. | ||
|
||
Args: | ||
study (Study): The study for which to retrieve the links. | ||
|
||
Returns: | ||
List[LinkDTO]: A list of links associated with the study. | ||
""" | ||
file_study = self.storage_service.get_storage(study).get_raw(study) | ||
result: t.List[LinkDTO] = [] | ||
|
||
for area_id, area in file_study.config.areas.items(): | ||
links_config = file_study.tree.get(["input", "links", area_id, "properties"]) | ||
|
||
for link in area.links: | ||
link_tree_config: t.Dict[str, t.Any] = links_config[link] | ||
link_tree_config.update({"area1": area_id, "area2": link}) | ||
|
||
link_internal = LinkInternal.model_validate(link_tree_config) | ||
result.append(link_internal.to_dto()) | ||
|
||
return result | ||
|
||
def create_link(self, study: Study, link_dto: LinkDTO) -> LinkDTO: | ||
""" | ||
Creates a new link for a study in persistent storage. | ||
|
||
This method converts the provided LinkDTO into an internal model, | ||
then creates a command to add the link to the study. The command is executed | ||
immediately or queued for later execution. | ||
|
||
Args: | ||
study (Study): The study where the link should be created. | ||
link_dto (LinkDTO): The link to be added. | ||
|
||
Returns: | ||
LinkDTO: The newly created link. | ||
""" | ||
link = link_dto.to_internal(StudyVersion.parse(study.version)) | ||
|
||
storage_service = self.storage_service.get_storage(study) | ||
file_study = storage_service.get_raw(study) | ||
|
||
command = CreateLink( | ||
area1=link.area1, | ||
area2=link.area2, | ||
parameters=link.model_dump(exclude_none=True), | ||
command_context=self.storage_service.variant_study_service.command_factory.command_context, | ||
study_version=file_study.config.version, | ||
) | ||
|
||
execute_or_add_commands(study, file_study, [command], self.storage_service) | ||
return link_dto | ||
|
||
def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None: | ||
""" | ||
Deletes a specific link from a study in persistent storage. | ||
|
||
This method creates a command to remove the link from the study. The command | ||
is executed immediately or queued for later excecution. | ||
|
||
Args: | ||
study (Study): The study containing the link to be deleted. | ||
area1_id (str): The ID of the source area of the link. | ||
area2_id (str): The ID of the target area of the link. | ||
""" | ||
file_study = self.storage_service.get_storage(study).get_raw(study) | ||
command = RemoveLink( | ||
area1=area1_id, | ||
area2=area2_id, | ||
command_context=self.storage_service.variant_study_service.command_factory.command_context, | ||
study_version=file_study.config.version, | ||
) | ||
execute_or_add_commands(study, file_study, [command], self.storage_service) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In the end the DAO layer should not use the commands, it's the opposite: Otherwise, we will not be able to change the underlying storage without changing the commands implementation. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A general comment:
I think we should isolate all the DAO layer in its own package (I propose
antarest.study.dao
).This DAO layer must be completely independent from the "business" layer which will use it.
The dependency must go only from the business layer to the DAO layer, not the other way around, so we should not mix them in the same package.