Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feat/poc add dao layer #2248

Draft
wants to merge 4 commits into
base: dev
Choose a base branch
from
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 94 additions & 0 deletions antarest/study/business/link/CompositeLinkDAO.py
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A general comment:
I think we should isolate all the DAO layer in its own package (I propose antarest.study.dao).

This DAO layer must be completely independent from the "business" layer which will use it.
The dependency must go only from the business layer to the DAO layer, not the other way around, so we should not mix them in the same package.

Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@
# Copyright (c) 2024, RTE (https://www.rte-france.com)
#
# See AUTHORS.txt
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
#
# SPDX-License-Identifier: MPL-2.0
#
# This file is part of the Antares project.

from typing import List

from antarest.study.business.link.LinkDAO import LinkDAO
from antarest.study.business.model.link_model import LinkDTO
from antarest.study.model import Study


class CompositeLinkDAO(LinkDAO):
"""
CompositeLinkDAO acts as a composite data access object (DAO) for managing study links.
It delegates operations to two underlying DAOs: one for cache and one for persistent storage.
This class ensures that the cache and storage are kept in sync during operations.

Attributes:
cache_dao (LinkDAO): The DAO responsible for managing links in the cache.
storage_dao (LinkDAO): The DAO responsible for managing links in persistent storage.
"""

def __init__(self, cache_dao: LinkDAO, storage_dao: LinkDAO):
"""
Initializes the CompositeLinkDAO with a cache DAO and a storage DAO.

Args:
cache_dao (LinkDAO): DAO for managing links in the cache.
storage_dao (LinkDAO): DAO for managing links in persistent storage.
"""
self.cache_dao = cache_dao
self.storage_dao = storage_dao

def get_all_links(self, study: Study) -> List[LinkDTO]:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

about naming: I think we should reserve DTO for the web layer. The DAO layer should have its own suffix, for example Data.

"""
Retrieves all links for a given study.

This method first tries to retrieve links from the cache. If the cache is empty,
it fetches the links from the persistent storage, updates the cache with the retrieved links,
and then returns the list of links.

Args:
study (Study): The study for which to retrieve the links.

Returns:
List[LinkDTO]: A list of all links associated with the study.
"""
links = self.cache_dao.get_all_links(study)
if not links:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a difficulty with using the LinkDAO interface for the cache:
we cannot really know if the data is not in cache, or if it's just that the data is an empty list.

We should have an Optional instead, but we don't want to introduce those optionals in the interface just for the need of the cache.
So we are left with 2 options I think:

  • either have a specific interface for the cache, with optionals
  • or use the approach of the other PoC branch where the CacheDAO implementation can check itself if the data is in the cache or not

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Or maybe a third alternative !

  • the cache implementation could throw a CacheMissError, that we will catch here

It will rely on a quite "hidden" convention but why not ! Relying on exceptions is quite pythonic from what I know ...

links = self.storage_dao.get_all_links(study)
for link in links:
self.cache_dao.create_link(study, link)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we should use the create_link method here, it's not the same thing to actually create a link and to just put an existing link in the cache.

return links

def create_link(self, study: Study, link_dto: LinkDTO) -> LinkDTO:
"""
Creates a new link for a given study.

This method adds the link to both the persistent storage and the cache
to ensure that they remain consistent.

Args:
study (Study): The target study.
link_dto (LinkDTO): The link to be added.

Returns:
LinkDTO: The link that was added.
"""
self.storage_dao.create_link(study, link_dto)
self.cache_dao.create_link(study, link_dto)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure it's a good idea that the cache implementation has to implement the modifications methods:
i can be error prone because we may need to implement duplicate logic between the "storage" implementation and the "cache" implementation, and ensure they are consistent.

In general, we populate a cache from what we read from the underlying storage, so we could just do nothing with the cache here except invalidate it (remove the link from the cache). Then on the next read it would be put in the cache again.
There will be a small performance loss but it's probably not too bad, at least in a first implementation, compared to the risk of error we introduce by duplicating the modification logic

return link_dto

def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None:
"""
Deletes a specific link for a given study.

This method removes the link from both the persistent storage and the cache
to ensure consistency between the two.

Args:
study (Study): The study containing the link to be deleted.
area1_id (str): The ID of the source area of the link.
area2_id (str): The ID of the target area of the link.
"""
self.storage_dao.delete_link(study, area1_id, area2_id)
self.cache_dao.delete_link(study, area1_id, area2_id)
57 changes: 57 additions & 0 deletions antarest/study/business/link/LinkDAO.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
# Copyright (c) 2024, RTE (https://www.rte-france.com)
#
# See AUTHORS.txt
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
#
# SPDX-License-Identifier: MPL-2.0
#
# This file is part of the Antares project.
from abc import ABC, abstractmethod
from typing import List

from antarest.study.business.model.link_model import LinkDTO
from antarest.study.model import Study


class LinkDAO(ABC):
"""
DAO interface for managing the links of a study.
Provides methods to access, add, save, and delete links.
"""

@abstractmethod
def get_all_links(self, study: Study) -> List[LinkDTO]:
"""
Retrieves all the links associated with a study.
Args:
study (Study): The study for which to retrieve the links.
Returns:
List[LinkInternal]: A list of links associated with the study.
"""
pass

@abstractmethod
def create_link(self, study: Study, link: LinkDTO) -> LinkDTO:
"""
Adds an individual link to a study.
Args:
study (Study): The target study.
link (LinkInternal): The link to be added.
"""
pass

@abstractmethod
def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None:
"""
Deletes a specific link associated with a study.

Args:
study (Study): The study containing the link to be deleted.
area1_id (str): The ID of the source area of the link.
area2_id (str): The ID of the target area of the link.
"""
pass

124 changes: 124 additions & 0 deletions antarest/study/business/link/LinkFromCacheDAO.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,124 @@
# Copyright (c) 2024, RTE (https://www.rte-france.com)
#
# See AUTHORS.txt
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
#
# SPDX-License-Identifier: MPL-2.0
#
# This file is part of the Antares project.

import json
from typing import List

from antarest.core.interfaces.cache import ICache
from antarest.study.business.link.LinkDAO import LinkDAO
from antarest.study.business.model.link_model import LinkDTO
from antarest.study.model import Study


class LinkFromCacheDAO(LinkDAO):
"""
LinkFromCacheDAO is responsible for managing study links in the cache.
It provides methods to retrieve, create, and delete links stored in the cache.

Attributes:
redis (ICache): The cache interface used for storing and retrieving links.
"""

def __init__(self, redis: ICache):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should not name it redis, the cache implementation is not always based on redis

"""
Initializes the LinkFromCacheDAO with a cache interface.

Args:
redis (ICache): The cache interface used for managing links.
"""
self.redis = redis

def _get_cache_key(self, study_id: str) -> str:
"""
Generates a unique key for storing the links of a study.

Args:
study_id (str): The ID of the studyy.

Returns:
str: A unique cache key for the study.
"""
return f"study:{study_id}:links"

def get_all_links(self, study: Study) -> List[LinkDTO]:
"""
Retrieves all links for a given study from the cache.

This method first checks if the cache contains any links for the given study.
If no links are found, it returns an empty list. Otherwise, it deserializes the
links and converts them into instances of LinkDTO.

Args:
study (Study): The study for which to retrieve the links.

Returns:
List[LinkDTO]: A list of links associated with the study.
"""
cache_key = self._get_cache_key(study.id)
cached_links = self.redis.get(cache_key)
if not cached_links:
return []
links_data = cached_links["links"]
return [LinkDTO.model_validate(link_data) for link_data in links_data]

def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None:
"""
Deletes all links associated with a given study from the cache.

This method invalidates the cache for the specified study, ensuring
that any subsequent requests will require fetching fresh data.

Args:
study (Study): The target study containing the links to delete.
area1_id (str): The source area of the link to delete.
area2_id (str): The target area of the link to delete.
"""
cache_key = self._get_cache_key(study.id)
self.redis.invalidate(cache_key)

def create_link(self, study: Study, link_dto: LinkDTO) -> LinkDTO:
"""
Adds a new link to the study in the cache.

This method retrieves the current links from the cache, appends the new link,
and updates the chache with the modified list. If the cache does not exist
or is invalid, it initialises a new cache entry.

Args:
study (Study): The study to which the link belongs.
link_dto (LinkDTO): The link to add.

Returns:
LinkDTO: The newly added link.
"""
cache_key = self._get_cache_key(study.id)
cached_links = self.redis.get(cache_key)

if isinstance(cached_links, str):
try:
cached_links = json.loads(cached_links)
except json.JSONDecodeError:
cached_links = {"links": []}
elif not isinstance(cached_links, dict):
cached_links = {"links": []}

links_data = cached_links.get("links", [])
if not isinstance(links_data, list):
links_data = []

link_data = link_dto.model_dump(by_alias=True, exclude_unset=True)
links_data.append(link_data)

cached_links["links"] = links_data
self.redis.put(cache_key, cached_links)

return link_dto
121 changes: 121 additions & 0 deletions antarest/study/business/link/LinkFromStorageDAO.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,121 @@
# Copyright (c) 2024, RTE (https://www.rte-france.com)
#
# See AUTHORS.txt
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
#
# SPDX-License-Identifier: MPL-2.0
#
# This file is part of the Antares project.
import typing as t
from typing import List

from antares.study.version import StudyVersion

from antarest.study.business.link.LinkDAO import LinkDAO
from antarest.study.business.model.link_model import LinkDTO, LinkInternal
from antarest.study.business.utils import execute_or_add_commands
from antarest.study.model import Study
from antarest.study.storage.variantstudy.model.command.create_link import CreateLink
from antarest.study.storage.variantstudy.model.command.remove_link import RemoveLink


class LinkFromStorageDAO(LinkDAO):
"""
LinkFromStorageDAO is responsible for managing study links in persistent storage.
It provides methods to retrieve, create, and delete links directly in the underlying storage.

Attributes:
storage_service: The service used to interact with the persistent storage.
"""

def __init__(self, storage_service) -> None:
"""
Initializes the LinkFromStorageDAO with a storage service.

Args:
storage_service: The service responsible for interacting with persistent storage.
"""
self.storage_service = storage_service

def get_all_links(self, study: Study) -> List[LinkDTO]:
"""
Retrieves all links for a given study from persistent storage.

This method reads the study configuration and retrieves all links between areas
defined in the study.

Args:
study (Study): The study for which to retrieve the links.

Returns:
List[LinkDTO]: A list of links associated with the study.
"""
file_study = self.storage_service.get_storage(study).get_raw(study)
result: t.List[LinkDTO] = []

for area_id, area in file_study.config.areas.items():
links_config = file_study.tree.get(["input", "links", area_id, "properties"])

for link in area.links:
link_tree_config: t.Dict[str, t.Any] = links_config[link]
link_tree_config.update({"area1": area_id, "area2": link})

link_internal = LinkInternal.model_validate(link_tree_config)
result.append(link_internal.to_dto())

return result

def create_link(self, study: Study, link_dto: LinkDTO) -> LinkDTO:
"""
Creates a new link for a study in persistent storage.

This method converts the provided LinkDTO into an internal model,
then creates a command to add the link to the study. The command is executed
immediately or queued for later execution.

Args:
study (Study): The study where the link should be created.
link_dto (LinkDTO): The link to be added.

Returns:
LinkDTO: The newly created link.
"""
link = link_dto.to_internal(StudyVersion.parse(study.version))

storage_service = self.storage_service.get_storage(study)
file_study = storage_service.get_raw(study)

command = CreateLink(
area1=link.area1,
area2=link.area2,
parameters=link.model_dump(exclude_none=True),
command_context=self.storage_service.variant_study_service.command_factory.command_context,
study_version=file_study.config.version,
)

execute_or_add_commands(study, file_study, [command], self.storage_service)
return link_dto

def delete_link(self, study: Study, area1_id: str, area2_id: str) -> None:
"""
Deletes a specific link from a study in persistent storage.

This method creates a command to remove the link from the study. The command
is executed immediately or queued for later excecution.

Args:
study (Study): The study containing the link to be deleted.
area1_id (str): The ID of the source area of the link.
area2_id (str): The ID of the target area of the link.
"""
file_study = self.storage_service.get_storage(study).get_raw(study)
command = RemoveLink(
area1=area1_id,
area2=area2_id,
command_context=self.storage_service.variant_study_service.command_factory.command_context,
study_version=file_study.config.version,
)
execute_or_add_commands(study, file_study, [command], self.storage_service)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the end the DAO layer should not use the commands, it's the opposite:
the commands should use the DAO layer.

Otherwise, we will not be able to change the underlying storage without changing the commands implementation.

Empty file.
Loading
Loading