Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[obsolete] pyDKB storages #244

Open
wants to merge 12 commits into
base: master
Choose a base branch
from
Open

[obsolete] pyDKB storages #244

wants to merge 12 commits into from

Conversation

mgolosova
Copy link
Collaborator

@mgolosova mgolosova commented Apr 26, 2019

Rewritten with #277

This PR introduces to pyDKB library new submodule storages.

The module is intended to accumulate all the functionality related to the interaction with the storages we use, including both external and internal ones. It basically provides unified way to interact with any of them: object of Storage class, that understands basic set of methods, like:

  • apply configuration (create client object(s));
  • send get() request (get item by ID);
  • send custom (parametric) query.

It is needed as we have at least 4 pieces of (Python) code that requires interaction with ES@DKB or Chicago ES:

Also, almost same way as in the last case, we need to update Stage 95 (AMI-related stage) (update: see #246).
To avoid copy-and-paste code cloning, this common module is developed.


ToDo / to think about:

  • It is supposed to be used in API server as well (only that it would be better to have possibility to import this submodule without the rest of the library).
  • Maybe it would be better to place it inside the pyDKB.common module. (discussed here);
  • Stage 009 dbConnection also calls for the integration to this submodule.
  • If there should be something DKB-specific (or, better to say, DKB@ATLAS-specific) -- it should be in a separate module, which is aware of the data scheme, etc. (discussed here).

For now I am going to leave this PR as is in favor of safe update issue (for details please refer to Trello).

@mgolosova mgolosova self-assigned this Apr 26, 2019
Sometimes we have to interact with the same storage in different
scripts. When it happens we got to implement same functionality multiple
times: read configuration, check that Python module is available, create
client, ...

This submodule is a place to implement this once and then reuse whwnever
it is needed.
It would be wierd to manage module global variable from class; it would
also be wierd to get errors like ``StorageAlreadyExists`` if one just
uses the Storage functionality and doesn't want any extra logic around
it.
…`client()`).

* `log()` -- quite useful method for all the derived classes;
* `client()` -- given that one day we may want to make multiple
  simultaneous requests to the storage, it doesn'y look good to use
  instance variable `c` directly; method that returns ready-to-use client
  seems to be a better way.
There may be more than one valuable parameter for record detection (e.g.
record ID and index in ES).
Added concept of "raw" queries: it is useful when query must be stored,
yet can not be converted to the format acceptable by storage client.

Here it is JSON with %(parameter)s values: it may fail to be parsed as
proper JSON if there is something like `"taskname": %(name)s`.
@mgolosova
Copy link
Collaborator Author

Force-push was just to fix codestyle in the original commits, instead of a separate 'Codestyle fix' commit.

@Evildoor Evildoor mentioned this pull request May 23, 2019
@mgolosova mgolosova mentioned this pull request Jul 10, 2019
@mgolosova mgolosova changed the title [WIP] pyDKB storages [obsolete] pyDKB storages Jul 10, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant