Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add the SqliteDosStorage storage backend #6148

Merged
merged 3 commits into from
Nov 16, 2023
Merged

Add the SqliteDosStorage storage backend #6148

merged 3 commits into from
Nov 16, 2023

Commits on Nov 16, 2023

  1. Add the SqliteDosStorage storage backend 

    The implementation subclasses the `PsqlDosBackend` and replaces the
PostgreSQL database with an sqlite database. By doing so, the
initialization of the storage only requires a directory on the local
file system where it will create the sqlite file for the database and a
container for the disk-objectstore.

The advantage of this `sqlite_dos` storage over the default `psql_dos`
is that it doesn't require a system service like PostgreSQL. As a
result, creating the storage is very straightforward and can be done
with almost no setup. The advantage over the existing `sqlite_zip` is
that the `sqlite_dos` is not read-only but can be used to write data
as well.

Combined with the `verdi profile setup` command, a working profile can
be created with a single command:

    verdi profile setup core.sqlite_dos -n --profile name --email e@mail

This makes this storage backend very useful for tutorials and demos
that don't rely on performance.
    @sphuber
    sphuber committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    0cf81c8 View commit details
    Browse the repository at this point in the history

  2. SqliteZipBackend: Return self in store 

    The `store` method of the `SqliteEntityOverride` class, used by the
`SqliteZipBackend` storage backend (and with that all other backends to
subclass this), did not return `self`. This is in conflict with the
signature of the base class that it is overriding.

Since the `SqliteZipBackend` is read-only and so `store` would never be
called, this problem went unnoticed. However, with the addition of the
`SqliteDosStorage` backend which is *not* read-only, this bug would
surface when trying to store a node since certain methods rely on this
method returning the node instance itself.
    @sphuber
    sphuber committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    527f515 View commit details
    Browse the repository at this point in the history

  3. Fix QueryBuilder.count for storage backends using sqlite 

    The storage backends that use sqlite instead of PostgreSQL, i.e.,
`core.sqlite_dos`, `core.sqlite_temp` and `core.sqlite_zip`, piggy back
of the ORM models defined by the `core.psql_dos` backend by dynamically
converting to the sqlite equivalent database models.

The current implementation of `SqlaGroup.count` would except when used
with an sqlite backend since certain columns would be ambiguously
defined:

    sqlite3.OperationalError: ambiguous column name: db_dbgroup.id

This is fixed by explicitly wrapping the classes that are joined in
`sqlalchemy.orm.aliased` which will force sqlalchemy to properly alias
each class removing the ambiguity.
    @sphuber
    sphuber committed Nov 16, 2023
    Configuration menu
    Copy the full SHA
    8087995 View commit details
    Browse the repository at this point in the history