Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Define NLDI Feature model #386

Open
gzt5142 opened this issue Apr 24, 2023 · 1 comment
Open

Define NLDI Feature model #386

gzt5142 opened this issue Apr 24, 2023 · 1 comment
Assignees

Comments

@gzt5142
Copy link
Collaborator

gzt5142 commented Apr 24, 2023

Define a common Feature model for use across the project (crawler client, server, other tools).
Can be a 'vanilla' dataclass, a pydantic model, or an ORM model. All sides of the workflow should agree on what a Feature looks like and how it is structured.

The python port of the crawler defines a SQLAlchemy ORM model (see https://github.com/gzt5142/nldi-crawler-py/blob/b87418874cb90cbf32ce8ea25bdbbddcef19c355/src/nldi_crawler/ingestor.py#L62

We should review this as a team to make sure that it matches the table schema exactly, and that it contains all of the features we will need going forward.

A common and importable model will require a minor refactor of the existing crawler port.

@gzt5142
Copy link
Collaborator Author

gzt5142 commented May 30, 2023

I'm using a pydantic dataclass to represent a single ingested feature -- a row of the feature table.

See https://github.com/gzt5142/nldi-crawler-py/blob/7e0aa4c6230c983a4931561ab6322c2d68ece527/src/nldi_crawler/feature.py#L14

The sqlalchemy mechanism to bind this dataclass to the relevant table is modeled here:
https://github.com/gzt5142/nldi-crawler-py/blob/7e0aa4c6230c983a4931561ab6322c2d68ece527/src/nldi_crawler/ingestor.py#L59

This method relies on the implied contract that the sql data access mechanism will connect to an appropriate database with a 'nldi_data' schema.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant