Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce DataSetImplements #146

Merged
merged 23 commits into from
Nov 8, 2023
Merged

Introduce DataSetImplements #146

merged 23 commits into from
Nov 8, 2023

Conversation

nanne-aben
Copy link
Collaborator

@nanne-aben nanne-aben commented Aug 13, 2023

Allows for set-ups such as:

class Age(Schema, Protocol):
    age: Column[LongType]


T = TypeVar("T", bound=Schema)


def birthday(df: DataSetImplements[Age, T]) -> DataSet[T]:
    return transform_to_schema(
        df,
        df.typedspark_schema,
        {Age.age: Age.age + 1},
    )

Where is birthday() defined to:

  1. Take as an input DataSetImplements[Age, T]: a DataSet that implements the protocol Age as T.
  2. Return a DataSet[T]: a DataSet of the same type as the one that was provided.

@nanne-aben nanne-aben changed the title Introduce PartialDataSet Introduce DataSetExtends Oct 4, 2023
@nanne-aben nanne-aben changed the title Introduce DataSetExtends Introduce DataSetImplements Oct 4, 2023
@nanne-aben nanne-aben marked this pull request as ready for review October 25, 2023 13:20
marijncv
marijncv previously approved these changes Nov 7, 2023
@nanne-aben nanne-aben enabled auto-merge (squash) November 8, 2023 10:47
@nanne-aben nanne-aben merged commit 5e734df into main Nov 8, 2023
4 checks passed
@nanne-aben nanne-aben deleted the partial branch November 8, 2023 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants