Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dataquality - Relation validation enhancements for optional from clause, query limit, unit tests #3295

Conversation

keenkeystrokes
Copy link
Contributor

@keenkeystrokes keenkeystrokes commented Dec 9, 2024

What type of PR is this?

  • Improvement

What does this PR do / why is it needed ?

  1. make from clause optional
  2. query limit - add support for limit expression in generated lambda
  3. Pure unit tests
  4. API enhancements to run only user query

Which issue(s) this PR fixes:

Other notes for reviewers:

Does this PR introduce a user-facing change?

No

@keenkeystrokes keenkeystrokes requested a review from a team as a code owner December 9, 2024 10:14
@keenkeystrokes keenkeystrokes changed the title Review dq relation from clause querylimit tests Dataquality - Relation validation enhancements for optional from clause, query limit, unit tests Dec 9, 2024
Copy link

github-actions bot commented Dec 9, 2024

Test Results

  1 016 files    1 016 suites   1h 45m 34s ⏱️
12 765 tests 12 656 ✔️ 109 💤 0
18 758 runs  18 649 ✔️ 109 💤 0

Results for commit 9381354.

♻️ This comment has been updated with latest results.

@rafaelbey
Copy link
Contributor

Do we need to have an explicit property for runtime? Our goal is to move towards usage of from function, and for newer grammars, adding this is taking steps on using deprecated patterns.

@keenkeystrokes
Copy link
Contributor Author

keenkeystrokes commented Dec 10, 2024

Do we need to have an explicit property for runtime? Our goal is to move towards usage of from function, and for newer grammars, adding this is taking steps on using deprecated patterns.

hi Rafael here we are ultimately using the separate property for runtime to construct the from function in the final lambda instead of having the user to define it himself in the query on which he is performing the quality checks.

functionName=from_T_m__Runtime_1__T_m_.name,
importGroup=system::imports::coreImport,
genericType=$returnType,
multiplicity=ZeroMany,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets double check that ZeroMany is correct here. I would expect multiplicity to be the same as the paramValue given the function signature

{
if (Objects.isNull(queryLimit))
{
return core_dataquality_generation_dataquality.Root_meta_external_dataquality_generateDataQualityQuery_DataQuality_1__Integer_MANY__LambdaFunction_1_(packageableElement, Lists.immutable.empty(), pureModel.getExecutionSupport());
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lets change limit to 0..1

Copy link
Contributor

@gs-jp1 gs-jp1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

approved, but follow up required.

@gs-jp1 gs-jp1 merged commit 127e82b into finos:master Dec 11, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants