Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE]Add isEmpty ppl command #512

Closed
YANG-DB opened this issue Aug 2, 2024 · 5 comments
Closed

[FEATURE]Add isEmpty ppl command #512

YANG-DB opened this issue Aug 2, 2024 · 5 comments
Labels
0.6 enhancement New feature or request Lang:PPL Pipe Processing Language support

Comments

@YANG-DB
Copy link
Member

YANG-DB commented Aug 2, 2024

Is your feature request related to a problem?
Add isEmpty command for PPL spark based execution driver .

Since spark sql doesnt offer such expression a possible option would be:

  • translate the expression into a where clause which is an equivalent of
SELECT CASE  
       WHEN length(trim(column_name)) = 0 THEN true 
        ELSE false     
END AS is_empty
  • register a custom expression IsEmptyExpression

Do you have any additional context?

@YANG-DB YANG-DB added enhancement New feature or request untriaged Lang:PPL Pipe Processing Language support labels Aug 2, 2024
@YANG-DB YANG-DB moved this to Todo in PPL Commands Aug 2, 2024
@YANG-DB YANG-DB removed the untriaged label Aug 6, 2024
@YANG-DB YANG-DB self-assigned this Aug 20, 2024
@YANG-DB YANG-DB added the 0.6 label Aug 20, 2024
@YANG-DB YANG-DB removed their assignment Aug 21, 2024
@kt-eliatra
Copy link
Contributor

kt-eliatra commented Sep 5, 2024

@YANG-DB
hi, just to confirm, isempty as well as ispresent and isblank - all of them should be implemented as commands?
e.g source=table | isempty(column);
Or rather as a functions? e.g. source=table | fields col_a, col_b | where isempty(col_c)?

@YANG-DB
Copy link
Member Author

YANG-DB commented Sep 5, 2024

IMO functions - @LantaoJin what are your thoughts?

@salyh salyh moved this from Todo to In Progress in PPL Commands Sep 9, 2024
@lukasz-soszynski-eliatra
Copy link
Contributor

lukasz-soszynski-eliatra commented Sep 16, 2024

@YANG-DB
I have a question related to the edge case associated with ternary logic. What should be the outcome of invoking eval r = isempty(null)? According to the below definition

SELECT CASE  
       WHEN length(trim(column_name)) = 0 THEN true 
        ELSE false     
END AS is_empty

isempty(null) evaluates to false. Is this expected?

@lukasz-soszynski-eliatra
Copy link
Contributor

The initial implementation is ready for review. The version contains the unresolved problem described in the above comment. @YANG-DB , please let me know what you think.

@salyh
Copy link
Contributor

salyh commented Sep 24, 2024

can we close this as #676 is merged?

@salyh salyh moved this from In Progress to Done in PPL Commands Sep 24, 2024
@salyh salyh mentioned this issue Oct 7, 2024
@YANG-DB YANG-DB closed this as completed Oct 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
0.6 enhancement New feature or request Lang:PPL Pipe Processing Language support
Projects
Status: Done
Development

No branches or pull requests

4 participants