Support InSubquery in PPL

[LantaoJin]

Description

InSubquery usage

• source = outer | where a in [ source = inner | fields b ]
• source = outer | where (a) in [ source = inner | fields b ]
• source = outer | where (a,b,c) in [ source = inner | fields d,e,f ]
• source = outer | where a not in [ source = inner | fields b ]
• source = outer | where (a) not in [ source = inner | fields b ]
• source = outer | where (a,b,c) not in [ source = inner | fields d,e,f ]
• source = outer | where a in [ source = inner1 | where b not in [ source = inner2 | fields c ] | fields b ] (nested)
• source = table1 | inner join left = l right = r on l.a = r.a AND r.a in [ source = inner | fields d ] | fields l.a, r.a, b, c (as join filter)

SQL Migration examples with IN-Subquery PPL:

1. tpch q4 (in-subquery with aggregation)

select
  o_orderpriority,
  count(*) as order_count
from
  orders
where
  o_orderdate >= date '1993-07-01'
  and o_orderdate < date '1993-07-01' + interval '3' month
  and o_orderkey in (
    select
      l_orderkey
    from
      lineitem
    where l_commitdate < l_receiptdate
  )
group by
  o_orderpriority
order by
  o_orderpriority

Rewritten by PPL InSubquery query:

source = orders
| where o_orderdate >= "1993-07-01" and o_orderdate < "1993-10-01" and o_orderkey IN
  [ source = lineitem
    | where l_commitdate < l_receiptdate
    | fields l_orderkey
  ]
| stats count(1) as order_count by o_orderpriority
| sort o_orderpriority
| fields o_orderpriority, order_count

2.tpch q20 (nested in-subquery)

select
  s_name,
  s_address
from
  supplier,
  nation
where
  s_suppkey in (
    select
      ps_suppkey
    from
      partsupp
    where
      ps_partkey in (
        select
          p_partkey
        from
          part
        where
          p_name like 'forest%'
      )
  )
  and s_nationkey = n_nationkey
  and n_name = 'CANADA'
order by
  s_name

Rewritten by PPL InSubquery query:

source = supplier
| where s_suppkey IN [
    source = partsupp
    | where ps_partkey IN [
        source = part
        | where like(p_name, "forest%")
        | fields p_partkey
      ]
    | fields ps_suppkey
  ]
| inner join left=l right=r on s_nationkey = n_nationkey and n_name = 'CANADA'
  nation
| sort s_name

Issues Resolved

Resolve #710 as a sub-task of #661

Check List

• Updated documentation (ppl-spark-integration/README.md)
• Implemented unit tests
• Implemented tests for combination with other commands
• Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[YANG-DB]

@LantaoJin can u plz resolve the conflict ?
thanks

[YANG-DB]

@LantaoJin
looking good - can u plz add some additional nested sub-query use cases in the examples ?
Could your also add a subQuery + stats - groupBy examples ?

In addition I think a dedicated inSubQuery doc with details about usage and detailed description of behaviour may be very useful - since it is a substantial capability which deserve a deep dive ...

[LantaoJin]

@LantaoJin looking good - can u plz add some additional nested sub-query use cases in the examples ? Could your also add a subQuery + stats - groupBy examples ?

Sure, I will add more cases.

In addition I think a dedicated inSubQuery doc with details about usage and detailed description of behaviour may be very useful - since it is a substantial capability which deserve a deep dive ...

I will add a dedicated subquery doc when the rest subquery issues are all completed.

[LantaoJin]

@YANG-DB more examples added in UT and README.md, so does in Description