Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add INPUT command to load a CSV file #678

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Add INPUT command to load a CSV file #678

wants to merge 1 commit into from

Conversation

LantaoJin
Copy link
Member

@LantaoJin LantaoJin commented Sep 18, 2024 •
edited
Loading

Description

This PR is for the syntax option A described in #638
The PR for option B is #678.

Syntax of new command input (option A):

input <fileUrl>

Usage:

input "s3://bucket_name/folder1/folder2/flights.csv"

Issues Resolved

Resolves #638

Check List

  • Updated documentation (ppl-spark-integration/README.md)
  • Implemented unit tests
  • Implemented tests for combination with other commands
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@LantaoJin LantaoJin mentioned this pull request Sep 18, 2024
Draft
4 tasks
@LantaoJin LantaoJin marked this pull request as ready for review September 18, 2024 15:56
@LantaoJin
Copy link
Member Author

@YANG-DB Which one do you think is better? #638 vs #678
The CI failed cause by

[PATH_NOT_FOUND] Path does not exist: file:/home/runner/work/opensearch-spark/opensearch-spark/integ-test/integ-test/src/integration/resources/opensearch_dashboards_sample_data_flights.csv.

Do you know why the resources path contains duplicated folders in our github build node? The actual csv file located in $PROJECT/integ-test/src/integration/resources/opensearch_dashboards_sample_data_flights.csv

@YANG-DB
Copy link
Member

YANG-DB commented Sep 19, 2024 •
edited
Loading

Let's keep this - option A #638 version as the one we go with

@YANG-DB YANG-DB added the Lang:PPL Pipe Processing Language support label Sep 19, 2024
@YANG-DB
Copy link
Member

YANG-DB commented Sep 19, 2024

@YANG-DB Which one do you think is better? #638 vs #678 The CI failed cause by

[PATH_NOT_FOUND] Path does not exist: file:/home/runner/work/opensearch-spark/opensearch-spark/integ-test/integ-test/src/integration/resources/opensearch_dashboards_sample_data_flights.csv.

Do you know why the resources path contains duplicated folders in our github build node? The actual csv file located in $PROJECT/integ-test/src/integration/resources/opensearch_dashboards_sample_data_flights.csv

Let me try to investigate this today...

@YANG-DB
Copy link
Member

YANG-DB commented Sep 19, 2024

@LantaoJin can u also plz revolve latest merge conflicts ?
thanks

penghuo
penghuo reviewed Sep 20, 2024
View reviewed changes
ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java
Comment on lines +96 to +100
// @Override
// public UnresolvedPlan visitSearchFilterFrom(OpenSearchPPLParser.SearchFilterFromContext ctx) {
// return new Filter(internalVisitExpression(ctx.logicalExpression()))
// .attach(visit(ctx.fromClause()));
// }
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

delete it.

ppl-spark-integration/src/main/java/org/opensearch/sql/ppl/parser/AstBuilder.java
@@ -427,4 +469,8 @@ private String getTextInQuery(ParserRuleContext ctx) {
Token stop = ctx.getStop();
return query.substring(start.getStartIndex(), stop.getStopIndex() + 1);
}

private boolean isTesting() {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

add comments why include testing mode.

@YANG-DB YANG-DB added the 0.6 label Sep 25, 2024
@LantaoJin
Copy link
Member Author

@LantaoJin can u also plz revolve latest merge conflicts ? thanks

@YANG-DB I need some time to double confirm the file loading only be executed at once. Let me see how to get the metrics from Spark side in IT suite. Will rebase the code when my checking done.

@YANG-DB
Copy link
Member

YANG-DB commented Oct 1, 2024

@LantaoJin can u plz change to PR into draft mode ?

@LantaoJin LantaoJin marked this pull request as draft October 2, 2024 01:35
@YANG-DB
Copy link
Member

YANG-DB commented Oct 7, 2024

@LantaoJin
please add the needed documentation in the following locations:

@LantaoJin LantaoJin removed the 0.6 label Nov 19, 2024
@opensearch-trigger-bot
Copy link

This PR is stalled because it has been open for 30 days with no activity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@penghuo penghuo penghuo left review comments

@dai-chen dai-chen Awaiting requested review from dai-chen dai-chen is a code owner

@rupal-bq rupal-bq Awaiting requested review from rupal-bq

@vamsimanohar vamsimanohar Awaiting requested review from vamsimanohar

@seankao-az seankao-az Awaiting requested review from seankao-az seankao-az is a code owner

@anirudha anirudha Awaiting requested review from anirudha anirudha is a code owner

@kaituo kaituo Awaiting requested review from kaituo kaituo is a code owner

@YANG-DB YANG-DB Awaiting requested review from YANG-DB YANG-DB is a code owner

@mengweieric mengweieric Awaiting requested review from mengweieric mengweieric will be requested when the pull request is marked ready for review mengweieric is a code owner

@noCharger noCharger Awaiting requested review from noCharger noCharger will be requested when the pull request is marked ready for review noCharger is a code owner

@ykmr1224 ykmr1224 Awaiting requested review from ykmr1224 ykmr1224 will be requested when the pull request is marked ready for review ykmr1224 is a code owner

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
Lang:PPL Pipe Processing Language support
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[FEATURE] Support load CSV in PPL (inputlookup or search)
3 participants
@LantaoJin @YANG-DB @penghuo