Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spline doesn't support MaxCompute table storage? #820

Closed
XavierZYXue opened this issue Jul 29, 2024 · 1 comment · Fixed by #828
Closed

Spline doesn't support MaxCompute table storage? #820

XavierZYXue opened this issue Jul 29, 2024 · 1 comment · Fixed by #828

Comments

@XavierZYXue
Copy link
Contributor

when I try to use MaxCompute table as spark storage, the agent cannot work well.
I found that there is no relevant plugin of MaxCompute(Ali Cloud) in this repo, so it doesn't support right?
any plan for this or what?

@cerveada cerveada added this to Spline Jul 29, 2024
@github-project-automation github-project-automation bot moved this to New in Spline Jul 29, 2024
@wajda
Copy link
Contributor

wajda commented Jul 29, 2024

We never tried it on MaxCompute or any other Alibaba services, so I cannot tell anything on that matter.
There are two different APIs in Spark (called DataSource API V1 and V2 respectively) through which Spark interacts with 3rd-party data providers, and through which Spline can receive metadata and capture lineage. If MaxCompute connector uses V2 then Spline should in theory support it out of the box. If it uses V1 then a separate Spline agent plugin would likely be required to capture lineage from it. It needs more investigation. But as I mentioned in the other ticket, if you could contribute with code we would try to provide maximum support.

BerlinJava added a commit to BerlinJava/spline-spark-agent-real that referenced this issue Sep 2, 2024
@wajda wajda added this to the 2.2.0 milestone Sep 3, 2024
@wajda wajda moved this from New to Review / PR Created in Spline Sep 3, 2024
BerlinJava added a commit to BerlinJava/spline-spark-agent-real that referenced this issue Sep 5, 2024
BerlinJava added a commit to BerlinJava/spline-spark-agent-real that referenced this issue Sep 5, 2024
wajda pushed a commit that referenced this issue Sep 5, 2024
* odpstable-fix-#820

* odpstable-fix-#820 : adjust the cases order for odps table
@wajda wajda closed this as completed in #828 Sep 5, 2024
@github-project-automation github-project-automation bot moved this from Review / PR Created to Done in Spline Sep 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Done
Development

Successfully merging a pull request may close this issue.

2 participants