Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Qualification/Profiling Tool: Store spark runtime for different application type #1413

Closed
Tracked by #251
parthosa opened this issue Nov 7, 2024 · 2 comments · Fixed by #1414
Closed
Tracked by #251
Assignees
Labels
core_tools Scope the core module (scala)

Comments

@parthosa
Copy link
Collaborator

parthosa commented Nov 7, 2024

As we add support for various runtimes (e.g., Spark, Photon, and future support for Velox), we should store the Spark Runtime information for each application.

This will be consumed by the Python CLI and QualX to make decisions based on the runtime type.

@tgravescs
Copy link
Collaborator

It would be good to specify in the description which tool(s) this feature is being added to - profiling or qualification.

If an application uses multiple in the future - like Velox and Spark Rapids I guess we can make a VELOX_AND_SPARK_RAPIDS type indicator, otherwise it may have been good to make this a list. Its not totally clear to me what all this is used for to decide which is better.

Is there a qualx issue this can be linked to, to understand the usage?

@parthosa parthosa changed the title [FEA] Store spark runtime for different application type [FEA] Qualification/Profiling Tool: Store spark runtime for different application type Nov 18, 2024
@parthosa
Copy link
Collaborator Author

parthosa commented Nov 18, 2024

It would be good to specify in the description which tool(s) this feature is being added to - profiling or qualification.

Updated the description to include the Tool

If an application uses multiple in the future - like Velox and Spark Rapids I guess we can make a VELOX_AND_SPARK_RAPIDS

Yes, we will have to extend the enums if an app uses multiple runtime

Is there a qualx issue this can be linked to, to understand the usage?

QualX Usage:

Python CLI Usage

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core_tools Scope the core module (scala)
Projects
None yet
2 participants