-
Notifications
You must be signed in to change notification settings - Fork 40
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add sparkRuntime property to capture runtime type in application_information #1414
Conversation
Signed-off-by: Partho Sarthi <[email protected]>
core/src/test/scala/com/nvidia/spark/rapids/tool/profiling/ApplicationInfoSuite.scala
Outdated
Show resolved
Hide resolved
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @parthosa! A minor nit.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @parthosa !
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @parthosa
Only need more comments/description to the new class/method
core/src/main/scala/org/apache/spark/sql/rapids/tool/util/CacheablePropsHandler.scala
Outdated
Show resolved
Hide resolved
core/src/main/scala/org/apache/spark/sql/rapids/tool/AppBase.scala
Outdated
Show resolved
Hide resolved
Signed-off-by: Partho Sarthi <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @parthosa
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
thanks @parthosa! LGTM.
Fixes #1413
This PR adds a new
getSparkRuntime
method to capture the Spark Runtime type(SPARK, PHOTON, SPARK_RAPIDS)
and store this inapplication_information.csv
Changes
Profiling Enhancements:
sparkRuntime
property toAppInfoProfileResults
to capture the runtime environment and updated theoutputHeaders
andconvertToSeq
methods to include this new property. [1] [2]AppInformationViewTrait
to map the newsparkRuntime
property when creatingAppInfoProfileResults
instances.Runtime Handling:
SparkRuntime
enumeration to represent different Spark runtimes (SPARK, PHOTON, SPARK_RAPIDS).getSparkRuntime
method toCacheablePropsHandler
Testing:
ApplicationInfoSuite
to validate the spark runtime value for different event logs.Output
File:
application_information.csv
SPARK Runtime:
SPARK_RAPIDS Runtime:
PHOTON Runtime:
cc: @leewyang