Skip to content
New issue

Have a question about this project? # for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “#”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? # to your account

Add Spark properties table to profiling tool output #4781

Merged
merged 8 commits into from
Feb 15, 2022

Conversation

tgravescs
Copy link
Collaborator

fixes #4753

The profiling tool now prints all the Spark properties into a separate table. It still prints just the rapids related ones into one table so you can easily see those and then all the other Spark properties into a separate table. The spark properties are very useful for comparing applications.

This also has a random change to change a log info message when we see events we don't handle to a log debug message.

examples:

rapids properties no change:

Spark Rapids parameters set explicitly:
+-----------------------------------+----------+----------+
|propertyName                       |appIndex_2|appIndex_1|
+-----------------------------------+----------+----------+
|spark.rapids.memory.pinnedPool.size|2G        |2G        |
|spark.rapids.sql.concurrentGpuTasks|2         |2         |
|spark.rapids.sql.explain           |NOT_ON_GPU|NOT_ON_GPU|
+-----------------------------------+----------+----------+

Added spark properties:

Spark Properties:
+---------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
|propertyName                     |appIndex_2                                                                                |appIndex_1                                                                                |
+---------------------------------+------------------------------------------------------------------------------------------+------------------------------------------------------------------------------------------+
|spark.app.id                     |local-1622821994212                                                                       |local-1622814619968                                                                       |
|spark.app.name                   |Spark shell                                                                               |Spark shell                                                                               |
....

@tgravescs tgravescs added this to the Feb 14 - Feb 25 milestone Feb 14, 2022
@tgravescs tgravescs self-assigned this Feb 14, 2022
@nartal1
Copy link
Collaborator

nartal1 commented Feb 14, 2022

LGTM. Do we need to update in spark-profiling-tool.md that spark properties are also covered along with rapids properties?

@tgravescs
Copy link
Collaborator Author

yes, thanks for pointing that out, I added it to list. I didn't put example as it seems like its pretty self explanatory.

@tgravescs
Copy link
Collaborator Author

build

@tgravescs tgravescs merged commit 6a096e5 into NVIDIA:branch-22.04 Feb 15, 2022
@tgravescs tgravescs deleted the sparkpropsProfile branch February 15, 2022 13:45
# for free to join this conversation on GitHub. Already have an account? # to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[FEA] Profiling tool should report all spark configs
2 participants