Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spilled data size in CLI and WebUI #426

Open
wants to merge 4 commits into
base: cw/spill-space-manager/2
Choose a base branch
from

Conversation

ilfrin
Copy link

@ilfrin ilfrin commented Nov 17, 2016

@ilfrin ilfrin changed the title Explain analyze spill Spilled data size in CLI Nov 17, 2016
@ilfrin
Copy link
Author

ilfrin commented Nov 17, 2016

@pnowojski I think we should remove the Driver and Pipeline level spill stats collecting and maintaining. They're not used anywhere.

@ilfrin ilfrin changed the title Spilled data size in CLI Spilled data size in CLI and WebUI Nov 17, 2016
Copy link

@pnowojski pnowojski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally looks good, however is there no tests for explain analyze output?

@@ -323,6 +331,11 @@ private static String formatFragment(Metadata metadata, Session session, PlanFra
return builder.toString();
}

private static boolean isNonZero(DataSize dataSize)
{
return dataSize != null && dataSize.getValue() != 0;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why it can be null?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it can't, it was a reflex to implement a static helper method as null safe, I can change that if you want

Wojciech Biela added 3 commits November 17, 2016 15:49
Add tracking of spilled data size on various levels (from operator to query) and expose it in JSON.
Data size added to EXPLAIN ANALYZE output, at Fragment and Operator
levels. This information is added to the "Cost" line and displayed only
if the value is non-zero.
When CLI is started with --debug then spilled data size for the entire
query is being displayed (both in the running total, and the final summary).
The information is added in a separate line below "Parallelism".
@kbajda
Copy link

kbajda commented Nov 17, 2016

@ilfrin for completeness, let's print total data spilled in the Presto UI on the query details page under Resource Utilization Summary (next to the CPU & Memory usage)

@sopel39
Copy link

sopel39 commented Nov 18, 2016

lgtm

Added spilled data size to the Query Details page in the Resource
Utilization Summary section and to the Live Plan page in the Web UI just below
the "Splits" line (per stage).
@ilfrin
Copy link
Author

ilfrin commented Nov 18, 2016

@KBP-TDC done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants