-
Notifications
You must be signed in to change notification settings - Fork 304
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HPCC-33022 Fix pathological dali load time #19350
HPCC-33022 Fix pathological dali load time #19350
Conversation
Jira Issue: https://hpccsystems.atlassian.net//browse/HPCC-33022 Jirabot Action Result: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@jakesmith one question, otherwise approved
system/jlib/jsecrets.cpp
Outdated
@@ -31,9 +31,6 @@ | |||
#if defined(__clang__) || defined(__GNUC__) | |||
#pragma GCC diagnostic push | |||
#pragma GCC diagnostic ignored "-Wformat-nonliteral" | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
was this deletion on purpose?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
yep, I'll reverse
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ghalliday - I've reverted that change (squashed in)
JPTree (used by Dali SDS) could perform horrendously slowly when dealing with a large list of child nodes sharing a common prefix. This was due to how the hashing function was used (no seed number provided). With many entries having the same prefix, it caused a high number of hash collisions, resulting in excessive scanning when adding new members. This particularly affected the /GeneratedDlls section of Dali. After fixing the issue, the time to load this section dropped from over 4700 seconds to 17 seconds. Also add a 'loadxml' utility to daliadmin, that loads and times loading an xml file, and traces the amount of memory it consumes. Signed-off-by: Jake Smith <[email protected]>
b9c6195
to
bb1a9e1
Compare
959f55e
into
hpcc-systems:candidate-9.2.x
Jirabot Action Result: |
JPTree (used by Dali SDS) could perform horrendously slowly when dealing with a large list of child nodes sharing a common prefix. This was due to how the hashing function was used (no seed number provided). With many entries having the same prefix, it caused a high number of hash collisions, resulting in excessive scanning when adding new members.
This particularly affected the /GeneratedDlls section of Dali. After fixing the issue, the time to load this section dropped from over 4700 seconds to 17 seconds.
Also add a 'loadxml' utility to daliadmin, that loads and times loading an xml file, and traces the amount of memory it consumes.
Type of change:
Checklist:
Smoketest:
Testing: