Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance issues - iterating over 9k rules for every domain #301

Closed
gugu opened this issue Aug 11, 2022 · 4 comments · Fixed by #334
Closed

Performance issues - iterating over 9k rules for every domain #301

gugu opened this issue Aug 11, 2022 · 4 comments · Fixed by #334
Assignees
@lupomontero
Labels
enhancement

Comments

@gugu
Copy link
Contributor

gugu commented Aug 11, 2022

I've made some benchmarks and performance of psl is around 2k ops/sec:

psl#isValid x 2,310 ops/sec ±2.00% (92 runs sampled)
psl#parse x 2,428 ops/sec ±0.24% (97 runs sampled)
psl#parse invalid domain x 2,415 ops/sec ±1.56% (94 runs sampled)

by using maps it is possible to increase speed to 300k ops/sec:

psl#isValid x 349,961 ops/sec ±0.48% (95 runs sampled)
psl#parse x 346,816 ops/sec ±0.18% (99 runs sampled)
psl#parse invalid domain x 375,047 ops/sec ±0.16% (96 runs sampled)
@gugu gugu mentioned this issue Aug 11, 2022
Closed
@gboer gboer mentioned this issue Sep 7, 2022
Closed
@gugu
Copy link
Contributor Author

gugu commented Sep 29, 2022

@lupomontero sorry for mentioning you. Can you please review this PR? I think 300x speed boost can helpful change to this module

@billneff79
Copy link

We frequently are parsing upwards of 100k domains. We love the accuracy of psl, but the performance of psl is a real hindrance. It would be great if the PR that addresses this could get reviewed/merged

@Stanback
Copy link

Stanback commented Jun 3, 2023

Would love to see this get merged, it's insanely more performant

@lupomontero
Copy link
Owner

Hi there, many thanks for the proposal and apologies for the delay in dealing with this. I'm finally revamping this module and this is a very much needed improvement. I'll be reviewing this carefully tomorrow.

@lupomontero lupomontero self-assigned this Nov 22, 2024
@lupomontero lupomontero linked a pull request Nov 25, 2024 that will close this issue
Use Map for rule search and add benchmark #334
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lupomontero lupomontero

Labels
enhancement
Projects
None yet
Milestone
No milestone
4 participants
@gugu @lupomontero @Stanback @billneff79