Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Perf/benchmarks #347

Open
wants to merge 10 commits into
base: master
Choose a base branch
from
Open

Perf/benchmarks #347

wants to merge 10 commits into from

Conversation

saurabhnanda
Copy link
Contributor

@saurabhnanda saurabhnanda commented Nov 8, 2017 •
edited
Loading

Fixes #340

@saurabhnanda
Copy link
Contributor Author

Creating the benchmark if you want to look at WIP code. Not ready for a merge yet.

@saurabhnanda
Initial commit for benchmarks
5b70bca 
- Extracted out code for creating a connection to the test DB
- Added ‘benchmark’ section/target to cabal file
@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw are you alright with a dependency on lens?

@tomjaguarpaw
Copy link
Owner

As long as the opaleye package itself doesn't depend on lens that's fine by me.

@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw do benchmarks like this seem helpful to you? Should I continue down this path?

  • narrowTable is a table with 5 cols (one of them is an id field). Plan to duplicate the entire benchmark for a wide table with 30 cols (with all kinds of data-types)
  • narrowTable related benchmarks are preparing the "flattest" queries.
  • narrowQueryNested related benchmarks are nested queries where each level of nesting adds an additional WHERE clause -- a typical use-case while composing queries. You can see the increase in execution time vis-a-vis narrowTable
  • narrowQueryLeftJoin is the simplest left-join of two "narrow tables". The query generation time is ~5x that of narrowTable.
benchmarking pepareQuery/narrowTable/findByPkSelect1
time                 53.17 μs   (52.19 μs .. 54.62 μs)
                     0.996 R²   (0.994 R² .. 0.999 R²)
mean                 52.82 μs   (52.28 μs .. 53.80 μs)
std dev              2.491 μs   (1.741 μs .. 3.661 μs)
variance introduced by outliers: 52% (severely inflated)

benchmarking pepareQuery/narrowTable/findByPkSelect3
time                 66.30 μs   (65.27 μs .. 67.78 μs)
                     0.998 R²   (0.996 R² .. 1.000 R²)
mean                 66.01 μs   (65.53 μs .. 66.93 μs)
std dev              1.975 μs   (1.306 μs .. 3.075 μs)
variance introduced by outliers: 29% (moderately inflated)

benchmarking pepareQuery/narrowTable/findByPkCompleteRow
time                 77.74 μs   (77.22 μs .. 78.27 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 78.40 μs   (77.44 μs .. 80.40 μs)
std dev              4.661 μs   (2.502 μs .. 7.427 μs)
variance introduced by outliers: 62% (severely inflated)

benchmarking pepareQuery/narrowQueryNested/findByPkSelect1
time                 73.78 μs   (72.72 μs .. 75.01 μs)
                     0.998 R²   (0.997 R² .. 0.999 R²)
mean                 74.17 μs   (73.14 μs .. 75.58 μs)
std dev              3.843 μs   (2.746 μs .. 4.879 μs)
variance introduced by outliers: 55% (severely inflated)

benchmarking pepareQuery/narrowQueryNested/findByPkSelect3
time                 87.53 μs   (86.63 μs .. 88.50 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 87.20 μs   (86.52 μs .. 88.08 μs)
std dev              2.557 μs   (2.033 μs .. 3.666 μs)
variance introduced by outliers: 27% (moderately inflated)

benchmarking pepareQuery/narrowQueryNested/findByPkCompleteRow
time                 102.2 μs   (99.19 μs .. 106.0 μs)
                     0.991 R²   (0.987 R² .. 0.997 R²)
mean                 100.9 μs   (99.08 μs .. 103.7 μs)
std dev              7.395 μs   (5.277 μs .. 9.609 μs)
variance introduced by outliers: 70% (severely inflated)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkSelect1
time                 243.8 μs   (238.8 μs .. 251.6 μs)
                     0.995 R²   (0.990 R² .. 0.999 R²)
mean                 240.4 μs   (237.8 μs .. 244.8 μs)
std dev              11.56 μs   (6.348 μs .. 17.96 μs)
variance introduced by outliers: 46% (moderately inflated)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkSelect3
time                 261.0 μs   (251.4 μs .. 274.6 μs)
                     0.992 R²   (0.985 R² .. 0.999 R²)
mean                 255.3 μs   (252.8 μs .. 260.2 μs)
std dev              11.50 μs   (6.893 μs .. 21.25 μs)
variance introduced by outliers: 42% (moderately inflated)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkCompleteRow
time                 307.5 μs   (298.9 μs .. 319.3 μs)
                     0.996 R²   (0.991 R² .. 1.000 R²)
mean                 300.3 μs   (298.1 μs .. 305.8 μs)
std dev              10.26 μs   (5.696 μs .. 19.15 μs)
variance introduced by outliers: 29% (moderately inflated)

@tomjaguarpaw
Copy link
Owner

Travis tests all the way back to 7.6. You'll need to import pure from Control.Applicative.

@tomjaguarpaw
Copy link
Owner

You'll also have to not use TypeApplications. I really don't like them anyway. Could you use a Proxy instead?

@tomjaguarpaw
Copy link
Owner

In fact I don't even see that you used TypeApplications in that module.

@saurabhnanda
Copy link
Contributor Author

Had tried TypeApplications, but found it too confusing. Had already moved to Proxy. I'm getting strange benchmarking results. While actually running the queries, the difference between opaleye generated queries and handwritten queries wrt narrow tables is barely perceptible in the numbers.

I'm going to try with wide tables, to be sure. Do you have any hypothesis why the benchmark numbers are wildly different from what we were observing in #340 ?

If we aren't doing anything with the query results (like, printing them or writing them to disk), will all the query parsers (pg -> haskell conversion) get executed?

benchmarking runQuery/narrowTable/findBySelect1/opaleye
time                 172.2 μs   (167.9 μs .. 178.0 μs)
                     0.991 R²   (0.980 R² .. 1.000 R²)
mean                 171.5 μs   (168.9 μs .. 176.9 μs)
std dev              12.15 μs   (4.428 μs .. 22.91 μs)
variance introduced by outliers: 67% (severely inflated)

benchmarking runQuery/narrowTable/findBySelect1/handwritten
time                 167.0 μs   (165.6 μs .. 168.8 μs)
                     0.995 R²   (0.991 R² .. 0.997 R²)
mean                 184.9 μs   (178.1 μs .. 196.7 μs)
std dev              30.45 μs   (20.94 μs .. 42.97 μs)
variance introduced by outliers: 92% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/opaleye
time                 207.3 μs   (194.3 μs .. 220.9 μs)
                     0.968 R²   (0.954 R² .. 0.985 R²)
mean                 208.5 μs   (199.9 μs .. 218.7 μs)
std dev              32.12 μs   (26.96 μs .. 38.45 μs)
variance introduced by outliers: 91% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/handwritten
time                 192.7 μs   (190.4 μs .. 195.4 μs)
                     0.998 R²   (0.996 R² .. 0.999 R²)
mean                 194.7 μs   (191.6 μs .. 201.0 μs)
std dev              14.65 μs   (8.732 μs .. 24.41 μs)
variance introduced by outliers: 69% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/opaleye
time                 408.0 μs   (394.3 μs .. 421.9 μs)
                     0.990 R²   (0.985 R² .. 0.995 R²)
mean                 414.0 μs   (403.3 μs .. 429.8 μs)
std dev              44.79 μs   (30.82 μs .. 61.63 μs)
variance introduced by outliers: 80% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/handwritten
time                 404.2 μs   (401.5 μs .. 407.1 μs)
                     0.997 R²   (0.992 R² .. 1.000 R²)
mean                 407.7 μs   (401.7 μs .. 422.2 μs)
std dev              28.24 μs   (11.89 μs .. 50.90 μs)
variance introduced by outliers: 61% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/opaleye
time                 525.1 μs   (490.9 μs .. 567.4 μs)
                     0.981 R²   (0.966 R² .. 0.997 R²)
mean                 502.1 μs   (494.6 μs .. 517.3 μs)
std dev              34.93 μs   (20.83 μs .. 61.86 μs)
variance introduced by outliers: 60% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/handwritten
time                 495.9 μs   (480.8 μs .. 521.7 μs)
                     0.971 R²   (0.951 R² .. 0.988 R²)
mean                 586.3 μs   (558.5 μs .. 624.6 μs)
std dev              110.1 μs   (76.64 μs .. 156.3 μs)
variance introduced by outliers: 92% (severely inflated)

@saurabhnanda
Copy link
Contributor Author

Let it be I found the error. I was running the Opaleye generated query instead of the handwritten query 🤦‍♂️

Here are the new benchmark results which clearly demonstrate that queries generated via Opaleye get progressively slower as the level of nesting increases:

benchmarking runQuery/narrowTable/findBySelect1/opaleye
time                 208.3 μs   (183.2 μs .. 232.4 μs)
                     0.949 R²   (0.924 R² .. 0.991 R²)
mean                 186.4 μs   (180.4 μs .. 198.2 μs)
std dev              25.51 μs   (15.13 μs .. 42.05 μs)
variance introduced by outliers: 89% (severely inflated)

benchmarking runQuery/narrowTable/findBySelect1/handwritten
time                 147.3 μs   (143.8 μs .. 152.4 μs)
                     0.992 R²   (0.986 R² .. 0.999 R²)
mean                 147.9 μs   (145.5 μs .. 151.8 μs)
std dev              9.810 μs   (6.200 μs .. 14.27 μs)
variance introduced by outliers: 65% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/opaleye
time                 202.7 μs   (195.3 μs .. 211.6 μs)
                     0.988 R²   (0.978 R² .. 0.998 R²)
mean                 201.2 μs   (197.4 μs .. 207.6 μs)
std dev              15.81 μs   (11.16 μs .. 24.05 μs)
variance introduced by outliers: 71% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/handwritten
time                 160.7 μs   (156.5 μs .. 167.5 μs)
                     0.988 R²   (0.978 R² .. 0.997 R²)
mean                 161.8 μs   (158.0 μs .. 167.6 μs)
std dev              15.64 μs   (9.625 μs .. 22.76 μs)
variance introduced by outliers: 79% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/opaleye
time                 401.5 μs   (392.5 μs .. 413.9 μs)
                     0.991 R²   (0.978 R² .. 0.999 R²)
mean                 396.1 μs   (389.9 μs .. 409.2 μs)
std dev              29.25 μs   (15.68 μs .. 52.80 μs)
variance introduced by outliers: 65% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/handwritten
time                 187.1 μs   (181.5 μs .. 195.1 μs)
                     0.987 R²   (0.976 R² .. 0.996 R²)
mean                 183.7 μs   (178.9 μs .. 191.4 μs)
std dev              18.56 μs   (13.05 μs .. 26.65 μs)
variance introduced by outliers: 80% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/opaleye
time                 513.0 μs   (498.8 μs .. 532.3 μs)
                     0.988 R²   (0.976 R² .. 0.998 R²)
mean                 516.2 μs   (504.4 μs .. 537.4 μs)
std dev              53.55 μs   (35.87 μs .. 77.10 μs)
variance introduced by outliers: 78% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/handwritten
time                 251.1 μs   (243.6 μs .. 264.2 μs)
                     0.980 R²   (0.967 R² .. 0.994 R²)
mean                 253.6 μs   (246.8 μs .. 263.6 μs)
std dev              28.15 μs   (18.92 μs .. 37.20 μs)
variance introduced by outliers: 83% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin3/CompleteRow/opaleye
time                 1.040 ms   (1.020 ms .. 1.069 ms)
                     0.990 R²   (0.979 R² .. 0.997 R²)
mean                 1.065 ms   (1.042 ms .. 1.096 ms)
std dev              88.44 μs   (62.16 μs .. 123.3 μs)
variance introduced by outliers: 64% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin3/CompleteRow/handwritten
time                 317.5 μs   (310.6 μs .. 328.0 μs)
                     0.984 R²   (0.964 R² .. 0.998 R²)
mean                 323.2 μs   (314.0 μs .. 338.0 μs)
std dev              38.01 μs   (21.58 μs .. 57.37 μs)
variance introduced by outliers: 83% (severely inflated)

@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw should I complete this for all use-cases, eg aggregates & limit/offst AND wide-tables, or is this good enough?

@tomjaguarpaw
Copy link
Owner

It would be best if you can complete it in for a wide range of use-cases because then we know any improvements I make are not making anything worse!

@saurabhnanda
Copy link
Contributor Author

Did you look at the prepareQuery numbers above? Do you find them interesting enough to keep in the benchmark?

@tomjaguarpaw
Copy link
Owner

Which numbers are the ones for prepareQuery?

@saurabhnanda
Copy link
Contributor Author

Which numbers are the ones for prepareQuery?

#347 (comment)

@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw you can take a look at the benchmarks now. The code is not pretty, but it should get the job done. Here are the benchmarks numbers on my machine:

benchmarking runQuery/narrowTable/findBySelect1/opaleye
time                 344.2 μs   (297.0 μs .. 385.6 μs)
                     0.925 R²   (0.901 R² .. 0.965 R²)
mean                 318.9 μs   (303.6 μs .. 340.6 μs)
std dev              64.10 μs   (53.56 μs .. 80.23 μs)
variance introduced by outliers: 94% (severely inflated)

benchmarking runQuery/narrowTable/findBySelect1/handwritten
time                 177.3 μs   (165.7 μs .. 186.7 μs)
                     0.970 R²   (0.959 R² .. 0.980 R²)
mean                 207.8 μs   (195.3 μs .. 219.8 μs)
std dev              42.33 μs   (35.84 μs .. 49.39 μs)
variance introduced by outliers: 95% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/opaleye
time                 374.1 μs   (354.7 μs .. 397.2 μs)
                     0.968 R²   (0.949 R² .. 0.984 R²)
mean                 409.8 μs   (381.5 μs .. 452.4 μs)
std dev              123.4 μs   (81.01 μs .. 177.4 μs)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/handwritten
time                 243.4 μs   (220.3 μs .. 263.3 μs)
                     0.943 R²   (0.917 R² .. 0.965 R²)
mean                 234.9 μs   (216.8 μs .. 252.6 μs)
std dev              55.14 μs   (46.69 μs .. 64.91 μs)
variance introduced by outliers: 96% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/opaleye
time                 516.4 μs   (474.8 μs .. 581.0 μs)
                     0.919 R²   (0.892 R² .. 0.953 R²)
mean                 589.5 μs   (557.4 μs .. 620.8 μs)
std dev              106.6 μs   (91.53 μs .. 132.4 μs)
variance introduced by outliers: 91% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/handwritten
time                 232.0 μs   (204.8 μs .. 257.3 μs)
                     0.950 R²   (0.933 R² .. 0.987 R²)
mean                 222.9 μs   (213.5 μs .. 235.5 μs)
std dev              37.69 μs   (30.80 μs .. 46.78 μs)
variance introduced by outliers: 92% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/opaleye
time                 721.9 μs   (585.5 μs .. 864.4 μs)
                     0.870 R²   (0.828 R² .. 0.970 R²)
mean                 653.9 μs   (612.8 μs .. 720.5 μs)
std dev              164.5 μs   (123.1 μs .. 231.9 μs)
variance introduced by outliers: 96% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/handwritten
time                 283.8 μs   (271.6 μs .. 299.7 μs)
                     0.979 R²   (0.969 R² .. 0.990 R²)
mean                 299.2 μs   (289.8 μs .. 311.9 μs)
std dev              37.86 μs   (31.85 μs .. 47.71 μs)
variance introduced by outliers: 86% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin3/CompleteRow/opaleye
time                 1.655 ms   (1.337 ms .. 1.964 ms)
                     0.883 R²   (0.845 R² .. 0.953 R²)
mean                 1.263 ms   (1.191 ms .. 1.382 ms)
std dev              300.1 μs   (196.0 μs .. 456.6 μs)
variance introduced by outliers: 93% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin3/CompleteRow/handwritten
time                 342.2 μs   (331.4 μs .. 357.8 μs)
                     0.988 R²   (0.982 R² .. 0.995 R²)
mean                 350.3 μs   (341.5 μs .. 361.6 μs)
std dev              33.76 μs   (26.66 μs .. 48.97 μs)
variance introduced by outliers: 77% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoinF/CompleteRow/opaleye
time                 2.021 ms   (1.819 ms .. 2.200 ms)
                     0.939 R²   (0.892 R² .. 0.974 R²)
mean                 1.852 ms   (1.780 ms .. 1.954 ms)
std dev              300.7 μs   (216.5 μs .. 528.3 μs)
variance introduced by outliers: 85% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoinF/CompleteRow/handwritten
time                 361.2 μs   (349.4 μs .. 375.7 μs)
                     0.985 R²   (0.977 R² .. 0.992 R²)
mean                 354.6 μs   (341.1 μs .. 370.0 μs)
std dev              45.55 μs   (36.62 μs .. 58.64 μs)
variance introduced by outliers: 85% (severely inflated)

benchmarking runQuery/narrowQuery/Limit-Offset/CompleteRow/opaleye
time                 823.9 μs   (793.1 μs .. 861.6 μs)
                     0.981 R²   (0.970 R² .. 0.989 R²)
mean                 854.0 μs   (825.8 μs .. 885.3 μs)
std dev              98.38 μs   (84.09 μs .. 117.9 μs)
variance introduced by outliers: 79% (severely inflated)

benchmarking runQuery/narrowQuery/Limit-Offset/CompleteRow/handwritten
time                 365.1 μs   (353.3 μs .. 380.7 μs)
                     0.978 R²   (0.960 R² .. 0.988 R²)
mean                 401.6 μs   (380.8 μs .. 427.7 μs)
std dev              82.92 μs   (63.52 μs .. 106.0 μs)
variance introduced by outliers: 94% (severely inflated)

benchmarking runQuery/narrowQuery/groupBy/CompleteRow/opaleye
time                 1.485 ms   (1.362 ms .. 1.604 ms)
                     0.940 R²   (0.905 R² .. 0.966 R²)
mean                 2.276 ms   (1.920 ms .. 3.382 ms)
std dev              2.052 ms   (820.7 μs .. 4.002 ms)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowQuery/groupBy/CompleteRow/handwritten
time                 1.040 ms   (1.007 ms .. 1.071 ms)
                     0.989 R²   (0.982 R² .. 0.995 R²)
mean                 1.040 ms   (1.007 ms .. 1.079 ms)
std dev              123.9 μs   (91.98 μs .. 174.4 μs)
variance introduced by outliers: 79% (severely inflated)

@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw can you help me fix the travis build?

@tomjaguarpaw
Copy link
Owner

Oh I see. I searched for "prepareQuery" but they're called "pepareQuery" :)

@saurabhnanda
Copy link
Contributor Author

Btw, encountered #348 as well

@tomjaguarpaw
Copy link
Owner

The build fails to install interpolatedstring-perl6 which is rather odd. I've kicked off the tests again. Maybe it will work this time round. The tests are often flaky.

@tomjaguarpaw
Copy link
Owner

Regarding prepareQuery the worst result is

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkCompleteRow
time                 307.5 μs   (298.9 μs .. 319.3 μs)
                     0.996 R²   (0.991 R² .. 1.000 R²)
mean                 300.3 μs   (298.1 μs .. 305.8 μs)
std dev              10.26 μs   (5.696 μs .. 19.15 μs)
variance introduced by outliers: 29% (moderately inflated)

It takes 300.3 μs to prepare the query and 653.9 μs to execute it, so I'd say it's worth keeping those benchmarks, wouldn't you?

@saurabhnanda
Copy link
Contributor Author

The build fails to install interpolatedstring-perl6 which is rather odd. I've kicked off the tests again. Maybe it will work this time round. The tests are often flaky.

Would you be open to moving to CircleCi. We've recently spent a lot of time in making bullet-proof CircleCI builds for Haskell.

@saurabhnanda
Copy link
Contributor Author

It takes 300.3 μs to prepare the query and 653.9 μs to execute it, so I'd say it's worth keeping those benchmarks, wouldn't you?

653.9 μs includes the query preparation time.

@tomjaguarpaw
Copy link
Owner

Would you be open to moving to CircleCi. We've recently spent a lot of time in making bullet-proof CircleCI builds for Haskell.

I'm definitely open to it. I piggy back on Neil Mitchell's neil package which provides a nice bunch of stuff for CI on Travis. Maybe it works on CircleCI too, or if you have some other cool tools to use there I'd be happy to give it a try.

@tomjaguarpaw
Copy link
Owner

It takes 300.3 μs to prepare the query and 653.9 μs to execute it, so I'd say it's worth keeping those benchmarks, wouldn't you?

653.9 μs includes the query preparation time.

So it's 300 μs to prepare and 353.6 μs to execute? Preparation still seems quite slow ... I think it's worth keeping both.

@tomjaguarpaw
Copy link
Owner

I really don't understand what's gone wrong with interpolatedstring-perl6 but it works on 8.0 and 8.2, which is a good first start.

@saurabhnanda
Copy link
Contributor Author

What's your dev machine config? What do these numbers look like on your machine?

@tomjaguarpaw
Copy link
Owner

I'm not at my dev machine but will try it when I am. Thanks!

@tomjaguarpaw
Copy link
Owner

I have an underpowered laptop but I'm sure the relative stats are the important thing :)

@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw I've added the prepareQueryBenchmarks for a few cases as well. Let me know if you need anything else. You can run these tests interactively as well, eg:

$ stack ghci opaleye:benchmark

GHCi> prepareQueryBenchmarks
benchmarking pepareQuery/narrowTable/findByPkSelect1
time                 87.67 μs   (86.69 μs .. 88.76 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 87.19 μs   (86.47 μs .. 88.28 μs)
std dev              3.006 μs   (1.916 μs .. 4.107 μs)
variance introduced by outliers: 34% (moderately inflated)

benchmarking pepareQuery/narrowTable/findByPkCompleteRow
time                 117.4 μs   (116.0 μs .. 119.2 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 115.7 μs   (115.2 μs .. 116.5 μs)
std dev              2.127 μs   (1.388 μs .. 3.470 μs)
variance introduced by outliers: 12% (moderately inflated)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkSelect1
time                 294.8 μs   (292.3 μs .. 297.8 μs)
                     0.998 R²   (0.996 R² .. 0.999 R²)
mean                 302.7 μs   (297.5 μs .. 310.6 μs)
std dev              22.69 μs   (16.67 μs .. 32.11 μs)
variance introduced by outliers: 67% (severely inflated)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkCompleteRow
time                 371.8 μs   (364.0 μs .. 386.8 μs)
                     0.996 R²   (0.990 R² .. 1.000 R²)
mean                 365.3 μs   (363.0 μs .. 372.6 μs)
std dev              12.15 μs   (5.050 μs .. 25.30 μs)
variance introduced by outliers: 27% (moderately inflated)

benchmarking pepareQuery/narrowQueryLeftJoinF/Select1
time                 342.1 μs   (338.1 μs .. 347.0 μs)
                     0.998 R²   (0.995 R² .. 0.999 R²)
mean                 339.3 μs   (336.0 μs .. 344.5 μs)
std dev              13.42 μs   (9.089 μs .. 20.79 μs)
variance introduced by outliers: 34% (moderately inflated)

benchmarking pepareQuery/narrowQueryLeftJoinF/CompleteRow
time                 488.9 μs   (477.5 μs .. 505.2 μs)
                     0.968 R²   (0.937 R² .. 0.988 R²)
mean                 597.8 μs   (565.5 μs .. 662.3 μs)
std dev              150.3 μs   (111.7 μs .. 223.9 μs)
variance introduced by outliers: 96% (severely inflated)

@tomjaguarpaw
Copy link
Owner

Thanks very much for doing this. I hope to look at it this weekend.

@tomjaguarpaw
Copy link
Owner

I haven't forgotten about this but will get round to it when I have some time.

@saurabhnanda
Copy link
Contributor Author

Thanks for the update 😄

@saurabhnanda
Copy link
Contributor Author

@tomjaguarpaw ping.

@tomjaguarpaw
Copy link
Owner

I still haven't forgotten but sorry I can't be specific about when I'll get time for this.

@tomjaguarpaw
Copy link
Owner

Still haven't forgotten! I will get to this as soon as life circumstances permit.

@tomjaguarpaw
Copy link
Owner

Still haven't forgotten ...

@tomjaguarpaw
Copy link
Owner

My results

Test suite benchmark: RUNNING...
benchmarking pepareQuery/narrowTable/findByPkSelect1
time                 49.45 μs   (49.20 μs .. 49.73 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 49.10 μs   (48.95 μs .. 49.35 μs)
std dev              633.3 ns   (479.0 ns .. 842.0 ns)

benchmarking pepareQuery/narrowTable/findByPkCompleteRow
time                 80.90 μs   (80.62 μs .. 81.18 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 80.60 μs   (80.40 μs .. 80.95 μs)
std dev              891.4 ns   (622.6 ns .. 1.398 μs)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkSelect1
time                 203.2 μs   (202.4 μs .. 204.5 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 203.0 μs   (201.9 μs .. 204.6 μs)
std dev              4.177 μs   (2.988 μs .. 6.176 μs)
variance introduced by outliers: 14% (moderately inflated)

benchmarking pepareQuery/narrowQueryLeftJoin/findByPkCompleteRow
time                 292.4 μs   (291.7 μs .. 293.4 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 291.1 μs   (290.2 μs .. 292.2 μs)
std dev              3.128 μs   (2.314 μs .. 4.162 μs)

benchmarking pepareQuery/narrowQueryLeftJoinF/Select1
time                 223.1 μs   (222.7 μs .. 223.6 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 221.7 μs   (221.2 μs .. 222.3 μs)
std dev              1.862 μs   (1.521 μs .. 2.312 μs)

benchmarking pepareQuery/narrowQueryLeftJoinF/CompleteRow
time                 386.5 μs   (385.2 μs .. 388.0 μs)
                     1.000 R²   (1.000 R² .. 1.000 R²)
mean                 383.3 μs   (382.2 μs .. 384.4 μs)
std dev              3.644 μs   (2.897 μs .. 4.654 μs)

Acquiring database connection...
Dropping & re-creating test tables...
benchmarking runQuery/narrowTable/findByPkSelect1/opaleye
time                 866.0 μs   (791.3 μs .. 939.2 μs)
                     0.903 R²   (0.846 R² .. 0.946 R²)
mean                 1.042 ms   (908.2 μs .. 1.347 ms)
std dev              629.9 μs   (322.4 μs .. 1.111 ms)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowTable/findByPkSelect1/handwritten
time                 488.1 μs   (450.3 μs .. 524.1 μs)
                     0.945 R²   (0.908 R² .. 0.974 R²)
mean                 540.2 μs   (478.5 μs .. 704.2 μs)
std dev              341.2 μs   (101.5 μs .. 630.4 μs)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/opaleye
time                 1.165 ms   (1.082 ms .. 1.234 ms)
                     0.916 R²   (0.856 R² .. 0.954 R²)
mean                 1.312 ms   (1.171 ms .. 1.622 ms)
std dev              662.6 μs   (379.0 μs .. 1.241 ms)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowTable/findByPkCompleteRow/handwritten
time                 480.0 μs   (445.7 μs .. 520.9 μs)
                     0.941 R²   (0.911 R² .. 0.975 R²)
mean                 518.0 μs   (468.4 μs .. 683.4 μs)
std dev              261.6 μs   (94.31 μs .. 505.6 μs)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/opaleye
time                 1.593 ms   (1.473 ms .. 1.703 ms)
                     0.914 R²   (0.858 R² .. 0.951 R²)
mean                 1.894 ms   (1.681 ms .. 2.300 ms)
std dev              958.6 μs   (612.8 μs .. 1.558 ms)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/Select1/handwritten
time                 625.9 μs   (516.0 μs .. 818.9 μs)
                     0.594 R²   (0.445 R² .. 0.946 R²)
mean                 662.8 μs   (563.4 μs .. 883.6 μs)
std dev              454.5 μs   (215.5 μs .. 735.3 μs)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/opaleye
time                 1.852 ms   (1.699 ms .. 2.000 ms)
                     0.909 R²   (0.862 R² .. 0.941 R²)
mean                 2.253 ms   (2.027 ms .. 2.639 ms)
std dev              984.6 μs   (629.0 μs .. 1.640 ms)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin/CompleteRow/handwritten
time                 611.2 μs   (560.9 μs .. 656.1 μs)
                     0.935 R²   (0.899 R² .. 0.963 R²)
mean                 670.9 μs   (601.7 μs .. 842.7 μs)
std dev              340.7 μs   (168.3 μs .. 636.7 μs)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin3/CompleteRow/opaleye
time                 3.824 ms   (3.560 ms .. 4.088 ms)
                     0.937 R²   (0.893 R² .. 0.964 R²)
mean                 4.059 ms   (3.696 ms .. 4.563 ms)
std dev              1.286 ms   (814.1 μs .. 2.159 ms)
variance introduced by outliers: 95% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoin3/CompleteRow/handwritten
time                 679.8 μs   (631.2 μs .. 733.1 μs)
                     0.936 R²   (0.898 R² .. 0.964 R²)
mean                 724.1 μs   (661.6 μs .. 848.8 μs)
std dev              287.4 μs   (161.9 μs .. 512.6 μs)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoinF/CompleteRow/opaleye
time                 3.312 ms   (2.981 ms .. 3.591 ms)
                     0.913 R²   (0.851 R² .. 0.951 R²)
mean                 4.062 ms   (3.401 ms .. 6.629 ms)
std dev              3.367 ms   (1.004 ms .. 7.316 ms)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowQueryLefttJoinF/CompleteRow/handwritten
time                 703.3 μs   (627.7 μs .. 760.3 μs)
                     0.794 R²   (0.529 R² .. 0.949 R²)
mean                 1.070 ms   (755.2 μs .. 2.190 ms)
std dev              1.782 ms   (329.9 μs .. 3.722 ms)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowQuery/Limit-Offset/CompleteRow/opaleye
time                 1.850 ms   (1.754 ms .. 1.962 ms)
                     0.917 R²   (0.854 R² .. 0.953 R²)
mean                 2.008 ms   (1.809 ms .. 2.333 ms)
std dev              793.6 μs   (558.0 μs .. 1.315 ms)
variance introduced by outliers: 98% (severely inflated)

benchmarking runQuery/narrowQuery/Limit-Offset/CompleteRow/handwritten
time                 626.9 μs   (586.3 μs .. 676.1 μs)
                     0.946 R²   (0.909 R² .. 0.975 R²)
mean                 666.8 μs   (611.7 μs .. 865.3 μs)
std dev              280.8 μs   (111.2 μs .. 580.2 μs)
variance introduced by outliers: 99% (severely inflated)

benchmarking runQuery/narrowQuery/groupBy/CompleteRow/opaleye
time                 26.29 ms   (23.24 ms .. 30.75 ms)
                     0.880 R²   (0.797 R² .. 0.938 R²)
mean                 22.80 ms   (19.41 ms .. 25.66 ms)
std dev              6.923 ms   (5.527 ms .. 8.595 ms)
variance introduced by outliers: 89% (severely inflated)

benchmarking runQuery/narrowQuery/groupBy/CompleteRow/handwritten
time                 20.06 ms   (12.32 ms .. 27.87 ms)
                     0.630 R²   (0.351 R² .. 0.864 R²)
mean                 26.88 ms   (22.71 ms .. 31.62 ms)
std dev              9.674 ms   (7.447 ms .. 12.96 ms)
variance introduced by outliers: 94% (severely inflated)

Test suite benchmark: PASS
Test suite logged to: dist/test/opaleye-0.6.0.0-benchmark.log

@tomjaguarpaw
Copy link
Owner

I have a lateral thinking suggestion to solve this. How about compiling the queries to prepared statements with parameters? That should then execute at full speed on the server. What do you think?

@saurabhnanda
Copy link
Contributor Author

saurabhnanda commented Dec 3, 2017 via email

@tomjaguarpaw
Copy link
Owner

Pg-simple doesn't supper prepare/execute, does it?

I don't know but I can't imagine why it wouldn't. Aren't they just standard Postgres commands that (presumably) pg-simple can send?

@tomjaguarpaw
Copy link
Owner

Thanks for these excellent benchmark which helped me understand the problem better. It seems clear that the nesting is causing the queries to take hundreds of microseconds longer to execute. I have a couple of problems

  • It's quite a lot of work to fix this (a few days) and I don't have any time to do so

  • I have not yet convinced myself that rewriting the queries will save time overall. The rewrite step make take more time than it saves! It would take approximately half a day more to benchmark to check this.

At this point I'm not really sure what to do besides discussing with you further how critical this is for your use case and whether we have the possibility of any other workarounds.

@tomjaguarpaw tomjaguarpaw mentioned this pull request Sep 1, 2020
Open
@tomjaguarpaw
Copy link
Owner

I brought this up-to-date with master on https://github.com/tomjaguarpaw/haskell-opaleye/tree/perf/benchmarks

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Serious performance degradation due to SQL query nesting
2 participants
@saurabhnanda @tomjaguarpaw