Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve use of SQLAlchemy statement cache with search queries #3101

Merged
merged 10 commits into from
Jul 3, 2023

Conversation

lonvia
Copy link
Member

@lonvia lonvia commented Jul 3, 2023

SQLAlchemy has an internal statement cache which caches rendered SQL. This PR makes some changes to the code to make better use of this cache:

  • Replace use of GeoAlchemy2 with a custom Geometry type. The Geometry type from GeoAlchemy does not support caching. Given that we only need a small subset of the functionality, a custom type is easier and also allows some adaptions to our data types and queries.
  • Replace regexp_replace() functions with simple operators to work around a SQLAlchemy bug.
  • Wrap penalty computation for weighing search results in a PL/pgSQL function. The approach of creating a huge CASE statement creates very variable SQL for each query, which is poison for the statement cache.
  • Queries now get the user-supplied parameters as bind parameters during exec. This is partially to make it easier to work with our custom Geometry type but also means more static SQL constructs.

Also contains some changes to the debug output of SQL queries. It is notoriously hard to get inlining of bind parameters right for all supported versions SQLAlchemy and Python.

Reverse queries are 15-20% faster with the change. Search queries are around 5-7% faster on SQLAlchemy 2.

Note that SQLAlchemy 1.4 still has some issues with caching very large queries, which concerns all searches with a housenumber. Thus it tends to be quite worse for search performance. Given that the issue is gone in SQLAlchemy 2, I'm not going to dig into this more. It may go away anyway when starting to use SQLAlchemy lambdas, which is next on the list.

@lonvia lonvia merged commit ce17b0e into osm-search:master Jul 3, 2023
@lonvia lonvia deleted the custom-geometry-type branch July 3, 2023 09:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant