⚡️ feat(middleware): Add functionality to cache homepage search results

[maxachis]

Fixes

• Partially addresses Automated search engine calls - Auto-Googler data-source-identification#53

Description

• Added new middleware file homepage_search_cache.py to handle caching of search results for agencies without homepage URLs
• Added SQL queries to retrieve agencies without homepage URLs and update the search cache
• Implemented functions to get agencies without homepage URLs and update search cache
• Added new resource HomepageSearchCache in resources/HomepageSearchCache.py to handle API requests for retrieving and updating search cache

Testing

• Not sure! I could set up an integration test for it, or we could deploy it and see if it works/doesn't break things when I reconfigure the data source identification code to work with it, or try some unknown third option.

Performance

• Creation of new endpoint. As untested, performance impact unknown, but likely minimal.

Docs

• https://app.gitbook.com/o/-MXypK5ySzExtEzQU6se/s/-MXyolqTg_voOhFyAcr-/~/changes/498/api/v2-api-docs-wip

[josh-chamberlain]

nice! could you please add a section in the docs for this endpoint? that will also take care of PR review—if a reviewer can use the doc to hit the endpoint successfully, that's good.

[maxachis]

@josh-chamberlain Done! Found here: Police-Data-Accessibility-Project/docs#15

Note that I am very unfamiliar with this format, and did not know how to test it. Additionally, because it was fairly redundant, I used ChatGPT to make the first draft. I anticipate this will need editing in the near future, especially as we test it; I'm just not sure how to do that at the moment.

[josh-chamberlain]

@maxachis

1. on docs formatting, the reason it looks weird like that is because we're using gitbook and it's their ~proprietary format. I find making these changes is much easier in the WYSIWYG editor, which is something I hate to say so it must be true. You should be in the gitbook org, and this link will take you to the edit view: https://app.gitbook.com/@pdap/s/pdap/

It's a little weird at first, but the formatting will make a lot more sense.

2. I should have thought of this before—did you consider modifying the agencies endpoint to accept an arg for the blank homepage, rather than creating a new endpoint?

[maxachis]

@maxachis

1. on docs formatting, the reason it looks weird like that is because we're using gitbook and it's their ~proprietary format. I find making these changes is much easier in the WYSIWYG editor, which is something I hate to say so it must be true. You should be in the gitbook org, and this link will take you to the edit view: https://app.gitbook.com/@pdap/s/pdap/

It's a little weird at first, but the formatting will make a lot more sense.

2. I should have thought of this before—did you consider modifying the agencies endpoint to accept an arg for the blank homepage, rather than creating a new endpoint?

@josh-chamberlain
For 1: Is that mentioned somewhere in the documentation? Might be useful to add if not.
For 2: The challenge is that this is specifically for recording a multitude of possible homepages, without a clear idea of which is most viable. Currently, it defaults to finding 10 possible homepages per agency. So we can't easily update the agencies row itself.

[josh-chamberlain]

@maxachis for 1. I left a comment on the docs PR
2. oh, right—mixed up the context! thanks

[maxachis]

@josh-chamberlain Made a Docs Pull Requests, which I've linked in the main post!

[josh-chamberlain]

@maxachis thank you! I approved, since the button was just looking at me, but if you are making docs in support of a PR I do not mind if you just reference + self-merge.