Proposed code to fix issue #2198 finding the spatialite dll in Windows #2339
Conversation
| #:\ indicates we're on a Windows machine study the argument a bit more | ||
| if ":\\" in r"%r" % value: | ||
| path_entry = value.split(":", 2) | ||
| if len(path_entry) < 3: | ||
| return value | ||
| # argument contains a Windows/DOS path and an entry point | ||
| path = path_entry[0] + ":" + path_entry[1] | ||
| entrypoint = path_entry[-1] | ||
| return path, entrypoint | ||
| if ":" not in value: |
There was a problem hiding this comment.
There may be something in the Python standard library - perhaps in the os module - that can handle this here.
|
I'm trying to figure out a good way to test this. It looks like I need spatialite-loadable-modules-5.0.0-win-amd64.7z from http://www.gaia-gis.it/gaia-sins/windows-bin-amd64-latest/ - which I should be able to run in a Here's an example workflow that seems to use a hosted copy of that: https://github.com/tyrylu/feel-the-streets/blob/d951a9fadbfecc8724e91b84e10e8f52747e4bde/.github/workflows/ci.yml#L47 Or this script: https://github.com/AequilibraE/mapmatcher/blob/3407a6b172b2e06d03c319b8a1ffb56a0fc1f9b2/tests/setup_windows_spatialite.py |
|
Ran into this problem:
|
|
Hah, I posted a workaround on that thread:
|
|
That workaround didn't work, not sure if it could EVER have worked since https://pypi.org/project/pysqlite3-binary/#files does not list any wheels for Windows. Oh - it's because I suggested the workaround in that other thread for Linux, not for Windows. |
|
Filed an issue here: |
|
Wow, yeah it turns out Claude wrote me the wrong code for that. My mistake. Moving on. |
|
Here's what's in that uncompressed archive (partly):
|
|
|
|
Still getting this error:
|
|
Maybe I'm using |
|
That failed too, same error. Looks like the machine is running this:
Hard to tell if that's AMD64 or x86 - so I downloaded https://github.com/actions/runner-images/releases/download/win22%2F20240811.1/sbom.windows-2022.json.zip to see if that had any clues. Unzipped that's a 345M JSON file! And I couldn't see clues in that either. |
|
Proved to myself that loading extensions on Windows CAN work, using name: sqlite-vec on Windows
on:
push:
branches: [ main ]
jobs:
setup:
runs-on: windows-latest
steps:
- name: Set up Python 3.12
uses: actions/setup-python@v5
with:
python-version: '3.12'
- name: Install sqlite-vec
run: |
pip install sqlite-vec
- name: Verify installation
run: |
python -c @"
import sqlite3, os
import sqlite_vec
db = sqlite3.Connection(':memory:')
db.enable_load_extension(True)
db.load_extension(sqlite_vec.loadable_path())
print(db.execute('select vec_version()').fetchall())
"@Output: |
|
https://github.com/mxschmitt/action-tmate supports Windows, I'm going to try that next. |
|
OK, this worked in a SSH session to that |
|
That worked! I have successfully run SpatiaLite on Windows on GitHub Actions. Testing it like this: Returned output that included this: "extensions": {
"json1": null,
"spatialite": {
"spatialite_version": "5.1.0",
"spatialite_target_cpu": "x86_64-w64-mingw32",
"check_strict_sql_quoting": 0, |
|
This is a lot of very awesome work we'll use going forward testing with Windows on our plugins! Thank you! Just in case this helps, there are test cases for the proposed Python fix that I believe show it doesn't break the path: https://github.com/hcarter333/datasette/blob/main/tests/test_loadextension.py |
|
When I SSH into the The trick is to run If I try to run it from anywhere else using a path to |
|
The problem I'm having here is that this fix is meant to allow you to run something like this: python -m datasette counties.db --load-extension=C:\Windows\System32\mod_spatialite.dllBut I can't figure out how to have (I'm not at all Windows literate so it's very likely I'm missing something basic here.) |
|
Maybe that's because |
|
With Claude's help this worked: |
|
Am I right in thinking that If so, does If that's the case then we should document it - tell people on Windows that they can either install SpatiaLite plus its dependencies in a folder-on-the-path, OR they can Or we could show people how to temporarily set their PATH while running Datasette, which I haven't quite 100% confidently figured out how to do yet. |
|
We added the standard Windows location to SPATIALITE_PATHS as in |
|
Here it is starting up with the run command at the bottom of https://github.com/hcarter333/rm-rbn-history/blob/main/docs/tool_suite.md spatialitewin.mp4 |
|
I just removed nopath.mp4followed by afternopath.mp4 |
|
Do you know if it’s possible to use SpatiaLite on Windows with this Datasette branch if the mod_spatialite.dll stays in another folder that isn’t on your Windows PATH? |
|
I’m looking to add definitive documentation describing how to run SpatiaLite with Datasette on Windows, so I’m trying to understand if you need to copy a bunch of files into |
|
I believe it will work. the videos shown above are with the spatialite directory removed from my path. The only file that's in C:\Windows\System32 is mod_spatialite.dll. I'm going to do the following experiment:
so far, so good dirpathshort.mp4Checking for actual querying now: worksdirpathquery.mp4Works well! |
|
Where did you get your SpatiaLite DLL from? Could it be different from the one I’ve been using, and hence not need additional DLLs to work? |
|
With the System32 path added in the pull request, the user doesn't need to add the path on the command line. If the user would like to place the file in a different location, they can and then define the path to the file on the command line. |
|
After an extensive conversation with Claude my current mental model is this: The version of SpatiaLite I have been using comes as an archive containing a bunch of different files - To successfully run the SpatiaLite extension, Windows needs to be able to find those other DLLs too. To find those, they need to be on the DLL search path - which by default includes the current directory in which the program is running, plus if those extra DLLs made it into As of Python 3.8 there is a |
Does that only works if both What happens if that installer overwrites an existing slightly incompatible |
|
I had forgotten about all the other files. What I would suggest, but am not currently doing, is to place the spatialite install files in their own directory. I was just reading through my TIL as I was installing all of this I found: =================================================================== This is a note for myself for later when I try to construct a better set of instructions for adding Spatialite to Windows for use with Datasette. You can instruct Windows to look for DLLs in a new directory simply by adding that directory to your PATH envrionment variable set PATH=C:\path to your dll;%PATH% via. ============end of TIL==================== However, at some point I moved the files into my System32 directory. So, it seems like placing the dlls in a separate directory of their own and then adding that directory via $PATH will work. Would it be possible to try that using the setup from #2339 ? |
|
Sorry, I'm a little discombobulated. I got my install files from |
|
Which of those two files did you use? Did you have to unzip the files and then manually copy them into the (I’m completely new to installing this kind of thing on Windows, so I’m asking a ton of questions to make sure I fully understand the process for the future.) |
|
Another extremely basic question: does Windows come with the ability to decompress those |
|
If I’m right in that SpatiaLite only works on Windows if you ensure that all of the files from that http://www.gaia-gis.it/gaia-sins/windows-bin-amd64/mod_spatialite-5.1.0-win-amd64.7z file have been somehow added to the Windows system path, then maybe Datasette itself needs to learn that if the user passes |
|
Running an informal survey on Mastodon to see if there are any other ways people install SpatiaLite on Windows that are worth knowing about for the docs: https://fedi.simonwillison.net/@simon/112969302245132317 |
|
I created https://github.com/simonw/spatialite-windows-binaries to provide stable URLs for SpatiaLite Windows binaries so I can download them from GitHub Actions workflows without worrying that the URLs on the official site might change in the future. |
|
Now when I SSH into a Windows GitHub worker (via https://github.com/simonw/playing-with-actions-single/blob/main/.github/workflows/windows-tmate.yml) I can run this: Unfortunately I can still only run SpatiaLite using this: cd mod_spatialite-5.1.0-win-amd64
datasette --load-extension mod_spatialite.dllI cannot figure out how to run it NOT from inside that directory. All of my attempts at adding that directory to the Windows PATH have failed: powershell
$env:PATH += ";$PWD\mod_spatialite-5.1.0-win-amd64"
cd
datasette --load-extension mod_spatialite.dllThat returns an error, but I was hoping it would pick up the I've also failed to get that to work using raw Python: import sqlite3
db = sqlite3.connect(":memory:")
db.enable_load_extension(True)
db.load_extension("mod_spatialite.dll")This works when I'm in the My goal here is to demonstrate to myself that you can run |
|
I am using the mod_spatialite 7z file, not the tools 7z file. It was July of 2023 when I first installed all this, so I think, but don't know, that I did manually unzip the file and copy the contents into my System32 folder. Except: I noticed that there's a sqlite3.exe file in the archive and that is definitely not in my System32 folder. Ha :) You may have come up with a reason for me to finally try Mastodon. :) If anything cool comes up, I'd love to know more because I've also been coming up to speed on adding dlls to Windows for the last year or so. Windows 11 will unzip .7z files using Windows Explorer. Just to throw out an idea, what if Datasette used py7zr to unzip the spatialite file in place, and then re-zipped the files when Datasette turned off? It would be a bit messy in the interim, but it sounds like it would fit the 'current directory for dlls' search option. We've been looking at a similar option of using pytar to unzip satellite data, (~4000 files per day of interest), into a directory for our satellite mapping Datasette app. |
Code fix addressing #2198 . The pull request modifies datasette
/utils/__init__.pyby adding code to handle the presence of:characters in Windows\DOS paths. The proposed fix maintains handling of:separated entry points in both Windows and Linux. This is demonstrated by the added pytest test cases. Documentation has been modified to include a line about installing Spatialite on Windows taken from the django documentation.📚 Documentation preview 📚: https://datasette--2339.org.readthedocs.build/en/2339/