Cedric injest attempt #9
Conversation
cedricdcc
commented
Nov 17, 2023
cedricdcc
commented
- Added auto ingest of files
- Added rdfj2 for teplate designing of the sparql querries.
- Created Watcher that will check for updates on mounted volume.
nicer container-names
make it easy to get the jupyer token
…ask/basic-ttl-ingest
but moving the log elsewhere
reverses symlink direction to make docker happy (and the other usage satisfied) applies the poetry build inside docker inage build to fix #1
further enhances the request from #3 some path changes allong the way (conform the reeversed symlinks)
this marks an important milestone for #4 as we can now insert triples!
…-2023 into task/basic-ttl-ingest
completes the goal of "test driving the ingest of rdf into graphdb" fixes #4
|
|
||
| - name: Commit and push changes | ||
| run: | | ||
| git config --global user.name 'cedricdcc' |
There was a problem hiding this comment.
? this auto-lint at server side is odd -- pls check how things were setup in pykg2tbl as part of the client-side commit (so no need for user in config)
There was a problem hiding this comment.
rather remove this one and add some todo to have the linting on the client - possibly with git-hook
| - name: Run pytest | ||
| run: | | ||
| cd docker/lwua-ingest/lwua-py | ||
| poetry run pytest ./tests/ |
There was a problem hiding this comment.
better to run this kind of stuff via Makefile --> again pykg2tbl shows how to set it up
There was a problem hiding this comment.
comment still applies
|
|
||
| # Assert | ||
| assert graph is not None | ||
| # Add more assertions based on what the read_graph function is supposed to |
| "../lwua/templates") | ||
| # init J2RDFSyntaxBuilder | ||
| context = f"{URN_BASE}:ADMIN" | ||
| j2rdf = J2RDFSyntaxBuilder( |
There was a problem hiding this comment.
not doable to just reuse the syntax-builder in graph.py?
if so you would be testing that too
There was a problem hiding this comment.
Not really, the template folder location is different
| print(query) | ||
| to_expect = "INSERT DATA { GRAPH <urn:lwua:INGEST:test_file.txt> { <http://example.com/subject> <http://example.com/predicate> <http://example.com/object> . } }" | ||
| # Assert | ||
| assert query == to_expect, f"Expected '{to_expect}', but got '{query}'" |
There was a problem hiding this comment.
unsure -- but if RDFlib also supports insert statements onto an in-memory graph -- then you could actually provide a mock_gdb variable that allows for further testing
|
For future comments on the dereferencing part of the project please use the following gh-issue. This is a temp solution until a seperate dereferencer branch is created. |
This reverts commit c6246c8.
…user-analysis-2023 into cedric_injest_attempt
There was a problem hiding this comment.
Would like to see this added to the codebase, but also add some issues to follow up on the missing parts:
- reconsider how the linting works and is applied
- introduce a Makefile for tests, lints, docker-build, ...
- then use the
make testin the testing workflow (not the direct call to pytest) - provide more and better tests as the current ones might give a false feeling of security
