Feat/data augmentation #13
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Add scripts for downloading data from polish court API * Refine logging * Refactor to make mongo bulk writes * Fix typing errors * Add missing dependency * Refine retiries and log warning on invalid pl_court_api params --------- Co-authored-by: Jakub Binkowski <[email protected]>
* Improve error handling in download_pl_content.py * Add dataset dump scrip * Add pl dataset to DVC * Add simple data analysis notebook * Extract text from pl judgements * Refine text extraction and add analysis * Add addtional details download and ingest * Refine extraction and ingest extracted data to mongo * Add script for chunked embeddings --------- Co-authored-by: Jakub Binkowski <[email protected]>
* first information extraction schema * mlflow tracking * add streamlit app * v1 prompt ready * nbdev * get mongo docs * Parse pl judgements (#4) * Improve error handling in download_pl_content.py * Add dataset dump scrip * Add pl dataset to DVC * Add simple data analysis notebook * Extract text from pl judgements * Refine text extraction and add analysis * Add addtional details download and ingest * Refine extraction and ingest extracted data to mongo * Add script for chunked embeddings --------- Co-authored-by: Jakub Binkowski <[email protected]> * first information extraction schema * mlflow tracking * add streamlit app * v1 prompt ready * nbdev * get mongo docs * update nbdev * small fixes --------- Co-authored-by: Jakub Binkowski <[email protected]> Co-authored-by: Jakub Binkowski <[email protected]>
* first information extraction schema * mlflow tracking * add streamlit app * v1 prompt ready * nbdev * get mongo docs * Add chain for transforming user queries into schema * merge artifact --------- Co-authored-by: Łukasz Augustyniak <[email protected]> Co-authored-by: Jakub Binkowski <[email protected]>
* dashboard reformat * notebooks noved to nbs * text analysis * makefile fix and nbdev * docker compsoe and streamlit updates * streamlit update * dashboard update * search for judgements works
…DDGES into feat/data-augmentation
…gh workflow error
* fix non starting postgres after restarts crashes * make dashboard nicer for judgements * show only subset * nbs checkpoints
| @@ -2,7 +2,7 @@ lint_dirs := juddges scripts dashboards tests | |||
| mypy_dirs := juddges scripts dashboards tests | |||
|
|
|||
| fix: | |||
| ruff check $(lint_dirs) --fix | |||
| ruff check $(lint_dirs) setup.py --fix | |||
There was a problem hiding this comment.
I dont see the point not to include this script in lint/format stages. Some IDEs will format the script after the smallest change on save. It can be annoying later
| from langchain_core.outputs import Generation | ||
| from langchain_core.utils.json import _parse_json | ||
|
|
||
| CUSTOM_PARSE_JSON_MARKDOWN = re.compile( |
There was a problem hiding this comment.
There was a problem hiding this comment.
original JsonOutputParser parses output successfully always when prompt does not contain any other JSON structure, and the parse_json_markdown pattern is greedy which is not nice
…ntext insights, create index with number of text tokens for judgements-pl
…DDGES into feat/data-augmentation
asawczyn
force-pushed
the
feat/data-augmentation
branch
from
4302ae9 to
b62b5c8
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.