Update ingest_service.py to fix issue Error: 'utf-8' codec can't decode #1171
Conversation
To fix issue: zylon-ai#1166 Error: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte zylon-ai#1166
| @@ -1,4 +1,5 @@ | |||
| import tempfile | |||
| import chardet # Chardet must be put in requirements or manually install with pip install chardet | |||
There was a problem hiding this comment.
You should add the dependency,
poetry add chardet
| @@ -77,7 +78,10 @@ def ingest(self, file_name: str, file_data: AnyStr | Path) -> list[IngestedDoc]: | |||
| # Read as a plain text | |||
| string_reader = StringIterableReader() | |||
| if isinstance(file_data, Path): | |||
| text = file_data.read_text() | |||
| with open(file_data, 'rb') as f2: | |||
There was a problem hiding this comment.
Try to give this variables proper names. Maybe file_handle instead of f2 and charset instead of result2
There was a problem hiding this comment.
I would go even further, and could you also try to avoid to read the file in it's entirety to detect the charset?
More information in chardet documentation: https://chardet.readthedocs.io/en/latest/usage.html#advanced-usage
While the existing implementation work, it is suboptimal in the sense that it reads the file in its entirety before re-reading it?
One could also do the following: read the file in binary mode, store it in text_bin, and then run chardet on it, so that we have text = text_bin.decode(encoding=chardet.detect(text_bin))
This is doing only a single read, and it re-uses the buffer in memory instead of re-reading it.
|
Make sure to run the formatter and lint |
|
Stale pull request |
To fix issue: #1166
Error: 'utf-8' codec can't decode byte 0xd0 in position 0: invalid continuation byte #1166