Feature/gale local ocr #671
Conversation
* Added units for local ocr code
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
ppa/archive/gale.py
Dismissed
|
|
||
| stub_dir = item_id[::3][1:] # Following conventions set in ppa-nlp | ||
| ocr_txt_fp = f"{ocr_dir}/{stub_dir}/{item_id}/{item_id}_{page_num}0.txt" | ||
| with open(ocr_txt_fp) as reader: |
Check failure
Code scanning / CodeQL
Uncontrolled data used in path expression High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix AI about 1 month ago
To fix the problem, we need to ensure that the constructed file path is contained within a safe root directory. This can be achieved by normalizing the path using os.path.normpath and then checking that the normalized path starts with the root directory. This approach will prevent path traversal attacks by ensuring that the final path does not escape the intended directory.
- Normalize the constructed file path using
os.path.normpath. - Check that the normalized path starts with the root directory (
ocr_dir). - Raise an exception if the check fails.
| @@ -38,3 +38,5 @@ | ||
| stub_dir = item_id[::3][1:] # Following conventions set in ppa-nlp | ||
| ocr_txt_fp = os.path.join(ocr_dir, stub_dir, item_id, f"{item_id}_{page_num}0.txt") | ||
| ocr_txt_fp = os.path.normpath(os.path.join(ocr_dir, stub_dir, item_id, f"{item_id}_{page_num}0.txt")) | ||
| if not ocr_txt_fp.startswith(os.path.normpath(ocr_dir)): | ||
| raise Exception("Access to the specified file is not allowed") | ||
| with open(ocr_txt_fp) as reader: |
There was a problem hiding this comment.
This copilot suggestion is interesting, I wasn't familiar with normpath. This shouldn't happen in our code the way we're using the new method, although the check shouldn't hurt. I wonder if using os.path.join helps any instead of using the f-string to construct the path.
There was a problem hiding this comment.
Yeah, using os.path.join seems like a better idea to me.
| } | ||
| mock_get_item.return_value = api_response | ||
| # Set up get_local_ocr so that only the 3rd page's text is found | ||
| mock_get_local_ocr.side_effect = [FileNotFoundError, FileNotFoundError, "local ocr text"] |
ppa/archive/gale.py
Dismissed
|
|
||
| stub_dir = item_id[::3][1:] # Following conventions set in ppa-nlp | ||
| ocr_txt_fp = f"{ocr_dir}/{stub_dir}/{item_id}/{item_id}_{page_num}0.txt" | ||
| with open(ocr_txt_fp) as reader: |
There was a problem hiding this comment.
This copilot suggestion is interesting, I wasn't familiar with normpath. This shouldn't happen in our code the way we're using the new method, although the check shouldn't hurt. I wonder if using os.path.join helps any instead of using the f-string to construct the path.
Co-authored-by: Rebecca Sutton Koeser <[email protected]>
No description provided.