Skip to content

Commit

Permalink
scraper.fix: url encoding for various urls
Browse files Browse the repository at this point in the history
Urls for fs.hacettepe and google docs were failing. Therefore,
I excluded "=", "?", "%" characters too.
  • Loading branch information
furkansimsekli committed Jun 8, 2023
1 parent 5be96e9 commit a099b18
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/scraper.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def _complete_url(self, url: str) -> str:

@staticmethod
def _fix_invalid_url(url: str) -> str:
return urllib.parse.quote(url, "\./_-:")
return urllib.parse.quote(url, "\./_-:=?%")

async def get_announcements(self) -> list[dict]:
async with aiohttp.ClientSession() as session:
Expand Down

0 comments on commit a099b18

Please sign in to comment.