You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, we are running the scrapper scheduler every one hour for all majors. This will make scrapped schedules outdated for pretty long (1 hour). If we run the scheduler in a shorter time, there's a concern that SIAK will block our IP or even block our SSO accounts, which is pretty bad since we collected credentials from so many people across majors and faculties.
Requested behavior:
Change the scrapper to be major-based. We may need to add last_updated field in major. If someone with major X logged in and the last_updated of X is 5 minutes or more before now, we will run the scrapper.
Why this behavior is better:
We will save many unnecessary requests we're doing currently.
The text was updated successfully, but these errors were encountered:
fairuzi10
changed the title
Change Scrapper Scheduler to Major Based
Change Scrapper Scheduler to be Major Based
Sep 1, 2020
This issue is a:
Current behavior:
Currently, we are running the scrapper scheduler every one hour for all majors. This will make scrapped schedules outdated for pretty long (1 hour). If we run the scheduler in a shorter time, there's a concern that SIAK will block our IP or even block our SSO accounts, which is pretty bad since we collected credentials from so many people across majors and faculties.
Requested behavior:
Change the scrapper to be major-based. We may need to add
last_updated
field inmajor.
If someone with major X logged in and thelast_updated
of X is 5 minutes or more before now, we will run the scrapper.Why this behavior is better:
We will save many unnecessary requests we're doing currently.
The text was updated successfully, but these errors were encountered: