Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change Scrapper Scheduler to be Major Based #5

Open
1 of 3 tasks
fairuzi10 opened this issue Sep 1, 2020 · 0 comments
Open
1 of 3 tasks

Change Scrapper Scheduler to be Major Based #5

fairuzi10 opened this issue Sep 1, 2020 · 0 comments

Comments

@fairuzi10
Copy link
Contributor

This issue is a:

  • bug report
  • feature request
  • styling change request

Current behavior:

Currently, we are running the scrapper scheduler every one hour for all majors. This will make scrapped schedules outdated for pretty long (1 hour). If we run the scheduler in a shorter time, there's a concern that SIAK will block our IP or even block our SSO accounts, which is pretty bad since we collected credentials from so many people across majors and faculties.

Requested behavior:

Change the scrapper to be major-based. We may need to add last_updated field in major. If someone with major X logged in and the last_updated of X is 5 minutes or more before now, we will run the scrapper.

Why this behavior is better:

We will save many unnecessary requests we're doing currently.

@fairuzi10 fairuzi10 changed the title Change Scrapper Scheduler to Major Based Change Scrapper Scheduler to be Major Based Sep 1, 2020
@fahdikrie fahdikrie transferred this issue from ristekoss/susunjadwal Oct 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants