Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add configurable timeout value #1

Open
wants to merge 5 commits into
base: master
Choose a base branch
from

Conversation

CodeNinja-Z
Copy link

Description

This PR adds a new configurable argument read_timeout to the options hash to prevent bot requests from taking too long and constantly occupying server threads, which could cause a slow response alert on our end.

JIRA issue: https://himama.atlassian.net/browse/FLRS-790

@CodeNinja-Z CodeNinja-Z self-assigned this Mar 2, 2023
if @options[:read_timeout]
error_type = '503 Service Unavailable'
error_msg = "ERROR: timed out while trying to connect. #{e.message}"
raise Timeout::Error.new(error_type), error_msg
Copy link
Author

@CodeNinja-Z CodeNinja-Z Mar 2, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We only need to catch the timeout exception to solve our issue. So I added the rescue specifically for timeout if the read_timeout option is provided. Otherwise, just follow what it was, ie. catch the StandardError but not do anything.
I'm not sure what format of the error data we should return to the crawlers bot, maybe JSON? I looked up but didn't find useful resources. So just return the Rails standard format.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant