-
Notifications
You must be signed in to change notification settings - Fork 74
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Proposal]: use case for SG AI #524
Labels
proposal
You want to address a specific problem? Let us know about your idea.
Comments
DiTo97
added
the
proposal
You want to address a specific problem? Let us know about your idea.
label
May 18, 2024
DiTo97
changed the title
[Proposal]: perfect use case for SG AI
[Proposal]: use case for SG AI
May 18, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Problem statement
A lot of manual work and tuning goes into every single publisher that's currently maintained, and still requires constant monitoring if anything changes in the supported news outlets or web sources.
Solution
replace manual and labour-intensive scraping code with SG AI, whose you-only-scrape-once (YOSO) concept serves that purpose specifically: you write the scraping pipeline once, and leverage powerful LLMs (open-source or closed-source) to extract the articles in the desired format regardless of the web source or its HTML code changing over time.
write a single smart scraper graph tailored for news and articles crawling in the desired relational format, common to all available publishers and outlets.
Draft
Open Questions
No response
The text was updated successfully, but these errors were encountered: