IJS newsfeed

A clean, continuous, real-time aggregated stream of semantically enriched news articles from RSS-enabled sites across the world.

What it Does

The pipeline performs the following main steps:

Periodically crawl a list of RSS feeds and a subset of Google News and obtain links to news articles
Download the articles, taking care not to overload any of the hosting servers
Parse each article to obtain
- Potential new RSS sources mentioned in the HTML, to be used in step (1)
- Cleartext version of the article body
Enrich the articles with semantic annotations. (using enrycher.ijs.si)
Expose the stream of news articles to end users.

See newsfeed.ijs.si.

BSD-3; see the LICENSE file.

Name		Name	Last commit message	Last commit date
Latest commit History 3 Commits
article_extractor		article_extractor
dispatch		dispatch
langdet		langdet
public_html		public_html
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
bottle.py		bottle.py
cleanDb.py		cleanDb.py
common.py		common.py
db2zmq_cleartext.py		db2zmq_cleartext.py
feedfinder.py		feedfinder.py
history_export.py		history_export.py
http2fs.py		http2fs.py
realtime_cleaner.py		realtime_cleaner.py
runtime_article_sucker_gls.py		runtime_article_sucker_gls.py
runtime_feed_sucker_gx.py		runtime_feed_sucker_gx.py
serialize.py		serialize.py
start_pipeline.py		start_pipeline.py
util.py		util.py
zmq2http_all.py		zmq2http_all.py
zmq2zmq_bloombergness.py		zmq2zmq_bloombergness.py
zmq2zmq_enrych.py		zmq2zmq_enrych.py
zmq2zmq_xenrych.py		zmq2zmq_xenrych.py