Sneakpeek

Sneakpeek - is a platform to author, schedule and monitor scrapers in an easy, fast and extensible way. It's the best choice for scrapers that have some specific complex scraping logic that needs to be run on a constant basis.

Key features

• Horizontally scalable
• Robust scraper scheduler and priority task queue
• Multiple storage implementations to persist scrapers' configs, tasks, logs, etc.
• JSON RPC API to manage the platform programmatically
• Useful UI to manage all of your scrapers
• Scraper IDE to enable you developing scrapers right in your browser
• Easily extendable via middleware

Demo

Here's a demo project which uses Sneakpeek framework.

You can also run the demo using Docker:

docker run -it --rm -p 8080:8080 flulemon/sneakpeek-demo

Once it has started head over to http://localhost:8080 to play around with it.

Documentation

For the full documentation please visit sneakpeek-py.readthedocs.io

Contributing

Please take a look at our contributing guidelines if you're interested in helping!

Future plans

• Headful and headless browser engines middleware (Selenium and Playwright)
• SQL and AmazonDB storage implementation
• Advanced monitoring for the scrapers' health