A simple framework for crawlers

Runtime Features

Progress visualization
Suspend crawling when meet errors or network issues and can continue later on
Depth first crawling to maximize parallelism
Parallelization controlled by thread pool to save memory
Decoupled connector to control connection behavior

Develop Features

Assemble crawl logics in a very flexible style like pipelines
Decouple crawl logics and collected contents
Debuggability: Automatically collect exceptions during crawling
Test mode (TODO)