this crawler application depends on apache httpclient4.5.2, lucene6.1 and htmlparser2.1
the main class at com.engine.launch.EngineLaunch.java
i specified ten threads at initConfiguration.properties, so the framework like this producer-consumer pattern.
-
public queue in class CrawlerManager, which maintains a instance of Todo and VisitedMap
-
Even if all the threads of producers, but also consumers