This repository has been archived by the owner on Nov 30, 2018. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 4
/
READMEHeritrix2.txt
115 lines (78 loc) · 4.33 KB
/
READMEHeritrix2.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
Welcome to HBase-Writer README for Heritrix 2.
This document can also be found online here:
http://code.google.com/p/hbase-writer/wiki/READMEHeritrix2
Specific versions of HBase-Writer now support different
version combinations of Heritrix and HBase. Please refer to
http://code.google.com/p/hbase-writer/wiki/VERSIONS
for a more detailed list.
Before reading this document, please make sure the
HBase-Writer version you downloaded is meant to work with
Heritrix 2.
= TABLE OF CONTENTS =
* SETUP
* CONFIGURING HERITRIX
* FILE FORMAT
* COMPILING THE SOURCE
* BUILDING THE JAR
* BUILDING THE SITE-REPORT
The hbase-writer is an extension to the Heritrix open
source crawler written by the Internet Archive (http://crawler.archive.org/)
that enables it to store crawled content directly into HBase, the Hadoop
Distributed Database (http://hbase.org/ http://hadoop.apache.org/). Hadoop implements
the Map/Reduce distributed computation framework on top of HDFS.
hbase-writer-processor writes crawled content into hbase table records with rowkeys.
Having the crawled data in hbase tables is directly supported by the Map/Reduce framework.
This facilitates running high-speed, distributed computations over content crawled with Heritrix.
= QUICK SETUP =
1. Start an instance of hbase.
2. Install heritrix-2.x.x
3. Copy the following jar files into the ${HERITRIX_HOME}/lib directory:
hbase-writer-x.x.x.jar
hbase-x.x.x.jar
zookeeper-x.x.x.jar
hadoop-x.x.x-core.jar
log4j-x.x.x.jar
4. Start Heritrix
= CONFIGURING HERITRIX =
On the "Settings for sheet 'global'":
- Click on a 'details' link and add new processor com.powerset.heritrix.writer.HBaseWriterProcessor.
- Return to the global sheet and set this as your writer (the page will not draw completely if
did not type in the name of the processor properly -- see heritrix_out.log for errors).
- Set at least the zkquorum and table configuration for HBaseWriterProcessor.
zkQuorum (required)
The zookeeper quroum that serves the hbase master address. Since hbase-0.20.0, the master server's address is returned by the zookeeper quorum.
So this value is a comma seperated list of the zk quorum.
e.g. zkHost1,zkHost2,zkHost3
hbaseTableName (required)
Which table in HBase to write the crawl to. This table will be created automatically if it doesnt exist.
e.g. crawl
zkPort (defaults to 2181, which is the zookeeper default)
The zookeeper quroum client port that clients should connect to to get HBase information.
e.g. 2181
contentColumnFamily
The column family name for where you want to save the content to. Defaults to "newcontent".
contentColumnName
The column qualifier name for where you want to save the content to. Defaults to "raw_data" which becomes "newcontent:raw_data"
curiColumnFamily
The column family name for storing the Crawl URI related information. Defaults to "curi".
ipColumnName
The column qualifier name for storing the IP address. Defaults to "ip" which becomes "curi:ip".
pathFromSeedColumnName
The column qualifier name for storing the path from seed. Defaults to "path-from-seed" which becomes "curi:path-from-seed".
isSeedColumnName
The column qualifier name for storing whether the crawl is a seed. Defaults to "is-seed" which becomes "curi:is-seed".
viaColumnName
The column qualifier name for storing where it came from. Defaults to "via" which becomes "curi:via".
urlColumnName
The column qualifier name for storing the URL. Defaults to "url" which becomes "curi:url".
requestColumnName
The column qualifier name for storing the request. Defaults to "request" which becomes "curi:request".
onlyWriteNewRecords
Set to "false" by default. In default mode, heritrix will crawl all urls regardless of existing rowkeys (urls).
By setting this to "true" you ensure that only new urls(rowkeys) are written to the crawl table.
onlyProcessNewRecords
Set to "false" by default. In default mode, heritrix will process (fetch and parse) all urls regardless of existing rowkeys (urls).
By setting this to "true" you ensure that only new urls(rowkeys) are processed by heritrix. Also, if set to "true",
heritrix doesnt download any content that is already existing as a record in the hbase table.
defaultMaxFileSizeInBytes
Set to 20MB (20*1024*1024 bytes) by default. If data item is fetched and it exceeds this amount, the content will not be written to hbase.