Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dev tor integration #1

Open
wants to merge 223 commits into
base: master
Choose a base branch
from
Open

Dev tor integration #1

wants to merge 223 commits into from

Conversation

edx173
Copy link

@edx173 edx173 commented Mar 2, 2023

No description provided.

gmacia and others added 30 commits January 18, 2019 17:37
@gmacia
Added authors to readme
82da64c
@robertomagan
An eepsite must be in just one status.
85c0527
@robertomagan
Changing UNKWNON status to DISCARDED
d223c16
@robertomagan
Refactoring configuration constants.
bf0ce00
@robertomagan
Refactoring in common functionality.
95b02ad
@robertomagan
Changing ok_files and fail_files lists.
f6fa9e5
@robertomagan
Adding extra debug traces
40f30e9
@robertomagan
Sometimes, a spiders tag as *.ok an empty json file. This way, the cu…
5c6ef78 
…rrent site is setup as ERROR.
@robertomagan
Minor changes on post-processing *.ok files
49168ab
@robertomagan
logs folder
fd0fb76
@robertomagan
Changing order in error_to_pending call
90f5f4e
@robertomagan
Changing the way of removing *.ok files
7e8518f
@robertomagan
Changing the way of removing *.ok files
fc62b27
@robertomagan
Dealing with large json files for extracting their connections.
bc503a8
@robertomagan
Using Scrapy Pipelines for overwriting JSON files thus keeping last s…
0762b5c 
…crapped items in one line file.
@robertomagan
Bug fixed
ff5d7fa
@robertomagan
Dealing with large json files for extracting their connections.
ac6003b
@robertomagan
Setting up the inferred site language in the data base
d355e29
@robertomagan
Getting original seeds.
b2f0511
@robertomagan
Code refactoring: files and folders.
967cd8b
@robertomagan
Deteting exception folders ...
ae7f318
@robertomagan
Moving method getting the crawler status to siteutils.
bdcec3f
@robertomagan
First approach on developing site discovering process.
216340f
@Abellan09
Adding memory limit algorithm to the spider
fdd0505
@Abellan09
Adding the eepsite's total number of internal pages
f78856e
@Abellan09
Analyzing details from the main page of an eepsite
60d653d
@robertomagan
Discovering procedure: sequential approach.
5bced33
@robertomagan
Discovering process: distributed approach.
dd631d3
@robertomagan
Adding regex package to requirements.txt file
13b8a4d
@robertomagan
Son minor processing status flow changes to reset ERROR tries when th…
cf54fee 
…e site is set up to DISCOVERING again.
EmilioFigueras and others added 30 commits September 14, 2021 15:42
@EmilioFigueras
Delete c4darknet_modules2.pdf
ec3edc1
@EmilioFigueras
Add files via upload
8ec932c
@EmilioFigueras
Update README.md
b9d00e2
@EmilioFigueras
Merge branch 'master' into master
bebff25
@nesg-ugr
Merge pull request #1 from EmilioFigueras/master
7080a9b 
c4i2p enhacements for a wide darknet use
@EmilioFigueras
Update README.md
5abb0f0
@EmilioFigueras
Merge pull request #2 from EmilioFigueras/patch-1
a02b0ab 
Update README.md
@robertomagan
Update LICENSE
df2d6d0
@robertomagan
Update README.md
7fd1e3c
@robertomagan
I added some comments ...
911f4ed
@edx173
clase TOR_Spider
ab318c3
@edx173
Adición del proxy TOR y de comentarios
07cf8db
@edx173
TOR_Spider, proxy configuration and discovertythread.py modifications…
7bd6cbc 
… added
@robertomagan
New deployment scripts
c1f5525
@robertomagan
New path for remote script execution
f8382e0
@robertomagan
Database connections settings updated
4024687
@robertomagan
Configuration script updated
4da82b6
@robertomagan
Configuration script updated
b1598aa
@robertomagan
Configuration script updated
15bf738
@robertomagan
Deployment script and configuration files - Ready for crawling TOR sites
b104099
@robertomagan
Homogenizing paths for crawling different darknets
7b23555
@robertomagan
Adding dummy floodfill_seeds.txt file for tor and freenet.
7a65f8e
@robertomagan
Adding new i2p seeds and vms for the deployment
8128557
@robertomagan
Adding languages types for detection language engines.
a1b87ac
@robertomagan
Adding scripts for dumping and downloading BBDD after the experiments.
cba7525
@robertomagan
Updating script
9a1c104
@robertomagan
Updating script
f3dd859
@robertomagan
Setting up complete crawling scenario of 15 vms in total.
7f18b2e
@robertomagan
Freenet seed - url quotes removed
61b1d58
@robertomagan
Experimental setup with only 2 machines crawling per darknet
4ba7ea4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@edx173 @gmacia @robertomagan @Abellan09 @EmilioFigueras @nesg-ugr