Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dev tor integration #1

Open
wants to merge 223 commits into
base: master
Choose a base branch
from
Open

Dev tor integration #1

wants to merge 223 commits into from

Commits on Jan 18, 2019

  1. Added authors to readme

    @gmacia
    gmacia committed Jan 18, 2019
    Configuration menu
    Copy the full SHA
    82da64c View commit details
    Browse the repository at this point in the history

Commits on Jan 20, 2019

  1. An eepsite must be in just one status.

    @robertomagan
    robertomagan committed Jan 20, 2019
    Configuration menu
    Copy the full SHA
    85c0527 View commit details
    Browse the repository at this point in the history

  2. Changing UNKWNON status to DISCARDED

    @robertomagan
    robertomagan committed Jan 20, 2019
    Configuration menu
    Copy the full SHA
    d223c16 View commit details
    Browse the repository at this point in the history

  3. Refactoring configuration constants.

    @robertomagan
    robertomagan committed Jan 20, 2019
    Configuration menu
    Copy the full SHA
    bf0ce00 View commit details
    Browse the repository at this point in the history

  4. Refactoring in common functionality.

    @robertomagan
    robertomagan committed Jan 20, 2019
    Configuration menu
    Copy the full SHA
    95b02ad View commit details
    Browse the repository at this point in the history

Commits on Jan 21, 2019

  1. Changing ok_files and fail_files lists.

    @robertomagan
    robertomagan committed Jan 21, 2019
    Configuration menu
    Copy the full SHA
    f6fa9e5 View commit details
    Browse the repository at this point in the history

Commits on Jan 22, 2019

  1. Adding extra debug traces

    @robertomagan
    robertomagan committed Jan 22, 2019
    Configuration menu
    Copy the full SHA
    40f30e9 View commit details
    Browse the repository at this point in the history

  2. Sometimes, a spiders tag as *.ok an empty json file. This way, the cu… 

    …rrent site is setup as ERROR.
    @robertomagan
    robertomagan committed Jan 22, 2019
    Configuration menu
    Copy the full SHA
    5c6ef78 View commit details
    Browse the repository at this point in the history

Commits on Jan 23, 2019

  1. Minor changes on post-processing *.ok files

    @robertomagan
    robertomagan committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    49168ab View commit details
    Browse the repository at this point in the history

  2. logs folder

    @robertomagan
    robertomagan committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    fd0fb76 View commit details
    Browse the repository at this point in the history

  3. Changing order in error_to_pending call

    @robertomagan
    robertomagan committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    90f5f4e View commit details
    Browse the repository at this point in the history

  4. Changing the way of removing *.ok files

    @robertomagan
    robertomagan committed Jan 23, 2019
    Configuration menu
    Copy the full SHA
    7e8518f View commit details
    Browse the repository at this point in the history

Commits on Jan 24, 2019

  1. Changing the way of removing *.ok files

    @robertomagan
    robertomagan committed Jan 24, 2019
    Configuration menu
    Copy the full SHA
    fc62b27 View commit details
    Browse the repository at this point in the history

Commits on Jan 25, 2019

  1. Dealing with large json files for extracting their connections.

    @robertomagan
    robertomagan committed Jan 25, 2019
    Configuration menu
    Copy the full SHA
    bc503a8 View commit details
    Browse the repository at this point in the history

Commits on Jan 28, 2019

  1. Using Scrapy Pipelines for overwriting JSON files thus keeping last s… 

    …crapped items in one line file.
    @robertomagan
    robertomagan committed Jan 28, 2019
    Configuration menu
    Copy the full SHA
    0762b5c View commit details
    Browse the repository at this point in the history

  2. Bug fixed

    @robertomagan
    robertomagan committed Jan 28, 2019
    Configuration menu
    Copy the full SHA
    ff5d7fa View commit details
    Browse the repository at this point in the history

Commits on Jan 29, 2019

  1. Dealing with large json files for extracting their connections.

    @robertomagan
    robertomagan committed Jan 29, 2019
    Configuration menu
    Copy the full SHA
    ac6003b View commit details
    Browse the repository at this point in the history

Commits on Jan 30, 2019

  1. Setting up the inferred site language in the data base

    @robertomagan
    robertomagan committed Jan 30, 2019
    Configuration menu
    Copy the full SHA
    d355e29 View commit details
    Browse the repository at this point in the history

  2. Getting original seeds.

    @robertomagan
    robertomagan committed Jan 30, 2019
    Configuration menu
    Copy the full SHA
    b2f0511 View commit details
    Browse the repository at this point in the history

Commits on Feb 13, 2019

  1. Code refactoring: files and folders.

    @robertomagan
    robertomagan committed Feb 13, 2019
    Configuration menu
    Copy the full SHA
    967cd8b View commit details
    Browse the repository at this point in the history

  2. Deteting exception folders ...

    @robertomagan
    robertomagan committed Feb 13, 2019
    Configuration menu
    Copy the full SHA
    ae7f318 View commit details
    Browse the repository at this point in the history

  3. Moving method getting the crawler status to siteutils.

    @robertomagan
    robertomagan committed Feb 13, 2019
    Configuration menu
    Copy the full SHA
    bdcec3f View commit details
    Browse the repository at this point in the history

  4. First approach on developing site discovering process.

    @robertomagan
    robertomagan committed Feb 13, 2019
    Configuration menu
    Copy the full SHA
    216340f View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2019

  1. Adding memory limit algorithm to the spider

    @Abellan09
    Abellan09 committed Feb 17, 2019
    Configuration menu
    Copy the full SHA
    fdd0505 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2019

  1. Adding the eepsite's total number of internal pages

    @Abellan09
    Abellan09 committed Feb 18, 2019
    Configuration menu
    Copy the full SHA
    f78856e View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2019

  1. Analyzing details from the main page of an eepsite

    @Abellan09
    Abellan09 committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    60d653d View commit details
    Browse the repository at this point in the history

  2. Discovering procedure: sequential approach.

    @robertomagan
    robertomagan committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    5bced33 View commit details
    Browse the repository at this point in the history

  3. Discovering process: distributed approach.

    @robertomagan
    robertomagan committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    dd631d3 View commit details
    Browse the repository at this point in the history

  4. Adding regex package to requirements.txt file

    @robertomagan
    robertomagan committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    13b8a4d View commit details
    Browse the repository at this point in the history

Commits on Feb 25, 2019

  1. Son minor processing status flow changes to reset ERROR tries when th… 

    …e site is set up to DISCOVERING again.
    @robertomagan
    robertomagan committed Feb 25, 2019
    Configuration menu
    Copy the full SHA
    cf54fee View commit details
    Browse the repository at this point in the history

  2. Comment add_fake_discovery_info

    @robertomagan
    robertomagan committed Feb 25, 2019
    Configuration menu
    Copy the full SHA
    55b16cc View commit details
    Browse the repository at this point in the history

Commits on Feb 26, 2019

  1. Some code lines has been added for a better debugging.

    @robertomagan
    robertomagan committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    c40349e View commit details
    Browse the repository at this point in the history

  2. Filtering and limiting the sample size in order not to crash the Goog… 

    …le Translator API
    @Abellan09
    Abellan09 committed Feb 26, 2019
    Configuration menu
    Copy the full SHA
    d974e74 View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2019

  1. Fixing bugs in discovering process and adding some debug code lines.

    @robertomagan
    robertomagan committed Mar 1, 2019
    Configuration menu
    Copy the full SHA
    c5ce9d0 View commit details
    Browse the repository at this point in the history

  2. Simple thread status management.

    @robertomagan
    robertomagan committed Mar 1, 2019
    Configuration menu
    Copy the full SHA
    a1c4069 View commit details
    Browse the repository at this point in the history

  3. Simple thread status management.

    @robertomagan
    robertomagan committed Mar 1, 2019
    Configuration menu
    Copy the full SHA
    43e0bfe View commit details
    Browse the repository at this point in the history

Commits on Mar 5, 2019

  1. Adding HTTP response time to SiteProcessingLog entity.

    @robertomagan
    robertomagan committed Mar 5, 2019
    Configuration menu
    Copy the full SHA
    eef8aba View commit details
    Browse the repository at this point in the history

  2. Adding HTTP response time to SiteProcessingLog entity.

    @robertomagan
    robertomagan committed Mar 5, 2019
    Configuration menu
    Copy the full SHA
    87418c1 View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2019

  1. Dividing into groups the number of words of the main page to facilita… 

    …te the task of language detection
    @Abellan09
    Abellan09 committed Mar 6, 2019
    Configuration menu
    Copy the full SHA
    f2760e6 View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2019

  1. Adding new info about the home, title, links on data base

    @robertomagan
    robertomagan committed Mar 8, 2019
    Configuration menu
    Copy the full SHA
    1f86924 View commit details
    Browse the repository at this point in the history

Commits on Mar 13, 2019

  1. Adding links that cause exceptions to visited_links

    @Abellan09
    Abellan09 committed Mar 13, 2019
    Configuration menu
    Copy the full SHA
    0f28866 View commit details
    Browse the repository at this point in the history

  2. Changing a function's name

    @Abellan09
    Abellan09 committed Mar 13, 2019
    Configuration menu
    Copy the full SHA
    609bde9 View commit details
    Browse the repository at this point in the history

  3. Loading the spider's status properly

    @Abellan09
    Abellan09 committed Mar 13, 2019
    Configuration menu
    Copy the full SHA
    235c802 View commit details
    Browse the repository at this point in the history

  4. Putting in the json file only the final results of the crawling that … 

    …are of our interest if the crawling was realised correctly
    @Abellan09
    Abellan09 committed Mar 13, 2019
    Configuration menu
    Copy the full SHA
    fb35caa View commit details
    Browse the repository at this point in the history

Commits on Mar 16, 2019

  1. Managing atomic database transactions when a new site is crawler. Bug… 

    … fixed with site title encoding and database
    @robertomagan
    robertomagan committed Mar 16, 2019
    Configuration menu
    Copy the full SHA
    5cb6ac2 View commit details
    Browse the repository at this point in the history

  2. Minor logging bugs fixed.

    @robertomagan
    robertomagan committed Mar 16, 2019
    Configuration menu
    Copy the full SHA
    0f8e974 View commit details
    Browse the repository at this point in the history

  3. Some minor changes in spider (init function)

    @Abellan09
    Abellan09 committed Mar 16, 2019
    Configuration menu
    Copy the full SHA
    e8f812c View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2019

  1. Adding the functionality of count the number of scripts and images th… 

    …at contains the main page
    @Abellan09
    Abellan09 committed Mar 18, 2019
    Configuration menu
    Copy the full SHA
    5715043 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2019

  1. Some minor changes in spider debugging

    @Abellan09
    Abellan09 committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    f8cc750 View commit details
    Browse the repository at this point in the history

Commits on Mar 22, 2019

  1. Changing the way of count the number of scripts and images that conta… 

    …ins the main page
    @Abellan09
    Abellan09 committed Mar 22, 2019
    Configuration menu
    Copy the full SHA
    a4fe136 View commit details
    Browse the repository at this point in the history

Commits on Mar 23, 2019

  1. Updating README and minor changes in manager to manage fail and ok si… 

    …tes.
    @robertomagan
    robertomagan committed Mar 23, 2019
    Configuration menu
    Copy the full SHA
    1898bd5 View commit details
    Browse the repository at this point in the history

  2. Update README.md

    @robertomagan
    robertomagan authored Mar 23, 2019
    Configuration menu
    Copy the full SHA
    c65103e View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2019

  1. All seeds found after adding floodfill sources and i2prouter addressb… 

    …ook.
    @robertomagan
    robertomagan committed Mar 25, 2019
    Configuration menu
    Copy the full SHA
    ad6a2ea View commit details
    Browse the repository at this point in the history

Commits on Mar 28, 2019

  1. Some minor changes in README

    @Abellan09
    Abellan09 committed Mar 28, 2019
    Configuration menu
    Copy the full SHA
    c76ec9e View commit details
    Browse the repository at this point in the history

Commits on Apr 5, 2019

  1. Adding ways get the sites in specific status depending on their statu… 

    …s timestamp in descending order.
    @robertomagan
    robertomagan committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    9548156 View commit details
    Browse the repository at this point in the history

  2. The number of tries to discover a site is equally distributed within … 

    …the max discovering time.
    @robertomagan
    robertomagan committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    aacd2a9 View commit details
    Browse the repository at this point in the history

  3. Getting seeds from floodfill node. A mayor bug has been solver in lin… 

    …king eepsites.
    @robertomagan
    robertomagan committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    c60f09b View commit details
    Browse the repository at this point in the history

  4. Managing IOError when seed files do not exists.

    @robertomagan
    robertomagan committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    0067c63 View commit details
    Browse the repository at this point in the history

  5. all_seeds.txt

    @robertomagan
    robertomagan committed Apr 5, 2019
    Configuration menu
    Copy the full SHA
    d291649 View commit details
    Browse the repository at this point in the history

Commits on Apr 8, 2019

  1. Logging management. Spiders outs log information to logs/spider.log.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    5b8b1a5 View commit details
    Browse the repository at this point in the history

  2. Reducing logging information.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    742211b View commit details
    Browse the repository at this point in the history

  3. Minor bug when restoring spiders in ONGOING status.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    f8b9394 View commit details
    Browse the repository at this point in the history

  4. Re-raise exception in parse method.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    6b3f8b3 View commit details
    Browse the repository at this point in the history

  5. Re-raise exception in parse method.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    7425712 View commit details
    Browse the repository at this point in the history

  6. More logging details.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    f2ec8e7 View commit details
    Browse the repository at this point in the history

  7. More logging details.

    @robertomagan
    robertomagan committed Apr 8, 2019
    Configuration menu
    Copy the full SHA
    1c7cfe2 View commit details
    Browse the repository at this point in the history

Commits on Apr 18, 2019

  1. Monitoring process script.

    @robertomagan
    robertomagan committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    02678e7 View commit details
    Browse the repository at this point in the history

  2. Debugging improvements.

    @robertomagan
    robertomagan committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    96c1c14 View commit details
    Browse the repository at this point in the history

  3. Returning main page tokenized words

    @Abellan09
    Abellan09 committed Apr 18, 2019
    Configuration menu
    Copy the full SHA
    cc2894a View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2019

  1. Checking status coherence of scrapy processes on DB and sub-processes.

    @robertomagan
    robertomagan committed Apr 29, 2019
    Configuration menu
    Copy the full SHA
    8232f11 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2019

  1. Adding log lines for debugging purposes.

    @robertomagan
    robertomagan committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    1c92d35 View commit details
    Browse the repository at this point in the history

  2. Scenario configuration parameters.

    @robertomagan
    robertomagan committed Apr 30, 2019
    Configuration menu
    Copy the full SHA
    bb97fc3 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2019

  1. Saving 'non_visited_links' list in the disk instead of in memory

    @Abellan09
    Abellan09 committed May 2, 2019
    Configuration menu
    Copy the full SHA
    c2d6247 View commit details
    Browse the repository at this point in the history

  2. Adding comments to some functions

    @Abellan09
    Abellan09 committed May 2, 2019
    Configuration menu
    Copy the full SHA
    0f254fc View commit details
    Browse the repository at this point in the history

Commits on May 4, 2019

  1. Removing the site name in non_visited

    @Abellan09
    Abellan09 committed May 4, 2019
    Configuration menu
    Copy the full SHA
    764b2a5 View commit details
    Browse the repository at this point in the history

Commits on May 10, 2019

  1. Debug logs.

    @robertomagan
    robertomagan committed May 10, 2019
    Configuration menu
    Copy the full SHA
    342f2ec View commit details
    Browse the repository at this point in the history

Commits on May 11, 2019

  1. Adding home site text to db.

    @robertomagan
    robertomagan committed May 11, 2019
    Configuration menu
    Copy the full SHA
    81636c8 View commit details
    Browse the repository at this point in the history

  2. Generating UUID for the crawling process.

    @robertomagan
    robertomagan committed May 11, 2019
    Configuration menu
    Copy the full SHA
    3c5aa9d View commit details
    Browse the repository at this point in the history

Commits on May 13, 2019

  1. Changing where the crawling procedure setting is.

    @robertomagan
    robertomagan committed May 13, 2019
    Configuration menu
    Copy the full SHA
    7a51f88 View commit details
    Browse the repository at this point in the history

  2. Deployment process in a distributed environment.

    @robertomagan
    robertomagan committed May 13, 2019
    Configuration menu
    Copy the full SHA
    10e8f63 View commit details
    Browse the repository at this point in the history

  3. Minor changes on deployment/management scripts

    @robertomagan
    robertomagan committed May 13, 2019
    Configuration menu
    Copy the full SHA
    71b001a View commit details
    Browse the repository at this point in the history

  4. Minor changes on deployment/management scripts

    @robertomagan
    robertomagan committed May 13, 2019
    Configuration menu
    Copy the full SHA
    8858f60 View commit details
    Browse the repository at this point in the history

  5. Minor changes on deployment/management scripts

    @robertomagan
    robertomagan committed May 13, 2019
    Configuration menu
    Copy the full SHA
    1a4b64e View commit details
    Browse the repository at this point in the history

Commits on May 14, 2019

  1. Minor changes on deployment/management scripts. Adding psutil depende… 

    …ncy.
    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    f298752 View commit details
    Browse the repository at this point in the history

  2. Minor changes on deployment/management scripts. Adding psutil depende… 

    …ncy.
    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    0855e9b View commit details
    Browse the repository at this point in the history

  3. Minor changes on deployment/management scripts.

    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    dd39d38 View commit details
    Browse the repository at this point in the history

  4. Minor changes on deployment/management scripts.

    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    fda28c9 View commit details
    Browse the repository at this point in the history

  5. Minor changes on deployment/management scripts.

    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    1f4564a View commit details
    Browse the repository at this point in the history

  6. Minor changes on deployment/management scripts.

    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    135aa85 View commit details
    Browse the repository at this point in the history

  7. Minor changes on deployment/management scripts.

    @robertomagan
    robertomagan committed May 14, 2019
    Configuration menu
    Copy the full SHA
    78280b6 View commit details
    Browse the repository at this point in the history

Commits on May 20, 2019

  1. Setting DISCOVERING status when a HTTP error is raised.

    @robertomagan
    robertomagan committed May 20, 2019
    Configuration menu
    Copy the full SHA
    49e538e View commit details
    Browse the repository at this point in the history

  2. Adding virtualized host instances and solving some mintor bug in stat… 

    …us.sh script
    @robertomagan
    robertomagan committed May 20, 2019
    Configuration menu
    Copy the full SHA
    4d8ca1c View commit details
    Browse the repository at this point in the history

Commits on May 23, 2019

  1. Deployment and management scripts modifications.

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    888dff6 View commit details
    Browse the repository at this point in the history

  2. Deleting pid.txt file

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    44328a2 View commit details
    Browse the repository at this point in the history

  3. Now we can also get the status of just one VM

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    fda61df View commit details
    Browse the repository at this point in the history

  4. Deployment improvements

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    85a204f View commit details
    Browse the repository at this point in the history

  5. Deployment improvements

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    22d9f2a View commit details
    Browse the repository at this point in the history

  6. Deployment improvements

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    41f800f View commit details
    Browse the repository at this point in the history

  7. Minor changes, variables response_code and response_time in http requ… 

    …est are not setup when errors.
    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    a25008c View commit details
    Browse the repository at this point in the history

  8. Adding root_path to the setup_bbdd.sh script

    @robertomagan
    robertomagan committed May 23, 2019
    Configuration menu
    Copy the full SHA
    d892778 View commit details
    Browse the repository at this point in the history

Commits on May 24, 2019

  1. Relative path to remotely rung the manager

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    145f581 View commit details
    Browse the repository at this point in the history

  2. Absolute path to remotely stop the manager

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    5d28639 View commit details
    Browse the repository at this point in the history

  3. BBDD VM does not have processes to stop

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    3ad4cfe View commit details
    Browse the repository at this point in the history

  4. Configuring manager

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    a1e109b View commit details
    Browse the repository at this point in the history

  5. Each crawling process must manage only its known sites.

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    a83776c View commit details
    Browse the repository at this point in the history

  6. Each crawling process must manage only its known sites.

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    eba160a View commit details
    Browse the repository at this point in the history

  7. Each crawling process must manage only its known sites.

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    5f14135 View commit details
    Browse the repository at this point in the history

  8. Each crawling process must manage only its known sites.

    @robertomagan
    robertomagan committed May 24, 2019
    Configuration menu
    Copy the full SHA
    7db30e3 View commit details
    Browse the repository at this point in the history

Commits on May 25, 2019

  1. Minor changes in logging and management processes.

    @robertomagan
    robertomagan committed May 25, 2019
    Configuration menu
    Copy the full SHA
    74c3f02 View commit details
    Browse the repository at this point in the history

  2. Minor changes in logging and management processes.

    @robertomagan
    robertomagan committed May 25, 2019
    Configuration menu
    Copy the full SHA
    4db905b View commit details
    Browse the repository at this point in the history

  3. To control sys.out output

    @robertomagan
    robertomagan committed May 25, 2019
    Configuration menu
    Copy the full SHA
    812dac8 View commit details
    Browse the repository at this point in the history

  4. Console log and specific VM stop.

    @robertomagan
    robertomagan committed May 25, 2019
    Configuration menu
    Copy the full SHA
    ef240bf View commit details
    Browse the repository at this point in the history

Commits on May 26, 2019

  1. exists() method launched a exception during if the site being created… 

    … has already been created.
    @robertomagan
    robertomagan committed May 26, 2019
    Configuration menu
    Copy the full SHA
    2ebf0ca View commit details
    Browse the repository at this point in the history

  2. Managing transaction integrity exception.

    @robertomagan
    robertomagan committed May 26, 2019
    Configuration menu
    Copy the full SHA
    43ea928 View commit details
    Browse the repository at this point in the history

Commits on May 28, 2019

  1. Managing transaction integrity exception in distributed environments.

    @robertomagan
    robertomagan committed May 28, 2019
    Configuration menu
    Copy the full SHA
    5bdd5e9 View commit details
    Browse the repository at this point in the history

  2. The usual way to start the crawling process.

    @robertomagan
    robertomagan committed May 28, 2019
    Configuration menu
    Copy the full SHA
    e5b6a00 View commit details
    Browse the repository at this point in the history

Commits on May 31, 2019

  1. Adding missing html5lib to requirements.txt

    @robertomagan
    robertomagan committed May 31, 2019
    Configuration menu
    Copy the full SHA
    05a121a View commit details
    Browse the repository at this point in the history

  2. Installing project dependencies when setting up the environment.

    @robertomagan
    robertomagan committed May 31, 2019
    Configuration menu
    Copy the full SHA
    e94192f View commit details
    Browse the repository at this point in the history

  3. Fixing some bugs on managing scripts.

    @robertomagan
    robertomagan committed May 31, 2019
    Configuration menu
    Copy the full SHA
    7408aac View commit details
    Browse the repository at this point in the history

Commits on Jun 3, 2019

  1. Initial and seeds fairly distribution among all the manager instances.

    @robertomagan
    robertomagan committed Jun 3, 2019
    Configuration menu
    Copy the full SHA
    537dc9b View commit details
    Browse the repository at this point in the history

  2. Config files.

    @robertomagan
    robertomagan committed Jun 3, 2019
    Configuration menu
    Copy the full SHA
    2392213 View commit details
    Browse the repository at this point in the history

  3. Increasing the period of seed self-assigment to 5 mins.

    @robertomagan
    robertomagan committed Jun 3, 2019
    Configuration menu
    Copy the full SHA
    c42de9c View commit details
    Browse the repository at this point in the history

  4. Config for zappa machines.

    @robertomagan
    robertomagan committed Jun 3, 2019
    Configuration menu
    Copy the full SHA
    e697ffa View commit details
    Browse the repository at this point in the history

  5. Config for metis machines.

    @robertomagan
    robertomagan committed Jun 3, 2019
    Configuration menu
    Copy the full SHA
    8b24305 View commit details
    Browse the repository at this point in the history

  6. Config for zappa machines.

    @robertomagan
    robertomagan committed Jun 3, 2019
    Configuration menu
    Copy the full SHA
    118fb99 View commit details
    Browse the repository at this point in the history

Commits on Jun 4, 2019

  1. Bug fixed in processing ok files when an exception is raised.

    @robertomagan
    robertomagan committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    bab90f9 View commit details
    Browse the repository at this point in the history

  2. Config.

    @robertomagan
    robertomagan committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    b3b0bb2 View commit details
    Browse the repository at this point in the history

  3. Config.

    @robertomagan
    robertomagan committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    a4e62de View commit details
    Browse the repository at this point in the history

  4. Config.

    @robertomagan
    robertomagan committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    2aaecf2 View commit details
    Browse the repository at this point in the history

  5. Config.

    @robertomagan
    robertomagan committed Jun 4, 2019
    Configuration menu
    Copy the full SHA
    478c283 View commit details
    Browse the repository at this point in the history

Commits on Jun 7, 2019

  1. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    cd8bb59 View commit details
    Browse the repository at this point in the history

  2. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    afbf299 View commit details
    Browse the repository at this point in the history

  3. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    8bd3db4 View commit details
    Browse the repository at this point in the history

  4. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    e5717d8 View commit details
    Browse the repository at this point in the history

  5. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    e29e824 View commit details
    Browse the repository at this point in the history

  6. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    70c131c View commit details
    Browse the repository at this point in the history

  7. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    684603d View commit details
    Browse the repository at this point in the history

  8. Config.

    @robertomagan
    robertomagan committed Jun 7, 2019
    Configuration menu
    Copy the full SHA
    1fe4a05 View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2019

  1. Performance improvements on managing running alive single discovering… 

    … threads.
    @robertomagan
    robertomagan committed Oct 21, 2019
    Configuration menu
    Copy the full SHA
    21db122 View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2019

  1. Fixing some errors

    @Abellan09
    Abellan09 committed Nov 10, 2019
    Configuration menu
    Copy the full SHA
    db7d2c6 View commit details
    Browse the repository at this point in the history

Commits on Nov 22, 2019

  1. Updating ...

    @robertomagan
    robertomagan committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    e8e227e View commit details
    Browse the repository at this point in the history

  2. Adding an utility to update siteconnectivitysummary table after the e… 

    …xperiment.
    @robertomagan
    robertomagan committed Nov 22, 2019
    Configuration menu
    Copy the full SHA
    92a64cc View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2020

  1. Minor changes in README

    @Abellan09
    Abellan09 committed Jan 14, 2020
    Configuration menu
    Copy the full SHA
    a3a60ae View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/nesg-ugr/I2P_Crawler

    @Abellan09
    Abellan09 committed Jan 14, 2020
    Configuration menu
    Copy the full SHA
    9e72b16 View commit details
    Browse the repository at this point in the history

Commits on Feb 3, 2020

  1. Updating notebooks for analysis ...

    @robertomagan
    robertomagan committed Feb 3, 2020
    Configuration menu
    Copy the full SHA
    5de71a0 View commit details
    Browse the repository at this point in the history

Commits on Apr 20, 2020

  1. New scripts form backing up an experiment from all machines

    @robertomagan
    robertomagan committed Apr 20, 2020
    Configuration menu
    Copy the full SHA
    df9b64b View commit details
    Browse the repository at this point in the history

Commits on May 4, 2020

  1. Snapshot scripts minor changes

    @robertomagan
    robertomagan committed May 4, 2020
    Configuration menu
    Copy the full SHA
    e0aa387 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2020

  1. Script modifications ...

    @robertomagan
    robertomagan committed May 6, 2020
    Configuration menu
    Copy the full SHA
    388f63c View commit details
    Browse the repository at this point in the history

Commits on May 7, 2020

  1. Adding new figures ..

    @robertomagan
    robertomagan committed May 7, 2020
    Configuration menu
    Copy the full SHA
    f97cb4b View commit details
    Browse the repository at this point in the history

  2. Adding and setting up new discovered eepsites now as seeds

    @robertomagan
    robertomagan committed May 7, 2020
    Configuration menu
    Copy the full SHA
    184d918 View commit details
    Browse the repository at this point in the history

Commits on May 11, 2020

  1. Setting up database connection on zappa machines

    @robertomagan
    robertomagan committed May 11, 2020
    Configuration menu
    Copy the full SHA
    b807c18 View commit details
    Browse the repository at this point in the history

  2. Setting up database connection on zappa machines

    @robertomagan
    robertomagan committed May 11, 2020
    Configuration menu
    Copy the full SHA
    3e0bc49 View commit details
    Browse the repository at this point in the history

  3. Setting up database connection on metis machines

    @robertomagan
    robertomagan committed May 11, 2020
    Configuration menu
    Copy the full SHA
    afafc35 View commit details
    Browse the repository at this point in the history

Commits on May 15, 2020

  1. Adding vai host (instead of zappa) and vai configuration.

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    7d4115a View commit details
    Browse the repository at this point in the history

  2. Vai configuration update

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    9335205 View commit details
    Browse the repository at this point in the history

  3. Setting up database

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    9344644 View commit details
    Browse the repository at this point in the history

  4. Vai configuration update

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    e7dd149 View commit details
    Browse the repository at this point in the history

  5. Metis configuration update

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    47e67d9 View commit details
    Browse the repository at this point in the history

  6. Some initial seeds were duplicated. Fixed!

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    572bae1 View commit details
    Browse the repository at this point in the history

  7. Vai's database connection configuration.

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    4cc47ea View commit details
    Browse the repository at this point in the history

  8. Config.

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    976e69d View commit details
    Browse the repository at this point in the history

  9. Config.

    @robertomagan
    robertomagan committed May 15, 2020
    Configuration menu
    Copy the full SHA
    23803fe View commit details
    Browse the repository at this point in the history

Commits on Jun 9, 2020

  1. Adding database post-processing utilities

    @robertomagan
    robertomagan committed Jun 9, 2020
    Configuration menu
    Copy the full SHA
    097e878 View commit details
    Browse the repository at this point in the history

Commits on Jun 11, 2020

  1. Removing waiting for a minute ...

    @robertomagan
    robertomagan committed Jun 11, 2020
    Configuration menu
    Copy the full SHA
    2c97b9e View commit details
    Browse the repository at this point in the history

Commits on Jul 30, 2020

  1. Freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    f2943e1 View commit details
    Browse the repository at this point in the history

  2. Update Git Ignore

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    bdbc01e View commit details
    Browse the repository at this point in the history

  3. Update Config

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    b3cf3eb View commit details
    Browse the repository at this point in the history

  4. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    2785bbc View commit details
    Browse the repository at this point in the history

  5. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    e540cfa View commit details
    Browse the repository at this point in the history

  6. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    bd93fa1 View commit details
    Browse the repository at this point in the history

  7. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    af92fa6 View commit details
    Browse the repository at this point in the history

  8. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    d8871e5 View commit details
    Browse the repository at this point in the history

  9. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 30, 2020
    Configuration menu
    Copy the full SHA
    b785e26 View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2020

  1. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 31, 2020
    Configuration menu
    Copy the full SHA
    20dae9a View commit details
    Browse the repository at this point in the history

  2. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Jul 31, 2020
    Configuration menu
    Copy the full SHA
    a3be2c5 View commit details
    Browse the repository at this point in the history

  3. minor changes to logs

    @EmilioFigueras
    EmilioFigueras committed Jul 31, 2020
    Configuration menu
    Copy the full SHA
    5531725 View commit details
    Browse the repository at this point in the history

Commits on Aug 3, 2020

  1. Update freenet scripts

    @EmilioFigueras
    EmilioFigueras committed Aug 3, 2020
    Configuration menu
    Copy the full SHA
    f70080c View commit details
    Browse the repository at this point in the history

Commits on Aug 4, 2020

  1. Bug fixes and settings changes.

    @EmilioFigueras
    EmilioFigueras committed Aug 4, 2020
    Configuration menu
    Copy the full SHA
    f1a69b6 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2020

  1. Changed nomenclature from I2P to Darknet

    @EmilioFigueras
    EmilioFigueras committed Aug 8, 2020
    Configuration menu
    Copy the full SHA
    52021b3 View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2020

  1. Fixed a minor bug with freesite urls

    @EmilioFigueras
    EmilioFigueras committed Aug 11, 2020
    Configuration menu
    Copy the full SHA
    0c444ef View commit details
    Browse the repository at this point in the history

Commits on Aug 17, 2020

  1. Seed list updated

    @EmilioFigueras
    EmilioFigueras committed Aug 17, 2020
    Configuration menu
    Copy the full SHA
    1904698 View commit details
    Browse the repository at this point in the history

  2. pull updated

    @EmilioFigueras
    EmilioFigueras committed Aug 17, 2020
    Configuration menu
    Copy the full SHA
    2867278 View commit details
    Browse the repository at this point in the history

  3. gitignore updated

    @EmilioFigueras
    EmilioFigueras committed Aug 17, 2020
    Configuration menu
    Copy the full SHA
    f771d4b View commit details
    Browse the repository at this point in the history

  4. Remove UUID

    @EmilioFigueras
    EmilioFigueras committed Aug 17, 2020
    Configuration menu
    Copy the full SHA
    e36ba04 View commit details
    Browse the repository at this point in the history

Commits on Aug 20, 2020

  1. Code analysis performed

    @EmilioFigueras
    EmilioFigueras committed Aug 20, 2020
    Configuration menu
    Copy the full SHA
    7774e2c View commit details
    Browse the repository at this point in the history

Commits on Aug 21, 2020

  1. Database deleted

    @EmilioFigueras
    EmilioFigueras committed Aug 21, 2020
    Configuration menu
    Copy the full SHA
    ea9ad72 View commit details
    Browse the repository at this point in the history

Commits on Aug 30, 2020

  1. Notebook updated

    @EmilioFigueras
    EmilioFigueras committed Aug 30, 2020
    Configuration menu
    Copy the full SHA
    fb11908 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2020

  1. Config.

    @robertomagan
    robertomagan committed Dec 29, 2020
    Configuration menu
    Copy the full SHA
    3374e6c View commit details
    Browse the repository at this point in the history

  2. README update

    @robertomagan
    robertomagan committed Dec 29, 2020
    Configuration menu
    Copy the full SHA
    f3bbbc6 View commit details
    Browse the repository at this point in the history

  3. README update

    @robertomagan
    robertomagan committed Dec 29, 2020
    Configuration menu
    Copy the full SHA
    954cfb5 View commit details
    Browse the repository at this point in the history

Commits on Jul 25, 2021

  1. Update

    @EmilioFigueras
    EmilioFigueras committed Jul 25, 2021
    Configuration menu
    Copy the full SHA
    91c8368 View commit details
    Browse the repository at this point in the history

  2. Updated the name of the configuration parameters

    @EmilioFigueras
    EmilioFigueras committed Jul 25, 2021
    Configuration menu
    Copy the full SHA
    417faa9 View commit details
    Browse the repository at this point in the history

Commits on Sep 14, 2021

  1. Add files via upload

    @EmilioFigueras
    EmilioFigueras authored Sep 14, 2021
    Configuration menu
    Copy the full SHA
    c68adb7 View commit details
    Browse the repository at this point in the history

  2. Delete c4darknet_modules2.pdf

    @EmilioFigueras
    EmilioFigueras authored Sep 14, 2021
    Configuration menu
    Copy the full SHA
    ec3edc1 View commit details
    Browse the repository at this point in the history

  3. Add files via upload

    @EmilioFigueras
    EmilioFigueras authored Sep 14, 2021
    Configuration menu
    Copy the full SHA
    8ec932c View commit details
    Browse the repository at this point in the history

  4. Update README.md

    @EmilioFigueras
    EmilioFigueras authored Sep 14, 2021
    Configuration menu
    Copy the full SHA
    b9d00e2 View commit details
    Browse the repository at this point in the history

Commits on Sep 17, 2021

  1. Merge branch 'master' into master

    @EmilioFigueras
    EmilioFigueras authored Sep 17, 2021
    Configuration menu
    Copy the full SHA
    bebff25 View commit details
    Browse the repository at this point in the history

Commits on Sep 18, 2021

  1. Merge pull request #1 from EmilioFigueras/master 

    c4i2p enhacements for a wide darknet use
    @nesg-ugr
    nesg-ugr authored Sep 18, 2021
    Configuration menu
    Copy the full SHA
    7080a9b View commit details
    Browse the repository at this point in the history

Commits on Sep 19, 2021

  1. Update README.md

    @EmilioFigueras
    EmilioFigueras authored Sep 19, 2021
    Configuration menu
    Copy the full SHA
    5abb0f0 View commit details
    Browse the repository at this point in the history

  2. Merge pull request #2 from EmilioFigueras/patch-1 

    Update README.md
    @EmilioFigueras
    EmilioFigueras authored Sep 19, 2021
    Configuration menu
    Copy the full SHA
    a02b0ab View commit details
    Browse the repository at this point in the history

Commits on Dec 28, 2021

  1. Update LICENSE

    @robertomagan
    robertomagan authored Dec 28, 2021
    Configuration menu
    Copy the full SHA
    df2d6d0 View commit details
    Browse the repository at this point in the history

Commits on Jan 14, 2022

  1. Update README.md

    @robertomagan
    robertomagan authored Jan 14, 2022
    Configuration menu
    Copy the full SHA
    7fd1e3c View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2023

  1. I added some comments ...

    @robertomagan
    robertomagan committed Feb 16, 2023
    Configuration menu
    Copy the full SHA
    911f4ed View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2023

  1. clase TOR_Spider

    @edx173
    edx173 committed Mar 1, 2023
    Configuration menu
    Copy the full SHA
    ab318c3 View commit details
    Browse the repository at this point in the history

Commits on Mar 2, 2023

  1. Adición del proxy TOR y de comentarios

    @edx173
    edx173 committed Mar 2, 2023
    Configuration menu
    Copy the full SHA
    07cf8db View commit details
    Browse the repository at this point in the history

Commits on May 10, 2023

  1. TOR_Spider, proxy configuration and discovertythread.py modifications… 

    … added
    @edx173
    edx173 committed May 10, 2023
    Configuration menu
    Copy the full SHA
    7bd6cbc View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2023

  1. New deployment scripts

    @robertomagan
    robertomagan committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    c1f5525 View commit details
    Browse the repository at this point in the history

  2. New path for remote script execution

    @robertomagan
    robertomagan committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    f8382e0 View commit details
    Browse the repository at this point in the history

  3. Database connections settings updated

    @robertomagan
    robertomagan committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    4024687 View commit details
    Browse the repository at this point in the history

  4. Configuration script updated

    @robertomagan
    robertomagan committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    4da82b6 View commit details
    Browse the repository at this point in the history

  5. Configuration script updated

    @robertomagan
    robertomagan committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    b1598aa View commit details
    Browse the repository at this point in the history

  6. Configuration script updated

    @robertomagan
    robertomagan committed Dec 12, 2023
    Configuration menu
    Copy the full SHA
    15bf738 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2023

  1. Deployment script and configuration files - Ready for crawling TOR sites

    @robertomagan
    robertomagan committed Dec 20, 2023
    Configuration menu
    Copy the full SHA
    b104099 View commit details
    Browse the repository at this point in the history

Commits on Dec 29, 2023

  1. Homogenizing paths for crawling different darknets

    @robertomagan
    robertomagan committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    7b23555 View commit details
    Browse the repository at this point in the history

  2. Adding dummy floodfill_seeds.txt file for tor and freenet.

    @robertomagan
    robertomagan committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    7a65f8e View commit details
    Browse the repository at this point in the history

  3. Adding new i2p seeds and vms for the deployment

    @robertomagan
    robertomagan committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    8128557 View commit details
    Browse the repository at this point in the history

  4. Adding languages types for detection language engines.

    @robertomagan
    robertomagan committed Dec 29, 2023
    Configuration menu
    Copy the full SHA
    a1b87ac View commit details
    Browse the repository at this point in the history

Commits on Jan 3, 2024

  1. Adding scripts for dumping and downloading BBDD after the experiments.

    @robertomagan
    robertomagan committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    cba7525 View commit details
    Browse the repository at this point in the history

  2. Updating script

    @robertomagan
    robertomagan committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    9a1c104 View commit details
    Browse the repository at this point in the history

  3. Updating script

    @robertomagan
    robertomagan committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    f3dd859 View commit details
    Browse the repository at this point in the history

  4. Setting up complete crawling scenario of 15 vms in total.

    @robertomagan
    robertomagan committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    7f18b2e View commit details
    Browse the repository at this point in the history

  5. Freenet seed - url quotes removed

    @robertomagan
    robertomagan committed Jan 3, 2024
    Configuration menu
    Copy the full SHA
    61b1d58 View commit details
    Browse the repository at this point in the history

Commits on Apr 10, 2024

  1. Experimental setup with only 2 machines crawling per darknet

    @robertomagan
    robertomagan committed Apr 10, 2024
    Configuration menu
    Copy the full SHA
    4ba7ea4 View commit details
    Browse the repository at this point in the history