Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Modernize Virustotal API scraping #327

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

theteleforce
Copy link

Address #194 by modernizing the Virustotal scraper.

Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.

This solves the same problem as #285, but does not require an API key and uses the existing parser logic.

This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

- Replace old Virustotal URL with new one
- Add required Virustotal headers required for successful request
Copy link

@its0x08 its0x08 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did almost the same thing for VirusTotal without using API key!
👍🏻

@smed79 smed79 mentioned this pull request Apr 23, 2022
@famdude
Copy link

famdude commented May 18, 2022

It worked. Tnx!

@ko2sec
Copy link

ko2sec commented Jun 29, 2022

works fine, thanks for the fix.

@OMGd0g
Copy link

OMGd0g commented Jul 9, 2022

I'm still learning to Linux here and I'm in douobt which file to change... When doing a search i find it at 2 locations
/usr/lib/python3/dist-packages/sublist3r.py
/usr/lib/python3/dist-packages/theHarvester/discovery/sublist3r.py
Should I edit both for this fix to work?

Thanks :)

@ko2sec
Copy link

ko2sec commented Jul 10, 2022

I'm still learning to Linux here and I'm in douobt which file to change... When doing a search i find it at 2 locations

/usr/lib/python3/dist-packages/sublist3r.py

/usr/lib/python3/dist-packages/theHarvester/discovery/sublist3r.py

Should I edit both for this fix to work?

Thanks :)

theHarvester is another tool, you need to edit /usr/lib/python3/dist-packages/sublist3r.py

@wvdongen
Copy link

The fix works, merge it!

@Defirence
Copy link

Defirence commented Aug 10, 2022

Can we get this merged please? cc: @aboul3la

# Virustotal requires specific headers to bypass the bot detection:
self.headers["X-Tool"] = "vt-ui-main"
self.headers["X-VT-Anti-Abuse-Header"] = "hm" # as of 1/20/2022, the content of this header doesn't matter, just its presence
self.headers["Accept-Ianguage"] = self.headers["Accept-Language"] # this header being present is required to prevent a captcha

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AetherBreeze self.headers["Accept-Ianguage"] should this not be corrected to self.headers["Accept-Language] ? My guess is this might cause an undefined variable error.

@xmegax
Copy link

xmegax commented Sep 12, 2022

it worked , thanks

@firingdove
Copy link

Address #194 by modernizing the Virustotal scraper.

Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.

This solves the same problem as #285, but does not require an API key and uses the existing parser logic.

This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

I am new to linux and using sublist3r. please it possible to provide a step by step instruction on how I can solve the problem. Because I tried using the API key and is not working. thanks

@firingdove
Copy link

it worked , thanks

Please How did you do it?

@xmegax
Copy link

xmegax commented Oct 5, 2022

it worked , thanks

Please How did you do it?

use this repo.
https://github.com/AetherBreeze/Sublist3r.git

@firingdove
Copy link

it worked , thanks

Please How did you do it?

use this repo. https://github.com/AetherBreeze/Sublist3r.git

thanks Man. it is working.

@Defirence
Copy link

Defirence commented Oct 5, 2022

Can we merge this change or is this project now considered being in "development hell"? If it works, WHY can't we commit this? Hello?

EDIT: Looking at @aboul3la 's front page it seems they're either a: working full-time b: dead (hopefully not) or c: this project has been abandoned.

I'm going with the latter being c: Project maintenance has been abandoned. This is a great tool, sad to see even PR's are not being merged. If anyone is looking for a working version that respects the new changes use @AetherBreeze 's forked repo here:

https://github.com/AetherBreeze/Sublist3r

Copy link

@panchal2001 panchal2001 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

working as good

@panchal2001
Copy link

Can we merge this change or is this project now considered being in "development hell"? If it works, WHY can't we commit this? Hello?

EDIT: Looking at @aboul3la 's front page it seems they're either a: working full-time b: dead (hopefully not) or c: this project has been abandoned.

I'm going with the latter being c: Project maintenance has been abandoned. This is a great tool, sad to see even PR's are not being merged. If anyone is looking for a working version that respects the new changes use @AetherBreeze 's forked repo here:

https://github.com/AetherBreeze/Sublist3r

Error: Virustotal probably now is blocking our requests
[!] Error: Google probably now is blocking our requests
[~] Finished now the Google Enumeration ...

@aleeshaford
Copy link

maybe it is possibility to decompile the inofficial VirusTotal Mobile app to have again a rate limit free scraping method without api key.

i haven't gotten that far yet, but it seems like the particular "partner" is calling urls to scan

@anurag-sachan
Copy link

works fine for me, thanks brother!

@Morgoth04
Copy link

Address #194 by modernizing the Virustotal scraper.
Some time in the past few years, Virustotal added a few verification methods to the subdomain API. This pull request changes the endpoint and adds headers so that the request succeeds.
This solves the same problem as #285, but does not require an API key and uses the existing parser logic.
This pull request has been tested on Python 3.9.2, Kali Linux rolling 2021/2.

I am new to linux and using sublist3r. please it possible to provide a step by step instruction on how I can solve the problem. Because I tried using the API key and is not working. thanks

Almost 4 years late, sorry:

  1. The url line in the class corresponding to Virustotal (class Virustotal(enumratorBaseThreaded):) of the sublist3er.py program must be corrected (base_url = 'https://www.virustotal.com/api/v3/domains/{domain}/subdomains').
    2)2) Place the apikey that was obtained once registered as a virustotal user in the variable sel.apikey (self.apikey = os.getenv('VT_APIKEY', 'hereenterthealphanumeric code'))
    3)Save the file
  2. Overwrite the sublist3r.py file in the Kali Linux path where the one you have was placed
  3. Try with the command sudo ./sublist3r.py -d domainchosen.com

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.