Collect Robots.txt parse and create a wordlist
This is a simple starter application in .NET5 Coreapp that takes a list of domains and gets the robots.txt if exists.
From the robots.txt it parses the values to create a dictionary wordlist for directory brute forcing. Why? I moved to Brazil and there is a lack of wordlists for directory brute forcing that contains portuguese words. In order to create a wordlist, I decided to start here with the robots.txt files and see what I get.